How to Measure AI Agents in the Contact Center: The Metrics That Replace Containment and FCR
Containment, FCR and handle time score failed AI conversations as wins. Here are the metrics contact centers should use to measure AI agents instead.
Make CX Current News one of your go-to sources on Google
Containment, first contact resolution, handle time and sampled QA were designed to manage people. Applied to AI agents, they routinely score failed conversations as wins. Contact centers running AI at scale need a different scorecard: verified resolution, factual accuracy, policy adherence, action verification and consistency, checked on every conversation rather than a sample.
The Air Canada chatbot case
In 2022, a passenger booking a flight after a death in the family asked Air Canada's website chatbot about bereavement fares. The chatbot indicated he could buy a full-price ticket and request the reduced fare afterward. When he later applied, he learned the airline's policy didn't allow retroactive claims.
On a standard contact center scorecard, that chat would have looked fine. The customer got a clear answer without a transfer and didn't contact the airline again that day, so it would have counted as a contained, first-contact resolution, most likely with a short handle time.
In February 2024, British Columbia's Civil Resolution Tribunal ruled in the passenger's favor, finding that a company is responsible for the information its chatbot provides in the same way it is for any other page on its website. The award was modest, about $812 in damages and fees.
That case involved one chat and a few hundred dollars. The exposure today is far larger. Gartner predicts that by 2029, agentic AI will resolve 80% of common customer service issues with no human involved, and most contact centers still grade those conversations with the same metrics that would have scored that chat as a success.
Where traditional contact center metrics fail with AI agents
Most contact center metrics were designed to diagnose human performance problems. Handle time flags agents who work slowly, first contact resolution flags handoffs that force customers to repeat themselves, and QA scorecards check that agents follow procedure and read required disclosures. When the agent is software, each of these ends up measuring something other than what it was designed for.
Containment and deflection
Containment, or deflection, records a win whenever a conversation ends without reaching a human. It can't tell the difference between a customer whose problem was solved and one who gave up. The Air Canada chat would have counted as contained, as would any conversation where the customer simply closed the window and went elsewhere for help.
A 2023 report from the Consumer Financial Protection Bureau found that bank chatbots handled basic questions reasonably well but became less effective as problems grew more complex. It described customers stuck in "doom loops" of repetitive automated answers with no route to a person, and warned that inaccurate chatbot answers could put institutions in violation of consumer finance law. Each of those conversations would register as contained.
First contact resolution
FCR asks whether the customer had to come back. With an AI agent, almost every conversation is a first contact, so a confidently wrong answer clears the bar as easily as a correct one.
FCR also becomes harder to read on the human side. Once AI absorbs the simple one-touch requests, human agents handle a higher share of complex, multi-contact cases, and their FCR tends to fall. That decline usually reflects a harder call mix rather than weaker performance, and treating it as a coaching issue misdiagnoses the problem.
Average handle time and CSAT
Average handle time was built around a person talking, holding and doing after-call work. An AI agent that answers fast and wrong posts a better number than one that takes longer and gets it right.
CSAT comparisons between AI and human agents are distorted by routing. If AI handles the simpler intents and people handle the harder ones, AI will usually score higher regardless of quality. Response bias adds to the problem. Gartner found that 64% of customers would prefer companies not use AI for service, and 53% would consider switching to a competitor over it. A customer who feels that way after a bad bot interaction is more likely to leave than to fill out a survey.
Sampled QA
Most QA teams can review only a small fraction of interactions by hand. Sampling works for estimating a team average. It's a poor way to find problems that happen on a small share of conversations.
Suppose a missed disclosure happens on 2% of calls, and a reviewer checks ten of an agent's calls at random. The chance of seeing even one instance is about 18%. Across a whole floor, a random sample finds roughly the same share of violations as the share of calls it covers. Review 3% of conversations and you'll see about 3% of the problems.
The limitation is more serious with AI agents. A human error is usually confined to one interaction. A flaw in a prompt, knowledge article or model version repeats on every conversation that follows the same path until someone finds it.
Scorecard design is a separate issue. Traditional QA forms grade tone, greetings, script adherence and compliance phrasing, the areas where human agents typically slip. They rarely include any check on whether the information the agent gave was correct, so an agent can follow every step and still pass while stating something false.
Common AI agent failure modes
AI agents fail differently from human agents, and most of these failures don't register on scorecards built for people. Five patterns come up repeatedly in production deployments.
Hallucinated policies and facts
In April 2025, users of the coding tool Cursor were being logged out unexpectedly. When they asked support why, an AI support agent named "Sam" explained it with a login policy that didn't exist. Some users canceled their subscriptions over the invented rule.
New York City's MyCity chatbot is a more serious example. Testing by The Markup in 2024 found it telling business owners they could take a cut of workers' tips and refuse cash payments, both against city law. It also didn't know the minimum wage. The city shut the bot down in 2026, with the new mayor calling it functionally unusable.
In both cases the responses were fluent and on topic, so a scorecard focused on tone and required phrasing would likely have passed them.
Inconsistent answers
The Markup's testing also showed how inconsistent the bot was. One reporter was told landlords had to accept housing vouchers. When ten other staffers asked the same question, all ten were told they didn't.
Because language models are probabilistic, some variation is expected, and average accuracy figures obscure it. An agent that handles a task correctly 90% of the time sounds dependable. Assuming each attempt is independent, the chance it handles the same task correctly ten times in a row is about 35%. Anthropic's guide to evaluating AI agents draws the same line between pass@k, the chance an agent succeeds at least once in k tries, and pass^k, the chance it succeeds every time. The two numbers match on a single attempt and move in opposite directions as attempts increase. For customer service, the second measure is the one that matters.
Multi-turn and off-script conversations
Customers rarely describe their whole issue at the start of a conversation. They add details, revise requests and change topics partway through. Cresta's engineering team described airline customers who start booking a flight, then stop midway to change passenger details, ask about mileage perks or merge miles before paying. Each detour requires the agent to switch tasks without losing track of where the booking stands. In the same post, the team described a security deployment where callers gave their authentication codeword as "codeword" or "pardon," and the agent refused to process them because it assumed those weren't valid inputs.
Actions that never execute
An AI agent acts through tool calls: authenticating a caller, issuing a refund, filing a ticket. A transcript can look correct even when the underlying action failed or never ran.
Cresta's engineers described a loan-servicing agent that had to authenticate callers before sharing payoff amounts. An LLM grader reading transcripts passed it 94% of the time, but adding a check that the authentication tool had actually been called dropped the pass rate to 71%. The grader had been inferring authentication from the flow of the conversation.
Regressions after model updates
An AI agent's behavior can shift whenever its underlying model, prompt or knowledge base changes, even when no one intended to change how it handles customers. After a routine model update at a financial services client, Cresta's regression tests caught the agent dropping a required compliance disclaimer. It was fixed within hours. Without automated monitoring, the team said, it would have surfaced in a compliance review or not at all.
What happened at Klarna
Klarna is the most widely cited case of strong operational metrics masking a quality problem. In its 2024 annual report, the company said its AI assistant did the work of 700 employees and cut average resolution time from 11 minutes to two, with customer satisfaction matching human agents. By May 2025, CEO Sebastian Siemiatkowski told Bloomberg the company was hiring human agents again, admitting that cost had been too dominant a factor and that the result was lower quality. By the metrics the company had been reporting, though, the program was working.
A metrics framework for AI agents
Legacy metrics don't all need to go. Cost and volume still matter, and several existing measures work with adjustments. The change is in which metrics are treated as evidence that an AI agent is performing well.
Metrics to retire for AI agents
Containment can stay on the dashboard as a volume figure, but it should never be reported as resolution. First contact resolution and average handle time tell you very little about an AI agent, for the reasons above, and should come off its scorecard entirely.
Metrics to keep, with adjustments
Human FCR still matters, but compare it by intent rather than in aggregate, and expect the overall number to fall as AI takes the simple requests. Break CSAT out by intent and channel, and watch survey response rates on AI conversations as closely as the scores. Keep cost per contact, but divide by verified resolutions instead of contained conversations.
Eight new metrics built for AI agents
Verified resolution should become the headline number. Count a conversation as resolved only if the customer doesn't come back on the same issue within a set window, such as seven days, through any channel. Most contact centers already have this data in their CRM. It just needs to be joined across channels, so that a customer who leaves the bot and calls a human two days later counts as a failure for the bot.
Accuracy needs to be measured in two directions. An agent can fail by leaving out something it was required to say, or by saying something it wasn't allowed to say. Cresta's engineering team scores those separately, because in regulated industries they carry different risks, and a single blended score hides which one is happening.
Policy adherence turns each business rule into a yes-or-no check that someone could verify from the transcript alone. Did the agent offer a billing specialist when the customer disputed a charge? Did it give the privacy disclosure before discussing health information?
Action verification confirms in system logs that the refund, authentication or transfer actually happened. Without it, transcript-based scoring can't detect actions that failed.
Consistency, measured as pass^k, is rarely reported today. Run the same scenario many times and record how often the agent gets it right on every single run.
Escalation quality grades whether handoffs came at the right moment and whether the human received the full context. Early escalations waste agent capacity, while late ones mean the customer has already sat through several failed attempts.
Regression rate tracks what broke after each model, prompt or knowledge change. Report it the way an engineering team reports failed releases.
Coverage should be 100%. Every AI conversation gets scored automatically, and human reviewers spend their time on the ones that were flagged or carry the most risk.
How to run QA on AI agents
The teams doing this well treat evaluation as a standing operation that runs before launch, at launch and every day after. Anthropic and Cresta both describe it with the Swiss cheese model from safety engineering: every layer of testing has holes, so you stack enough layers that a failure slipping through one gets caught by another. A working program usually includes eight practices.
Write requirements a stranger could grade. A vague requirement such as "be helpful" produces inconsistent verdicts, while a specific one such as "confirm the appointment date and time before ending the call" does not. Cresta's team holds each requirement to what it calls the intern test: could someone with no context, reading only the transcript, reach the same verdict?
Build test cases from real conversations. Scripted happy paths miss how customers actually behave. In one healthcare deployment, simulated customers drawn from historical calls exposed a gap the scripted tests never touched: urgent requests were being handled exactly like routine ones. Anthropic suggests starting with a small set of tasks drawn from real failures rather than waiting for a perfect suite.
Use an LLM to grade language and code to grade actions. A language model is a good judge of tone, accuracy against approved claims and whether a policy was followed in what was said. It's a poor judge of whether a tool fired. Anything recorded in structured data, such as a transfer, a retrieved article or a captured account number, should be checked directly. That's the gap that took the loan-servicing agent from 94% to 71%.
Calibrate the grader before trusting it. Have human experts score the same conversations as the automated judge and resolve every disagreement. Most disagreements reveal a requirement that was written too loosely. Promote the grader to production only once its verdicts match human judgment.
Test the same scenario many times. A single passing run says little about reliability. Repeat each scenario enough times to see the variance, and report pass^k alongside the average.
Rerun everything after every change. A model upgrade, a rewritten prompt or an edited knowledge article can each break behavior that passed last week. Treat them like a software release, with a full regression run before anything ships.
Score every production conversation. Automated grading removes the sampling limit that hides rare failures. Human reviewers then spend their time on flagged conversations, high-risk intents and calibration, instead of on random draws.
Turn production failures into tests. When a failure appears in production, add it to the regression suite so it is caught automatically if it recurs.
This also changes what a QA team does. Analysts who used to listen to calls become the people who write requirements, calibrate graders and investigate patterns across thousands of conversations. Those skills are closer to test engineering than to call monitoring, and most organizations will need to train for them.
What this means for CX leaders
The Air Canada ruling rests on a straightforward principle: a company is responsible for what its AI agent tells customers, just as it is for anything else it publishes.
That makes AI agent measurement a leadership issue as well as an operational one. Organizations that report containment as their primary AI metric are tracking the kind of numbers that looked strong at Klarna before its reversal. A more defensible position is to be able to show, for any given conversation, that the agent's answer was correct, that the promised action was completed and that the agent would have answered the next customer the same way.
Frequently asked questions
What metrics should you use to measure AI agents in a contact center?
Lead with verified resolution rate, meaning the customer didn't return on the same issue within a set window. Pair it with accuracy against approved information, policy adherence, confirmation that system actions actually happened, and consistency across repeated attempts. Keep cost per resolved contact and segmented CSAT. Treat containment as a volume figure only.
Is containment rate a good measure of AI agent performance?
No. Containment only shows that a conversation ended without a human. It counts customers who gave up, and customers who left with a wrong answer, as successes.
Why doesn't first contact resolution work for AI agents?
Nearly every AI conversation is a first contact, so wrong answers pass as easily as right ones. FCR for human agents also tends to fall once AI takes the simpler requests, which reflects call mix rather than performance.
How many AI agent conversations should be reviewed?
All of them, automatically. Random samples of a few percent will miss most failures that occur on a small share of conversations. Human reviewers should focus on flagged conversations, high-risk intents and checking the automated grader.
What is pass^k?
Pass^k is the probability that an AI agent succeeds on the same task in every one of k attempts. It measures reliability, which an average success rate can overstate. An agent that is right 90% of the time on a task is right on ten runs in a row only about a third of the time.
Can an LLM grade another AI agent's conversations?
Yes, for things that live in the language of the conversation, such as tone, accuracy and policy compliance, provided its verdicts are calibrated against human experts first. For actions like authentication, refunds and transfers, check system logs directly.
