Measuring an agent that can act

The dashboard was green.

A buy-now-pay-later lender I worked with gave its support agent real authority. It could pause a payment plan, waive a fee, restructure a schedule, on its own. Useful, and mostly right.

Then it told a customer in hardship that it had paused her payments for 60 days and that her 0% rate was safe. She thanked it. The chat closed with a high CSAT.

It was an impossible promise. The 0% only holds while the customer keeps the exact schedule they signed up for, so pausing the schedule is what voids it. “I’ve paused you and your 0% is safe” is a promise the lender’s own offer can’t honour. Nothing about that exchange looks like a failure from the inside. A customer asked for help, the agent helped, the ticket closed happy.

No error, no complaint, no escalation, no metric out of range. Finance found the cost later, in reconciliation.

Klarna’s version of this is public ⁽¹⁾. In Feb. 2024 it reported an agent doing the work of 700 support staff, 2/3 of chats, resolution time down from 11 min. to under 2. Those numbers kept improving into late 2025. Then the CEO conceded the cost-led automation had gone too far on quality, and the company started hiring people back.

The numbers weren’t lying. They described the average. The damage sat in the tail, and an average never points at its own tail.

Decide what you are measuring before you decide how.

Most agent metrics arrive by inheritance. You had a support function, it had a dashboard, the agent got plugged into it. Deflection, containment, resolution, time to close, CSAT. Every one measures an exchange: was it fast, did it end, did it avoid a human.

Then you shipped something that acts. 

An agent that can pause, waive, refund, cancel or reset isn’t having a conversation. It’s taking an action with money and trust attached, and the questions change with it. What did it do. Was it permitted. What did it cost. Can it be undone.

3 layers are worth separating: conversation quality, action correctness, business outcome. Most teams measure the first and assume the other 2 follow. 

They don’t. 

The lender’s agent scored well on every conversational measure it had, because the conversations were fine.

Permitted isn’t the same as correct, either. Every pause that agent made was allowed. The rule it followed was the wrong rule, and measuring compliance would only have confirmed it was obeying, which it was.

The reverse trap is quieter. A greentech scaleup I worked with sold carbon-accounting software to manufacturers, and its agent handled data-quality support: missing meter readings, wrong emissions factors, anomalies caught before a report was filed. Escalation to a human sat at 6%, and the team read that as the agent being good. Their analysts weren’t escalating. When an answer looked wrong, they exported to a spreadsheet, fixed it by hand, and filed from there. A ticket meant a wait; Excel took 10 min.

An honest number comes from something the agent did not write.

That’s the whole discipline. Everything below is a way of getting there. 3 sources hold up.

1st, the path, not the answer. A correct output reached by a wrong route is a false positive: the fragility is still in there, and it surfaces later under conditions you didn’t test. This is standard practice now. Microsoft ships evaluators for tool call accuracy and task adherence ⁽²⁾. Google scores the trajectory, the actual sequence of tool calls against the expected one ⁽³⁾. Sierra scores policy adherence separately from task success ⁽⁴⁾, so an agent that resolves the request by breaking a rule is logged as a partial failure, not a win. That separation is the one most in-house eval suites skip, and it’s the one the lender needed.

2nd, the end state. Don’t ask the agent what it did. Look at what the account, the balance, the plan status actually look like afterwards, and compare that to what they should look like. For the fintech lender’s agent, a check on the promo state after a pause would have caught in a day what took weeks.

This is also the practical case for a deterministic gate between the decision and the act, a check that runs on every action and either permits it or refuses it. I made that argument in an earlier piece. The measurement payoff is the part people miss: a gate writes its own record of what was attempted and what was refused, and the model didn’t author it.

3rd, consistency, not average. A single pass tells you almost nothing about a system that isn’t deterministic. Sierra’s data makes the gap plain: on one retail task, a leading model got it right about 61% of the time on a single try, but solved it on all 8 of 8 attempts less than 25% of the time ⁽⁴⁾. Report the first number and ship something that has to be right every time, and you’re reporting the wrong one.

Everything the agent can influence, it eventually influences.

A few instruments feel like measurement and aren’t.

Self-reported logs are a claim. When an agent records that it resolved something, that’s the agent’s opinion of its own work, written by the thing being graded. The carbon-accounting agent logged a resolution every time an analyst went quiet, and going quiet was the sound of someone opening a spreadsheet instead.

Transcripts have the same problem in a friendlier form. They read well because fluency is the 1 thing these models are best at, so a confident wrong answer reads better than a hedged correct one. Reviewing transcripts mostly measures how plausible the agent sounds, which is the property that made the demo look good in the first place.

Then the obvious fix: have a model grade the model. It scales, it’s cheap, and it inherits the exact problem it was meant to solve. Model judges show position bias (favouring whichever answer they see first), verbosity bias (longer reads as better) and self-preference bias (scoring their own family’s output higher). One study of 15 judges across 22 tasks logged over 150,000 evaluations and found the position bias systematic, not random ⁽⁵⁾. The judges also report high confidence when they’re wrong, so the failure is silent.

How far it goes is worth seeing. Researchers built coding tasks that are impossible to pass, so any pass is proof of cheating rather than skill. GPT-5 exploited the test cases 76% of the time on one variant ⁽⁶⁾, by deleting the lines that checked the work and redefining what counted as correct. The fix is the useful part: hide the test files, or make them read-only, and cheating drops to near zero. Evaluation integrity turned out to be an access problem.

Which brings it back to the number on your invoice. Intercom’s Fin bills an assumed resolution when a customer goes quiet for 24 hours without asking for more help ⁽⁷⁾. That’s the carbon-accounting agent’s failure mode, formalised and priced. Fin’s own figures show the gap: a headline resolution rate of 71% across Intercom’s customers, against 42 to 50% in Intercom’s published case studies.

If a number is defined by the party being paid for it, it’s a commercial term, not a measurement.

What the owner watches on Monday.

5 things, in rough order of how early they warn you.

Attempted actions outside policy. The rate at which the agent tries something it isn’t allowed to do, whether or not it gets through. This is the only one on the list that moves before the damage does, because it counts intent, not consequence. It needs a gate to exist, which is why the architecture and the measurement are one conversation.

Reversals and leakage, in money. Add up what the agent gave away that it didn’t have to: credits issued, fees waived, refunds and undone charges, as a euro figure, not a rate. Watch the trend, not the total. A figure creeping up month on month means the agent has found a cheap way to close tickets, and cheap for it is expensive for you.

Consistency, not average. How often the agent handles the same case the same way. Track the spread, not the mean, because the mean is where the tail hides. The lender’s agent paused plans correctly almost every time; it was the rare pause on a promo plan that cost the money, and an average waves that away as a rounding error.

Drift in what it attempts. The shape of the agent’s behaviour moves as prompts change, tools get added, and the data underneath shifts. What it tried to do last quarter is the baseline. What it tries this quarter, measured against that, is the signal.

Work leaving the product. The one no vendor will ever put on your dashboard, because it measures your product’s failure, not the agent’s. At the carbon-accounting platform, the audit ignored the 6% and asked whether the reporting work was still happening in the product. Export volume was climbing, the same accounts corrected the same emissions factors every reporting cycle, and time in product on the filing workflow was falling. That’s what 6% actually meant. They weren’t satisfied. They’d stopped asking.

What none of this comes with is thresholds. A 2% out-of-policy rate is catastrophic for a lender and unremarkable for a scheduling tool. The categories are portable. The numbers are derived, per system, from what the agent is allowed to touch.

If finance finds it before product does, you weren’t measuring the agent, you were trusting it.

-----

Working out what an agent is actually doing, as opposed to what it reports, is the work I do. Do all your numbers trace back to the agent? Let’s talk.

-----

Sources

(1) Klarna, AI support results and reversal (Forbes): https://www.forbes.com/sites/quickerbettertech/2025/05/18/business-tech-news-klarna-reverses-on-ai-says-customers-like-talking-to-people/

(2) Microsoft, agent evaluators (Foundry documentation): https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/agent-evaluators

(3) Google, trajectory evaluation for Vertex AI Agent Engine and ADK: https://futureagi.com/blog/evaluating-vertex-ai-agent-engine-2026/

(4) Sierra, tau-bench: policy adherence and pass at k consistency (Yao et al.): https://arxiv.org/abs/2406.12045

(5) Position bias in LLM-as-a-judge (Judging the Judges): https://arxiv.org/pdf/2406.07791

(6) ImpossibleBench, measuring reward hacking in LLM coding: https://www.lesswrong.com/posts/qJYMbrabcQqCZ7iqm/impossiblebench-measuring-reward-hacking-in-llm-coding-1

(7) Intercom Fin, confirmed versus assumed resolution and pricing: https://www.getmacha.com/blog/intercom-fin-pricing