The score improved. The failures migrated: what aggregate AI agent evals hide

Writing agent code is becoming faster. Verifying that a release has not introduced regressions, or merely exchanged one failure mode for another, is not. Evals help, but an aggregate score can conceal how agent behavior changed underneath it.

I wanted to understand the behavioral cost of switching a deployed agent to a lower-cost open model. The question was not only whether the evaluation score changed. It was how the agent's behavior changed and what that could mean for users.

Most AI agent evals eventually compress a release into a number: pass rate, task completion, average reward, or an aggregate quality score. That number helps compare candidates. It also hides which users, actions, and parts of the agent loop are now failing.

Two candidates can land near the same score while breaking differently. One repeatedly inspects a page without acting. Another stops before gathering the evidence needed to answer. A third reaches the correct intent but produces an invalid tool action. Those failure modes need different mitigations.

The experiment

I replayed the same frozen set of 100 SentinelBench scenarios against two deployed agent configurations:

• GLM-5.2 through OpenRouter, with the z-ai provider pinned: 50 passed and 50 failed.
• GPT-5.6 Terra through Azure: 58 passed and 42 failed.

The scenario suite, prompts, local simulation harness, evaluator, and deterministic failure detector stayed fixed. The deployed setup changed. This compared two real serving configurations, not clean model weights, because provider, route, and supported temperature also differed.

This was one accepted run per configuration. The result is an observed difference worth investigating, not a stable estimate of either model's intrinsic failure distribution. Each run retained its official result, browser trace, screenshots, and provider ledger. Deterministic code then classified failures from that frozen evidence.

What the score hid

The aggregate result improved by eight points. Underneath it, the failure distribution moved substantially.

39

scenarios failed under both configurations

17

shared the same strict deterministic failure signature

22

failed in both, but through a different mechanism

Across the complete comparison, 33 signatures appeared only in the GLM deployment and 25 only in the Terra deployment. GLM produced more missed actions and missing evidence. Terra produced much more loop churn: 18 Terra-only cases versus two GLM-only cases.

That does not establish permanent model personalities. It records what happened in these two accepted deployments. The useful release fact is narrower: choosing by the improved aggregate score alone would hide a meaningful migration in how the agent failed.

My original hypothesis was that switching to the higher-scoring configuration would improve the system and reduce failures. The aggregate score improved by eight points. What the score did not capture was the behavior users could encounter when the remaining failures changed shape.

In this experiment, inspecting the failure migration took longer than making the model change. The score told me how much failed; the traces told me how failure changed.

The stronger claim the evidence did not support

I was tempted to treat a failure recurring across two deployments as proof that the task or harness owned the problem. The evidence could not support that conclusion.

Recurrence shows that a failure was not unique to one observed deployment. It does not identify the cause. Both models can make the same mistake. A shared prompt, browser policy, or adapter can induce it. A deterministic signature can also group traces whose finer execution evidence differs.

Six of the 17 shared signatures contained different finer churn or error evidence. The right conclusion was to narrow the claim and run another test.

The correction study

I selected three shared failures as targeted investigations, not as a representative sample: a calendar case with missing evidence, a jobs case with loop churn, and a music case with an output-contract failure.

A deterministic golden agent followed the known-correct path through the unchanged frontend, simulator, and evaluator. It used zero model calls. All three trajectories passed.

That rules out an impossible task or an evaluator that always rejects the correct state for those three cases. It does not prove that model weights caused the original failures. It also does not exclude the prompt, policy, adapter, or frontend difficulty. The honest result is that none of the three was confirmed as harness-owned.

What I want to make easier

This experiment does not prove that teams need another release-control platform. It exposes a smaller practical problem: an aggregate eval result can hide consequential behavioral movement between two deployed agent configurations.

That is the first workflow I want to test with Triana Preview, a local tool under development. It starts with situations grounded in an agent's production conversations, runs the baseline and candidate from the same historical context, repeats executions, and shows where behavior reliably diverges. Paired transcripts expose the first divergence. Simulated continuation and missing evidence remain explicitly labeled.

The output is not a universal ship-or-block verdict. It is a reviewable account of what moved, how stable the observation was, and what the available evidence cannot establish. Braintrust, LangSmith, and existing eval systems can keep owning traces, datasets, scorers, and experiments. The question is whether this comparison can become simple and reusable enough that teams stop rebuilding scripts whenever they make a consequential agent change.

If you operate a conversational or tool-using agent and have production traces plus a safe baseline and candidate, I would like to understand how you inspect behavioral movement today. Do you see it before release, or only after users find it? You can bring a release to Triana.