Model swaps in agent harnesses behave more like migrations

At a glance

What this is: This is a controlled comparison of seven models under one agent harness, showing that final correctness stayed relatively close while operational behavior diverged.

Why it matters: For IAM and NHI practitioners, the finding matters because agent governance depends on more than answer quality: tool use, retries, latency, and failure paths all shape control risk.

By the numbers:

• The harnessed runs clustered between 79.6% and 85.1% correctness across seven model targets.
• The sweep covered 1,330 attempted examples across 19 tasks, seven models, and ten runs per model.
• Sonnet 4.6 averaged 9.5 seconds per task and 1.45 tool calls, while GPT-5 averaged 32.2 seconds and 3.14 tool calls.
• Gemini 3 Flash averaged 5.82 tool calls per task, the highest tool count in the harnessed sweep.

👉 Read Arize's analysis of model swaps under a shared agent harness

Context

Model selection in agentic systems is not just a cost or benchmark question. When an agent is wrapped in tools, state, retries, and evaluator logic, the model becomes one component in a broader operating system, and the governance problem shifts from answer correctness to runtime behaviour, reliability, and control stability.

That makes the article relevant to identity governance because AI agents increasingly act like non-human identities in production workflows. When model swaps change latency, tool usage, or retry patterns, they can also change the effective blast radius of delegated access, even if the final response still appears acceptable.

Key questions

Q: How should teams evaluate model swaps in agentic systems?

A: Treat model swaps like production migrations. Keep the harness, tools, fixtures, prompts, and scoring constant, then compare correctness with latency, retries, tool calls, and timeout behaviour. A model that still answers correctly can still create more operational risk if it takes longer or uses the tool path less efficiently.

Q: Why do agents need governance beyond final-answer accuracy?

A: Because the answer is only the end state. In agentic systems, the path to that answer determines cost, exposure, and reliability. Two models can produce the same result while one triggers more tool activity, more retries, and more timeout risk, which changes the effective security and operational profile.

Q: What do security teams get wrong about agent harnesses?

A: They often treat the harness as a testing convenience rather than part of the control plane. In reality, instructions, permissions, retries, schemas, and tool access shape the agent’s behaviour and can either constrain or amplify risk. The harness is part of the governance boundary.

Q: Should organisations trust a model that performs well in a benchmark?

A: Only if the benchmark mirrors the production harness and task mix. Benchmark scores can hide whether the model behaves safely under real tool pressure, longer sessions, or different permission boundaries. Trust should be earned in the same execution conditions the system will face in production.

Technical breakdown

Why agent harnesses change model behaviour

An agent harness is the surrounding execution layer that turns a model call into a task-capable system. It includes instructions, tool permissions, schemas, state, retries, rate limiting, and evaluation rules. That layer constrains the model but also shapes how often it calls tools, how long it runs, and how it recovers from ambiguity. In practice, the same model can look stable in one harness and fragile in another because the orchestration layer changes the decision path, not just the output format.

Practical implication: evaluate model changes inside the same harness, not as isolated prompt tests.

Why final-answer evals miss operational risk

Final-answer scoring tells you whether the outcome matched expectation, but it does not tell you what the agent had to do to get there. Two models can reach the same answer while one uses one clean query and the other burns through retries, redundant searches, or timeouts. That difference matters because operational friction creates cost, latency, and failure exposure. In agentic systems, the path is part of the control surface, not just an implementation detail.

Practical implication: score latency, tool calls, retries, and timeout rate alongside correctness.

What harness stability really proves

A stable harness does not eliminate model drift. It reduces noise by fixing the tool path, task format, and scoring conditions so differences are easier to compare. That is useful, but it only proves that the harness can hold behaviour within a tighter band under test conditions. It does not guarantee production sameness, especially when real repositories, changing permissions, and uneven task complexity introduce new failure modes.

Practical implication: treat harness stability as a gating signal, not evidence that models are interchangeable.

NHI Mgmt Group analysis

Model swaps in agent systems are migration events, not configuration changes. The article shows that the API surface may stay nearly constant while the underlying execution profile shifts materially. That is exactly how governance blind spots form: teams compare the model name, not the runtime path. In identity terms, the delegated work performed by the agent can change even when the request looks identical, which means access governance must follow the execution path, not the branding of the model.

Operational behaviour is the real control variable in agent governance. Correctness clustered tightly, but latency, retries, and tool calls diverged enough to change production risk. For AI systems that function like non-human identities, those runtime characteristics are part of privilege behaviour, because they determine how much system access is consumed and how often controls are exercised. Practitioners should treat excessive tool traffic and repeated retries as governance signals, not just performance noise.

Harness design is becoming the equivalent of identity policy for agents. The benchmark suggests that fixed scaffolding can constrain drift, but only within the assumptions the harness encodes. That creates a named governance concept worth tracking: harness-bound trust, where control confidence depends on the exact execution wrapper rather than the model itself. The practical conclusion is that agent governance needs versioned harnesses, not just versioned models.

Model evaluations should be judged like access reviews, not like leaderboard checks. A leaderboard can hide whether the agent reached the answer through one bounded path or a sprawling sequence of tool actions. That is a dangerous assumption in environments where model changes can alter data access, side effects, and downstream workload identity exposure. The better standard is to verify both task success and the behavioural cost of achieving it.

Agent orchestration is now part of the security boundary. The same model under a fuller scaffold performed more reliably than in the raw baseline, which means the wrapper absorbs real risk. In broader identity and AI governance programmes, that makes orchestration logic, tool permissions, and retry policy first-class controls. Teams that ignore the harness will underestimate how much of agent safety actually lives outside the model.

What this signals

Agent governance is converging with identity governance because the runtime path now matters as much as the final decision. When a model swap changes tool volume or timeout behaviour, the system’s effective privilege pattern changes too, which means agent reviews need the same discipline applied to non-human identities and service accounts.

Harness-bound trust: the confidence you place in an agent is only as strong as the wrapper that constrains it. That makes version control, permission scoping, and retry policy operational controls, not engineering preferences. For readers building agentic AI programmes, the next step is to treat the harness as a governed asset alongside the model and the data source.

The broader signal is that agent evaluations must move closer to control validation. NIST AI Risk Management Framework and OWASP Agentic AI Top 10 both align with this direction because they emphasise lifecycle governance, misuse resistance, and exposure management, not just benchmark performance.

For practitioners

• Freeze the harness before swapping models Keep tools, prompts, fixtures, scoring, and retry policy constant so model-to-model comparison stays clean. If the harness changes, you are measuring two variables at once and cannot tell whether behaviour drift came from the model or the wrapper.
• Score operational behaviour, not just correctness Track latency, tool-call count, retry frequency, timeout rate, and tool-path efficiency alongside final-answer accuracy. Those measures reveal whether a model is becoming more expensive, fragile, or dependent on compensating controls.
• Set model-change gates for agentic workloads Require a controlled migration review before routing production traffic to a new model version. Use repeated runs on representative tasks to confirm that acceptable correctness is not masking unsafe increases in tool usage or execution time.
• Review the agent harness as a governance artifact Document which instructions, schemas, permissions, and retry rules the model depends on, then version them like code. That makes the harness auditable and helps security teams understand where behavioural trust actually comes from.

Key takeaways

• Agent model swaps can preserve correctness while still changing the system’s operational risk profile.
• Latency, retries, and tool calls are governance signals, not secondary performance metrics.
• Model migration decisions should be made on the basis of harnessed behaviour, not benchmark scores alone.

Key terms

• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Operational Drift: Operational drift is the gap that forms when routine administration is delayed, inconsistent, or applied differently across environments. In credentials and identity systems, drift often appears first in logs, storage, or lifecycle tasks before it becomes visible to users or auditors.
• Harness-Bound Trust: Harness-bound trust is the degree of confidence placed in an agent based on the specific wrapper that constrains it. The model may be unchanged, but the controls, permissions, and retry logic in the harness can materially alter how safe or predictable the system is.
• Tool Discipline: Tool discipline is the extent to which an agent uses the smallest useful number of tool actions to complete a task. It is a practical control signal because unnecessary queries, retries, and redundant actions increase latency, cost, and failure surface without improving the answer.

What's in the full article

Arize's full analysis covers the operational detail this post intentionally leaves for the source:

• Task-level benchmark tables showing how each model behaved across the 19 GitHub agent tasks
• Raw and harnessed comparison data for correctness, latency, tool adherence, and failure patterns
• The evaluator design used to score output quality, efficiency, and tool discipline
• Per-model observations that help teams understand where operational drift appears in practice

👉 The full Arize article shows the task tables, evaluator setup, and raw-versus-harnessed comparison data.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle controls. It helps practitioners build the governance discipline needed for agentic systems that behave like non-human identities in production.