Notifications
Clear all
Topic starter
20/08/2026 5:33 pm
TL;DR: Correctness clustered from 79.6% to 85.1% across seven models under a fixed GitHub agent harness, but latency, tool calls, retries, and timeout risk varied much more than final-answer quality, according to Arize. The lesson is that model swaps behave like product migrations, because operational behavior can change even when the answer still looks right.
NHIMG editorial — based on content published by Arize: What we learned testing 7 models under the same agent harness
By the numbers:
- The harnessed runs clustered between 79.6% and 85.1% correctness across seven model targets.
- The sweep covered 1,330 attempted examples across 19 tasks, seven models, and ten runs per model.
Questions worth separating out
Q: How should teams evaluate model swaps in agentic systems?
A: Treat model swaps like production migrations.
Q: Why do agents need governance beyond final-answer accuracy?
A: Because the answer is only the end state.
Q: What do security teams get wrong about agent harnesses?
A: They often treat the harness as a testing convenience rather than part of the control plane.
Practitioner guidance
- Freeze the harness before swapping models Keep tools, prompts, fixtures, scoring, and retry policy constant so model-to-model comparison stays clean.
- Score operational behaviour, not just correctness Track latency, tool-call count, retry frequency, timeout rate, and tool-path efficiency alongside final-answer accuracy.
- Set model-change gates for agentic workloads Require a controlled migration review before routing production traffic to a new model version.
What's in the full article
Arize's full analysis covers the operational detail this post intentionally leaves for the source:
- Task-level benchmark tables showing how each model behaved across the 19 GitHub agent tasks
- Raw and harnessed comparison data for correctness, latency, tool adherence, and failure patterns
- The evaluator design used to score output quality, efficiency, and tool discipline
- Per-model observations that help teams understand where operational drift appears in practice
👉 Read Arize's analysis of model swaps under a shared agent harness →
Agent harness model swaps: what changes beyond final accuracy?
Explore further
20/08/2026 6:19 pm
Model swaps in agent systems are migration events, not configuration changes. The article shows that the API surface may stay nearly constant while the underlying execution profile shifts materially. That is exactly how governance blind spots form: teams compare the model name, not the runtime path. In identity terms, the delegated work performed by the agent can change even when the request looks identical, which means access governance must follow the execution path, not the branding of the model.
A question worth separating out:
Q: Should organisations trust a model that performs well in a benchmark?
A: Only if the benchmark mirrors the production harness and task mix. Benchmark scores can hide whether the model behaves safely under real tool pressure, longer sessions, or different permission boundaries. Trust should be earned in the same execution conditions the system will face in production.
👉 Read our full editorial: Model swaps in agent harnesses behave more like migrations
