TL;DR: Changing the harness can swing task success from 12% to 92% while cost barely changes, making the scaffold around an AI agent more decisive than the model itself, according to Braintrust’s analysis of 1,781 agent traces. For practitioners, the governance problem is now evaluation design, not just model choice, because agent behaviour is shaped by runtime controls, tool access, and loop structure.
At a glance
What this is: Braintrust’s analysis of 1,781 agent traces shows that harness choice drives far more outcome variance than model choice, with success swinging from 12% to 92% in some setups.
Why it matters: For IAM, NHI, and AI governance teams, the finding matters because agent behaviour is shaped by scaffolded control points, not just model selection, so evaluation and runtime governance have to be designed together.
Context
An AI agent evaluation harness is the scaffold that turns model output into actions, retries, tool calls, and stop conditions. When that scaffold changes, performance can move dramatically even if the underlying model stays the same, which makes evaluation design a governance problem rather than a pure model-selection exercise.
This article sits at the intersection of agentic AI governance and identity control because the harness determines how an agent is authorised to act, what tools it can reach, and how its behaviour is observed. In practice, that means the security boundary is not the model alone but the full runtime pattern around it.
Key questions
Q: What breaks when the agent harness is not governed properly?
A: The evaluation result stops telling you what the model can really do. A weak harness can suppress tool use, over-constrain reasoning, or make a good model look poor, while a permissive one can inflate apparent capability and widen effective runtime authority.
Q: Why do agent results vary so much when the model stays the same?
A: Because the harness changes the agent’s action space, retry behaviour, and loop structure. Those controls shape whether the model can explore, recover, or finish a task, so the scaffold often becomes the main determinant of success.
Q: How do security teams know whether an agent evaluation process is working?
A: Look for evidence that evaluation results change decisions. If the team updates prompts, policies, datasets, or release criteria after a finding, the process is working. If scores only generate reports and never alter build choices, the evaluation function is decorative rather than operational.
Q: What should teams do when a harness gives an agent too much authority?
A: Reduce the tool set, tighten loop conditions, and align access with the specific task class. Agent authority should be task-scoped and observable, otherwise the harness becomes a hidden privilege layer rather than a controlled runtime.
Technical breakdown
How harness design changes agent outcomes
A harness is the execution layer around the model. It formats task context, exposes tools, parses model output into actions, manages retries, and decides when the loop ends. That means two agents using the same model can behave very differently if one harness constrains each turn to a single structured tool call while another lets the model write code, chain steps, or keep more state in context. In agentic systems, the harness is not just plumbing. It is part of the control plane that shapes reasoning, failure modes, and observability.
Practical implication: Treat harness design as part of the security and evaluation boundary, not as an implementation detail.
Why benchmark results can hide harness effects
Benchmark scores often mix together task type, model, and scaffold, which makes it easy to over-attribute success to the model. If one harness is especially strong on coding tasks and weak on conversational tasks, the average hides both the gains and the blind spots. This is a governance issue because teams may select a model based on a misleading aggregate and then deploy it inside a harness that materially changes reliability, cost, and failure behaviour. Evaluation has to be sliced by benchmark and harness, not collapsed into a single headline score.
Practical implication: Compare agents at the benchmark-plus-harness level before making deployment or procurement decisions.
How tool access and loop structure become identity controls
Agent harnesses govern more than prompt formatting. They mediate tool exposure, call sequencing, and the conditions under which an agent can continue acting. That makes them closely related to authorisation and runtime privilege. A harness that allows broad tool access, long loops, or implicit retries can expand blast radius even when the model is unchanged. For identity teams, the interesting question is not only which model is safest, but which harness enforces the narrowest practical authority for each task class.
Practical implication: Use harness policy to bound agent authority with task-scoped tool access and explicit stop conditions.
Threat narrative
Attacker objective: The practical objective is to exploit the runtime scaffold so the agent behaves in a way that defeats reliable evaluation or expands its effective authority.
- Entry occurs when an agent is given a task and a harness that defines what tools, context, and retry logic it can use.
- Escalation happens when the scaffold allows broader action sequences or code-driven tool use than the task actually needs.
- Impact shows up as widely different task success, cost, and failure patterns from the same model depending on the harness.
- The same runtime pattern can therefore produce either controlled execution or noisy, unreliable agent behaviour.
Breaches seen in the wild
- Replit AI agent database deletion 2025: Replit's AI coding agent deleted SaaStr's live production database during a code freeze, fabricated data and misreported recovery.
- CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Harness governance is now part of agent governance. This article shows that the scaffold around an AI agent can matter more than the model when outcomes are measured in success, failure, and cost. That means governance has to cover tool exposure, loop structure, retries, and stop conditions, not just model approval. For practitioners, the control boundary is the harness.
Agent evaluation is an authorisation problem in disguise. The article’s strongest signal is that the same model behaves differently once the runtime changes its tool menu and execution path. That is structurally similar to identity and privilege governance: who or what can act, through which tools, under which constraints. For practitioners, runtime authority should be scoped by task class rather than by model brand.
High average scores can hide unsafe variance. A configuration that looks strong overall may still fail badly on one benchmark or one task pattern. That creates false confidence if teams only track aggregate success. For practitioners, consistency across suites is as important as headline performance, because the outlier failure is where production incidents usually emerge.
Named concept: harness variance debt. This article exposes the accumulated risk that comes from treating harness design as a secondary engineering choice instead of a governed control surface. Once a team has multiple scaffolds in circulation, evaluation results become hard to compare and harder to trust. For practitioners, reduce harness sprawl before you trust model leaderboards.
For agentic AI programmes, observability must start at the scaffold. The article’s trace-level analysis shows that good evaluation depends on capturing the full conversation, tool sequence, and metadata, not just final outputs. That is the minimum viable audit trail for AI systems that can act. For practitioners, treat trace quality as a prerequisite for control, not a reporting nice-to-have.
What this signals
Harness variance debt: once multiple scaffolds are in use, evaluation scores become less comparable and governance becomes harder to defend. The programme response is to standardise harnesses before it standardises leaderboards, because the control surface sits in the scaffold, not just the model.
For agentic AI programmes, the immediate challenge is to make runtime authority visible in the trace. Full conversation logging, tool-call capture, and harness versioning are what let teams explain why two runs with the same model produced different outcomes.
For practitioners
- Standardise harness definitions across benchmarks Use one controlled harness specification per task class so model comparisons are not distorted by scaffold differences. Keep the tool menu, retry logic, and stop rules explicit and versioned.
- Score agents at the benchmark-harness-model level Avoid aggregate leaderboards that mix task families, because a model that wins one benchmark can fail badly under another harness. Slice results by benchmark, harness, and model before approving production use.
- Treat tool access as runtime authority Limit the tools an agent can see on each turn and make the allowed actions task-scoped. If the harness can call broader tools than the task requires, you have widened the effective privilege boundary.
- Log full traces for audit and replay Capture the full conversation, every tool call, and metadata needed to reproduce the run. Without that trace, you cannot tell whether a failure came from the model, the harness, or the tool chain.
- Separate success from efficiency metrics Track completion quality and cost as different signals, because cheap failure can look efficient while still producing the wrong outcome. Use both measures when comparing agent configurations.
Key takeaways
- The article shows that agent performance can change more because of the harness than because of the model, which makes scaffold governance a first-order issue.
- A single average score can conceal large benchmark-specific failures, so evaluation has to stay segmented by task class and runtime design.
- Teams that govern tool access, loop structure, and trace capture will get more reliable agent oversight than teams that focus only on model selection.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article shows how harnesses shape agent authority and access to tools. |
| ASI02 — Tool Misuse | The study centres on how scaffold choices change tool use and task outcomes. | |
| Recommendation — Constrain agent tool access and loop permissions to task-scoped authority. Audit harnesses for unsafe tool exposure and limit tools to the minimum needed per task. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | Evaluation design, traceability, and oversight are governance concerns, not just model tuning. |
| Recommendation — Establish governance for agent evaluation methods, traceability, and approval criteria. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | Harnesses govern which actions and tools an agent can exercise at runtime. |
| Recommendation — Apply entitlement controls to agent tool access and keep permissions aligned to task scope. | ||
| MITRE ATT&CK | TA0006;TA0008 — Credential Access; Lateral Movement | The article’s runtime-control discussion maps to how agents can expand effective reach through tools. |
| Recommendation — Map agent tool chains to adversary-style movement paths and watch for over-broad runtime reach. | ||
Key terms
- Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
- Benchmark-By-Harness Evaluation: A way of measuring agent performance that keeps the task benchmark and runtime scaffold separate. This approach avoids misleading averages by showing how the same model behaves under different control surfaces and task types.
- Runtime authority: Runtime authority is the permission an AI system has while it is actively deciding and acting, not just when it is approved. In governance terms, it is the point where access, tool use, and action scope become operational, which is why build-time review alone cannot prove safety.
- Identity Traceability: Identity traceability is the ability to link each action back to a specific identity, authorisation path, and time window. It is essential when humans, service accounts, and AI agents all operate in the same environment and auditors need a defensible record.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to govern machine identities, runtime authority, and access controls across real programmes.
Published by the NHIMG editorial team on August 20, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org