Harness choice drives most agent eval variance, not model selection

At a glance

What this is: This analysis uses 1,781 real agent traces to show that harness design can outweigh model choice in agent success, with performance swings as high as 81 percentage points.

Why it matters: IAM and AI governance teams need to treat agent harnesses as part of the control surface, because tool routing, loop design, and scoring can change both risk and reliability.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
• The harness matters about 7× more than the model in the reported agent evals, with success swinging from 12% to 92% when the scaffold changes.
• Braintrust analysed 1,781 agent traces across six benchmarks and about 49,000 child spans.

👉 Read Braintrust's analysis of agent eval harness performance and trace scoring

Context

Agent evaluation is not just a model comparison exercise. In production, the harness around an AI agent determines how tasks are framed, how tools are exposed, how retries happen, and how outcomes are scored, which means the control plane around the model can dominate apparent performance.

That matters for identity and access governance because agent harnesses effectively govern non-human behaviour: which tools can be called, when credentials are used, and how much visibility security teams retain. The article’s central point is that agent reliability is highly dependent on the scaffold, not just the model, and that is typical of real-world agent deployments.

Key questions

Q: How should organisations evaluate AI agents without relying on one average success score?

A: Break evaluation into benchmark families, harness types, and workflow classes. A single average can hide severe failure variance across tasks, so teams need score slices that show where the agent is reliable, where it thrashes, and where the scaffold itself is driving the result. Governance decisions should follow the narrowest meaningful slice.

Q: Why do agent harnesses matter so much to security and governance outcomes?

A: Because the harness controls tool exposure, retry behaviour, stopping conditions, and how model output becomes action. That makes it part of the execution policy, not a neutral wrapper. If the scaffold is loose, the agent may overreach; if it is too tight, it may fail silently or look cheaper by giving up early.

Q: What do security teams get wrong about judging AI agent reliability?

A: They often trust the model label more than the runtime design. But the same model can succeed or fail dramatically depending on the harness and task type, so model selection alone is not a meaningful governance control. Teams should test the full agent stack, including trace quality and tool boundaries.

Q: How should security teams govern AI agents that choose tools at runtime?

A: Security teams should treat runtime tool choice as a governed access event, not a normal application call. That means task-scoped credentials, explicit approval boundaries for sensitive actions, and logs that record both the tool selected and the identity used. If the agent can change its plan, the control model must be able to change with it.

Technical breakdown

Why harness architecture changes agent outcomes so sharply

A harness is the orchestration layer that turns model output into action. It manages context, tool exposure, retries, and stopping conditions, so two agents using the same model can behave very differently if the scaffold differs. This is why a structured JSON function-calling loop, a code-writing agent loop, and a thin wrapper can produce radically different success rates on the same task set. In practice, the harness is part of the runtime policy boundary, not a cosmetic implementation detail.

Practical implication: evaluate harness behaviour as a controlled variable before blaming the model.

Why benchmark-specific agent governance is unavoidable

The article shows that coding, research, and customer-support benchmarks reward different failure modes and different control patterns. A harness that encourages persistence may help software tasks but hurt conversational tasks by creating unnecessary thrash, while a tighter harness can reduce variability but also suppress useful exploration. For security and AI governance teams, this means a single agent policy cannot be assumed to fit every workflow. The governing question is whether the agent’s access model matches the task’s operational shape.

Practical implication: define task-specific harness policies instead of using one agent template everywhere.

How traceability turns agent evaluation into an audit problem

The dataset had traces but no ground-truth verdicts, so Braintrust had to build a judge-based scoring layer and write the results back into structured metadata. That is the real lesson for enterprise use: without queryable traces, stable span IDs, and auditable scoring fields, agent evaluation becomes anecdotal. In identity terms, the agent’s actions are only governable if the surrounding telemetry preserves who did what, with which tools, and under which configuration.

Practical implication: make trace structure and scoring metadata first-class governance assets.

NHI Mgmt Group analysis

Harness discipline is becoming a control problem, not a benchmarking preference. When the same model swings from poor to excellent depending on the scaffold, the harness is functioning as an access and execution policy layer. That means agent governance cannot stop at model approval; it has to evaluate tool exposure, retry logic, and loop termination as part of the control design. For security teams, the lesson is to govern the runtime boundary, not just the model catalogue.

Agent evaluation is converging with identity governance because the agent is the actor. In production, an agent is a non-human runtime entity that consumes tools, credentials, and permissions through its harness. That places it squarely inside NHI governance concerns, even when the article is framed as eval tooling rather than IAM. The practical conclusion is that agent identity, not just model accuracy, must be visible in policy and telemetry.

Trace data without structure creates governance debt. The article shows that raw logs are not enough when verdicts, metadata, and scores are missing or inconsistent. Enterprises that cannot query agent traces by model, harness, and task will struggle to prove control effectiveness or investigate failures. The named concept here is agent observability debt: the gap between recorded activity and decision-grade telemetry that prevents reliable oversight.

Success metrics can mislead when task shape is ignored. A high average score can hide catastrophic variance across specific workflows, which is a familiar control failure in identity and access programmes too. This is why teams should resist treating aggregate agent performance as sufficient evidence of safety. The correct standard is consistent behaviour under the exact workflow and access conditions the agent will face.

Open-weight model adoption now depends on governance readiness, not just capability. The article shows that open-weight models can perform strongly in production-style tasks, but only when the harness and scoring model are aligned to the workload. That means the decision is no longer whether open-weight models can work, but whether the organisation can govern them with sufficient auditability and least-privilege tooling.

What this signals

Agent evaluation is becoming a proxy for identity governance maturity. If an organisation cannot explain why one harness produces safer behaviour than another, it does not yet have control over the non-human actor it is deploying. That is why the combination of runtime traceability, tool scoping, and policy enforcement will matter more than leaderboard-style model comparisons.

The most useful next step for practitioners is to separate model capability from operational permission. An agent can be technically competent and still be ungoverned if its credential scope, tool set, or session boundaries are not bounded in policy and telemetry. For identity teams, this is a direct extension of least privilege into AI execution.

For practitioners

• Treat the harness as a governed control point Document the agent scaffold, tool exposure, retry policy, and stop conditions alongside the model approval record. This lets security, IAM, and AI governance teams review the runtime shape that actually drives outcomes.
• Score agents by benchmark and workflow class Avoid a single aggregate success rate for all agents. Break evaluation into task families so that coding, support, and research workflows are judged against the control patterns they actually need.
• Persist trace metadata for audit and replay Store stable span IDs, per-call tool metadata, and score outputs so the same run can be queried, re-scored, and reviewed without losing lineage.
• Separate performance testing from access testing Validate whether an agent can complete a task and whether it should have had the tools and credentials required to do so. Those are different governance questions and should not share the same approval gate.

Key takeaways

• Agent performance is not just a model question, because the harness can dominate success or failure.
• Trace structure, scoring metadata, and task-specific evaluation slices are now essential governance inputs.
• AI agents behave like non-human identities in practice, so their runtime access model must be governed like any other privileged actor.

Key terms

• Agent Harness: The agent harness is the runtime layer that wraps a model and turns it into an acting system. It usually includes the loop, tools, context handling, permissions, hooks, and logs. In security terms, it is often the real place where privilege sits and where identity evidence must be governed.
• Judge-Based Scoring: Judge-based scoring uses an LLM or rubric-driven evaluator to estimate whether a task succeeded when ground truth is unavailable. It is useful for trace analysis, but it introduces its own reliability limits and must be validated against benchmark shape and visible evidence.
• Trace Telemetry: Trace telemetry is structured runtime data that records agent tool calls, inputs, outputs, and timing. For identity security, it becomes evidence of who or what touched a secret, which paths were accessed, and whether behaviour crossed an approved boundary.

What's in the full article

Braintrust's full article covers the operational detail this post intentionally leaves for the source:

• The exact import workflow for turning Hugging Face Parquet shards into queryable Braintrust spans.
• The LLM-as-judge grading prompt used to assign success, confidence, and reasoning to each run.
• The regression and slice analysis used to isolate harness effects from model effects across benchmark classes.
• The per-benchmark result tables that show how success, duration, and token use vary by harness and model.

👉 Braintrust's full post includes the import workflow, scoring method, and benchmark tables behind the findings

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps practitioners translate runtime access risk into controls that fit identity and security programmes.