Notifications
Clear all
Topic starter
20/08/2026 5:20 pm
TL;DR: Changing the harness can swing task success from 12% to 92% while cost barely changes, making the scaffold around an AI agent more decisive than the model itself, according to Braintrust’s analysis of 1,781 agent traces. For practitioners, the governance problem is now evaluation design, not just model choice, because agent behaviour is shaped by runtime controls, tool access, and loop structure.
NHIMG editorial — based on content published by Braintrust: Using Braintrust to eval agentic setups from large-scale Hugging Face data
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
- The harness matters about 7× more than the model in the reported agent evals, with success swinging from 12% to 92% when the scaffold changes.
- Braintrust analysed 1,781 agent traces across six benchmarks and about 49,000 child spans.
Questions worth separating out
Q: How should organisations evaluate AI agents without relying on one average success score?
A: Break evaluation into benchmark families, harness types, and workflow classes.
Q: Why do agent harnesses matter so much to security and governance outcomes?
A: Because the harness controls tool exposure, retry behaviour, stopping conditions, and how model output becomes action.
Q: What do security teams get wrong about judging AI agent reliability?
A: They often trust the model label more than the runtime design.
Practitioner guidance
- Treat the harness as a governed control point Document the agent scaffold, tool exposure, retry policy, and stop conditions alongside the model approval record.
- Score agents by benchmark and workflow class Avoid a single aggregate success rate for all agents.
- Persist trace metadata for audit and replay Store stable span IDs, per-call tool metadata, and score outputs so the same run can be queried, re-scored, and reviewed without losing lineage.
What's in the full article
Braintrust's full article covers the operational detail this post intentionally leaves for the source:
- The exact import workflow for turning Hugging Face Parquet shards into queryable Braintrust spans.
- The LLM-as-judge grading prompt used to assign success, confidence, and reasoning to each run.
- The regression and slice analysis used to isolate harness effects from model effects across benchmark classes.
- The per-benchmark result tables that show how success, duration, and token use vary by harness and model.
👉 Read Braintrust's analysis of agent eval harness performance and trace scoring →
Agent eval harnesses: why the scaffold matters more than the model?
Explore further
20/08/2026 6:11 pm
Harness discipline is becoming a control problem, not a benchmarking preference. When the same model swings from poor to excellent depending on the scaffold, the harness is functioning as an access and execution policy layer. That means agent governance cannot stop at model approval; it has to evaluate tool exposure, retry logic, and loop termination as part of the control design. For security teams, the lesson is to govern the runtime boundary, not just the model catalogue.
A question worth separating out:
Q: How should security teams govern AI agents that choose tools at runtime?
A: Security teams should treat runtime tool choice as a governed access event, not a normal application call. That means task-scoped credentials, explicit approval boundaries for sensitive actions, and logs that record both the tool selected and the identity used. If the agent can change its plan, the control model must be able to change with it.
👉 Read our full editorial: Harness choice drives most agent eval variance, not model selection
