Topic starter
20/08/2026 5:20 pm
TL;DR: Changing the harness can swing task success from 12% to 92% while cost barely changes, making the scaffold around an AI agent more decisive than the model itself, according to Braintrust’s analysis of 1,781 agent traces. For practitioners, the governance problem is now evaluation design, not just model choice, because agent behaviour is shaped by runtime controls, tool access, and loop structure.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “Using Braintrust to eval agentic setups from large-scale Hugging Face data”.
Key questions
Q: What breaks when the agent harness is not governed properly?
A: The evaluation result stops telling you what the model can really do.
Q: Why do agent results vary so much when the model stays the same?
A: Because the harness changes the agent’s action space, retry behaviour, and loop structure.
Q: How do security teams know whether an agent evaluation process is working?
A: Look for evidence that evaluation results change decisions.
Practitioner guidance
- Standardise harness definitions across benchmarks Use one controlled harness specification per task class so model comparisons are not distorted by scaffold differences.
- Score agents at the benchmark-harness-model level Avoid aggregate leaderboards that mix task families, because a model that wins one benchmark can fail badly under another harness.
- Treat tool access as runtime authority Limit the tools an agent can see on each turn and make the allowed actions task-scoped.
Bottom line: The article shows that agent performance can change more because of the harness than because of the model, which makes scaffold governance a first-order issue.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:11 pm
Harness governance is now part of agent governance. This article shows that the scaffold around an AI agent can matter more than the model when outcomes are measured in success, failure, and cost. That means governance has to cover tool exposure, loop structure, retries, and stop conditions, not just model approval. For practitioners, the control boundary is the harness.
A question worth separating out:
Q: What should teams do when a harness gives an agent too much authority?
A: Reduce the tool set, tighten loop conditions, and align access with the specific task class. Agent authority should be task-scoped and observable, otherwise the harness becomes a hidden privilege layer rather than a controlled runtime.
👉 Read our full editorial: Harness choice drives most agent eval variance, not model selection