Topic starter
20/08/2026 5:20 pm
TL;DR: Success can swing from 12% to 92% when the harness changes, while token cost barely moves, according to Braintrust. Agent oversight has to account for tool access, loop structure, and trace capture, or the score will misstate what the model can do.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “Using Braintrust to eval agentic setups from large-scale Hugging Face data”.
Key questions
Q: Why do agent results vary so much when the model stays the same?
A: Because the harness changes the agent’s action space, retry behaviour, and loop structure.
Q: How do security teams know whether an agent evaluation process is working?
A: Look for evidence that evaluation results change decisions.
Q: What breaks when the agent harness is not governed properly?
A: The evaluation result stops telling you what the model can really do.
Practitioner guidance
- Standardise harness definitions across benchmarks Use one controlled harness specification per task class so model comparisons are not distorted by scaffold differences.
- Score agents at the benchmark-harness-model level Avoid aggregate leaderboards that mix task families, because a model that wins one benchmark can fail badly under another harness.
- Treat tool access as runtime authority Limit the tools an agent can see on each turn and make the allowed actions task-scoped.
Bottom line: The article shows that agent performance can change more because of the harness than because of the model, which makes scaffold governance a first-order issue.
Glossary terms: Agent Harness, Benchmark-By-Harness Evaluation, Runtime authority, Identity Traceability
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 2 days ago by NHI Mgmt Group
20/08/2026 6:11 pm
Harness governance is now part of agent governance. This article shows that the scaffold around an AI agent can matter more than the model when outcomes are measured in success, failure, and cost. That means governance has to cover tool exposure, loop structure, retries, and stop conditions, not just model approval. For practitioners, the control boundary is the harness.
A question worth separating out:
Q: What should teams do when a harness gives an agent too much authority?
A: Reduce the tool set, tighten loop conditions, and align access with the specific task class. Agent authority should be task-scoped and observable, otherwise the harness becomes a hidden privilege layer rather than a controlled runtime.
👉 Read our full editorial: Harness choice drives most agent eval variance, not model selection