Notifications
Clear all
Topic starter
20/08/2026 5:40 pm
TL;DR: Harness reliability depends on model behaviour and task shape, according to Arize, with GPT-4o still producing two false finishes in a 117-run benchmark even after broader model convergence. The practical lesson is that agent loops need adaptive exit checks, configurable tunings, and multi-model evals because static assumptions about narration, tool use, and completion can fail quietly.
NHIMG editorial — based on content published by Arize: Agent harnesses have an expiration date
By the numbers:
- The Adaptive Harness used 6% more tokens than the Implicit Harness while matching the Explicit Harness on correctness.
Questions worth separating out
Q: How should teams prevent AI agents from exiting before work is complete?
A: Use explicit or adaptive closure checks instead of assuming a text-only response means the task is done.
Q: Why do AI agent harnesses break when model behaviour changes?
A: Harnesses often encode unstated assumptions about when narration ends and tool use begins.
Q: What do security teams get wrong about agent completion logic?
A: They often treat completion as a language problem when it is really a control problem.
Practitioner guidance
- Make exit conditions explicit Require a dedicated completion signal or finish tool before an agent can close a task, and validate that the harness checks the final state before accepting it as complete.
- Test harnesses against multiple model checkpoints Run the same eval suite across at least two model families and more than one checkpoint so that narrate-then-act, silent exits, and formatting changes surface early.
- Promote hidden tunings into testable configuration Move iteration caps, retry budgets, and parsing heuristics out of code defaults and into controlled settings that your eval suite can exercise under load.
What's in the full article
Arize's full analysis covers the implementation detail this post intentionally leaves for the source:
- The exact pseudo-code patterns used for implicit, explicit, and adaptive finish logic in the benchmark runner
- The 117-run test structure across Claude Sonnet 4, GPT-4o, and Gemma 4 31B, including task categories and outcome scoring
- The specific detection heuristic used to identify narrate-then-act responses in trailing output
- The token-cost comparison and why the adaptive design balanced observability with lower overhead than the explicit version
👉 Read Arize's analysis of AI agent harness drift and finish logic →
AI agent harnesses and model drift: are your exit conditions safe?
Explore further
20/08/2026 6:24 pm
Adaptive closure is becoming a core control for agent governance. The article shows that a runtime can look correct while silently exiting before work is complete, which is a control failure rather than a model error. For agentic AI, task completion is equivalent to privilege release, so closure checks must be treated as a governance boundary. Practitioners should read this as a signal to formalise finish logic, not to trust a clean transcript.
A question worth separating out:
Q: How do you know if an AI agent harness is actually reliable?
A: You need cross-model and cross-task evals, not a single happy-path benchmark. Reliability shows up when the same loop survives model drift, longer task chains, and different output styles while still catching false finishes and enforcing the intended end state.
👉 Read our full editorial: AI agent harnesses need adaptive finish logic to survive model drift
