Topic starter
18/08/2026 2:14 pm
TL;DR: Long-horizon production agents break basic LLM judging because trajectories exceed context windows, state changes must be verified in source systems, and rubrics drift as tools and workflows change, according to Judgment Labs. The evaluation problem is shifting from final-answer grading to trajectory inspection, environment verification, and adaptive governance for agent behaviour.
Editorial analysis by NHI Mgmt Group, based on content published by Judgment Labs: “Agent Judge: Solving Long-Horizon Evals for Production Agents”.
Key questions
Q: What breaks when fixed LLM judges are used for long-horizon agents?
A: Fixed LLM judges break when the important evidence is spread across many tool calls, state transitions, and retries rather than concentrated in one final answer.
Q: Why do stateful agent actions need environment verification?
A: Because an agent can describe the right outcome without actually changing the system of record.
Q: How can teams keep agent evaluation rubrics from going stale?
A: By versioning them and updating them from production feedback.
Practitioner guidance
- Adopt trajectory-level evaluation for production agents Require evaluators to inspect the full sequence of tool calls, retries, retrieved evidence, and state changes instead of scoring only the final response.
- Verify agent claims against source systems Compare claimed actions with durable evidence in systems such as CRM, GitHub, databases, or workflow logs before accepting the eval result.
- Version and refresh evaluation rubrics Treat rubrics as living controls that are updated from human feedback, judge disagreement, and production outcomes when agent behaviour shifts.
Bottom line: Long-horizon agent evals fail when they stop at the final answer and ignore the evidence chain behind the action.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:06 pm
Long-horizon agent evaluation is becoming an identity and governance problem, not just a model-scoring problem. Once an agent can act across CRM, cloud, code, and scheduling systems, the judge is no longer evaluating text in isolation. It is evaluating whether delegated actions stayed within intended authority, which makes the evaluation layer part of the access-control and accountability stack.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What is the difference between trajectory review and final-answer grading?
A: Final-answer grading checks the endpoint, while trajectory review checks the sequence of evidence, tool use, and state changes that produced it. For production agents, trajectory review is stronger because failures often happen earlier in the run and are invisible in the last message alone.
👉 Read our full editorial: Agentic judges are becoming essential for long-horizon agent evals