Notifications
Clear all
Topic starter
18/08/2026 2:14 pm
TL;DR: Long-horizon production agents break basic LLM judging because trajectories exceed context windows, state changes must be verified in source systems, and rubrics drift as tools and workflows change, according to Judgment Labs. The evaluation problem is shifting from final-answer grading to trajectory inspection, environment verification, and adaptive governance for agent behaviour.
NHIMG editorial — based on content published by Judgment Labs: Agent Judge: Solving Long-Horizon Evals for Production Agents
Questions worth separating out
Q: How should organisations evaluate AI agents that act across multiple systems?
A: They should evaluate the full trajectory, not just the final response.
Q: Why do long-horizon agents expose weaknesses in current governance models?
A: Because they compress many decisions into one run, while traditional governance assumes a small number of reviewable events.
Q: What do security teams get wrong about governing AI agents?
A: They often treat agents like another automation layer instead of governed non-human actors with their own access paths.
Practitioner guidance
- Instrument trajectory-level evidence capture Store tool calls, retrieved records, retries, and state changes in a form that can be queried after the run.
- Add source-of-truth verification steps Require evals to check the systems the agent modified, such as GitHub, CRM, cloud logs, or databases.
- Version evaluation rubrics as control assets Treat the rubric like a living policy document.
What's in the full article
Judgment Labs' full article covers the operational detail this post intentionally leaves for the source:
- The search-and-verification harness design for inspecting long trajectories without manual trace review
- The rubric refinement loop that uses production feedback, judge disagreement, and human labels
- The benchmark methodology behind accuracy, recall, precision, and F1 comparisons across evaluator types
- The production improvement loop that turns eval findings into better prompts, code paths, and workflows
👉 Read Judgment Labs' analysis of long-horizon agent evals and agentic judges →
Long-horizon agent evals: are fixed LLM judges still enough?
Explore further
18/08/2026 3:06 pm
Fixed judges create an evaluation trust gap: when an agent's work spans many turns and multiple systems, a single-pass LLM judge cannot reliably see the full control surface. That is not just a scaling issue, it is a governance failure because the review mechanism is narrower than the actor being reviewed. For IAM and PAM teams, the lesson is that oversight must match the operational reach of the software actor, or blind spots will persist.
A question worth separating out:
Q: How can teams keep agent evaluation rubrics from going stale?
A: By versioning them and updating them from production feedback. Human labels, judge disagreements, recurring false positives, and new tool patterns should feed the next rubric revision. A static rubric quickly drifts away from real behaviour, especially when agents, workflows, and integrations change over time.
👉 Read our full editorial: Agentic judges are becoming essential for long-horizon agent evals
