Topic starter
06/11/2026 6:55 pm
TL;DR: Searching agent traces at scale becomes practical when eval results are reused as searchable columns, according to Judgment Labs's analysis of production traffic. That changes trace investigation from keyword hunting to verifying already-scored evidence, which is faster and more dependable for operators.
Editorial analysis by NHI Mgmt Group, based on content published by Judgment Labs: “Searching a billion tokens of agent data”.
Key questions
Q: Why does keyword search miss important agent failures?
A: Keyword search misses important agent failures because many failures are behavioural, not lexical.
Q: How should teams validate evals before using them as indexes?
A: Teams should validate evals as indexes by checking that the rubric actually retrieves the failures they care about across a labelled test set.
Q: How do evals improve agent trace search?
A: Evals improve agent trace search by converting subjective behavior into structured labels that can be filtered like database columns.
Practitioner guidance
- Define behavioral labels before production rollout Choose a small set of searchable outcomes such as completed, failed, recovered and unresolved, and make sure each one has a clear judging rubric before the agent goes live.
- Store eval results as first-class telemetry Write eval labels, reasons and supporting span IDs into the same system as traces so investigators can filter on behavior and then open the evidence behind each result.
- Calibrate judges against hand-labeled examples Test every eval against a small verified corpus and review false positives and false negatives before using the output as a search index.
Bottom line: Agent trace investigations become much more practical when teams reuse eval outputs as structured search filters rather than reading raw spans one by one.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
06/11/2026 6:56 pm
Evals are becoming the missing control plane for agent telemetry: raw traces alone are too unstructured to support dependable investigation at production scale. The important shift is not better keyword search but pre-classified evidence that can be filtered, audited and rechecked. For agentic programmes, that turns monitoring data into a governance asset rather than a forensic afterthought.
A question worth separating out:
Q: What is the difference between vector search and eval-based search?
A: Vector search retrieves similar text or embeddings by proximity in semantic space. Eval-based search retrieves examples by matching scored behaviours, outcomes, or failure patterns against a rubric, so it can find agent mistakes that look different on the surface but fail the same control, tool-use, or policy condition.
👉 Read our full editorial: Evals-as-indexes make agent trace search practical at production scale