LLM-as-a-judge fails when eval criteria are underspecified

At a glance

What this is: This is Arize’s guide to building LLM-as-a-Judge evaluators that work in production, with the central finding that the evaluator fails when criteria are vague or detached from traces.

Why it matters: It matters to IAM, NHI, and AI governance teams because the same measurement failures that hide agent mistakes also hide privilege misuse, unsafe tool use, and policy drift in identity-adjacent systems.

👉 Read Arize's guide to building LLM-as-a-Judge evaluators for production

Context

LLM-as-a-Judge is an evaluation pattern, not a control by itself. The article’s core warning is that teams often celebrate a passing score while the underlying trace shows a failed workflow, which means the evaluator is measuring appearance rather than actual task completion. In AI governance terms, that is a measurement integrity problem before it is a model quality problem.

For identity and access programmes, the analogy is direct. If a judge cannot see tool calls, retrieval context, and session history, it cannot reliably tell whether an agent stayed inside its intended boundary, and that leaves governance blind spots around action approval, delegated access, and policy enforcement. The article’s starting point is common in AI teams and increasingly relevant in identity-adjacent operations.

Key questions

Q: How should teams implement LLM-as-judge without trusting the score blindly?

A: Use the judge as one control in a wider evaluation process, not as an oracle. Rotate answer order, decompose the rubric into separate dimensions, and compare results with human-labelled samples. If the judge cannot match human reasoning closely enough, its output should inform testing, not drive production decisions.

Q: Why do LLM judges need fixed labels instead of open-ended scores?

A: Fixed labels make the decision contract explicit and reduce ambiguity. If the judge can choose between resolved, partially_resolved, unresolved, and insufficient_evidence, the team can route work consistently and compare results over time. Open-ended scores look precise, but they usually hide weak criteria and unstable judgment.

Q: What breaks when an agent evaluator cannot see the full trace?

A: The evaluator may reward a plausible final answer while missing the failed tool call, unsupported claim, or skipped step that actually caused the problem. That creates false confidence in the system and weakens incident review, because the team cannot tell whether the issue came from planning, retrieval, or execution.

Q: How do teams know whether an LLM judge is actually working?

A: Compare it with human labels on a representative validation set, then watch whether agreement holds across model versions, domains, and repeated runs. A useful judge is one that supports the decision it drives, whether that is a release gate, a review queue, or a routing choice.

Technical breakdown

When code evaluators are enough for production checks

Deterministic checks belong in code because they test things that do not require interpretation. Schema validity, exact match, regex patterns, latency thresholds, token counts, and approved tool names can be verified faster and more reliably by programmatic logic than by a language model. LLM judges become useful only when meaning matters, such as whether a response is grounded, safe, or appropriate for the user’s intent. The production pattern is hybrid: code for hard constraints, judges for semantic judgment, and trace context for inspection.

Practical implication: route every deterministic control to code first, and reserve the judge for semantic decisions that cannot be reduced to rules.

Why evaluation criteria matter more than the judge model

A judge cannot score what has not been defined. Strong evaluation criteria specify the target, allowed labels, decision rules, evidence inputs, and edge cases, which turns a vague rating request into a measurable contract. Without that structure, 1-to-5 helpfulness scores drift with prompt wording and model choice, making results hard to compare over time. Fixed labels such as resolved, partially_resolved, unresolved, and insufficient_evidence are more stable because they make ambiguity explicit instead of hiding it inside a forced score.

Practical implication: write the rubric before choosing the model, and include a separate uncertain or needs_review state when the evidence is incomplete.

How trajectory evaluation exposes agent failure modes

For agents, the final answer is only one part of the evaluation. A trajectory includes tool choice, tool arguments, redundant steps, recovery behavior, and session outcome, and those elements often explain why the answer succeeded or failed. A workflow can sound correct while skipping the refund tool, selecting the wrong retrieval path, or recovering too late from a bad step. That is why the evaluator has to inspect traces, not just outputs, and why session-level review is essential for multi-step systems.

Practical implication: evaluate the full trace for agent systems, and separate answer quality from tool-use quality so each failure mode can be fixed independently.

Threat narrative

Attacker objective: The objective is not a classic external intrusion but a false sense of system correctness that lets unsupported agent actions evade governance and review.

1. Entry occurs when an AI system produces a plausible response without checking the underlying tool evidence or trace context, so the failure is invisible at the surface.
2. Escalation happens when the judge marks the output as successful even though the agent never executed the required action, which turns missing evidence into false confidence.
3. Impact is a governance blind spot where dashboards and automated routing trust the score rather than the session, allowing unsupported or unsafe agent behaviour to pass review.

NHI Mgmt Group analysis

Judgment without evidence is not governance: LLM-as-a-Judge only improves security decision-making when it is tied to traces, session records, and explicit labels. A score without inspectable evidence creates the same blind spot as any other opaque control, because the organisation cannot tell whether the evaluator measured substance or surface. For identity and AI governance teams, that means evaluation design is part of control design, not a downstream reporting exercise.

Trace-aware evaluation is the right control concept for agent oversight: agents can look successful while skipping the exact tool call or approval step that mattered. That creates a named gap we can call trace-evidence mismatch, where the label says one thing and the execution record says another. In governance terms, this is why task completion, tool selection, and evidence availability must remain separate dimensions. Practitioners should treat that separation as a control boundary, not an implementation detail.

Human calibration remains the benchmark for judge reliability: the article correctly places human review before automation gates, dashboards, and routing. That is especially relevant wherever AI decisions intersect with identity, privileges, or customer impact, because false positives and false negatives both become operational risk. The practical standard is agreement with human labels on the cases that matter, then continuous monitoring for drift.

Boolean certainty is often safer than numeric precision: the piece’s preference for fixed labels reflects a broader governance truth. Continuous scores invite false precision unless the underlying criteria are tightly anchored and repeatedly validated. In identity-adjacent AI systems, discrete labels usually map more cleanly to policy decisions, escalation paths, and review queues, which makes them easier to defend operationally.

What this signals

Trace evidence is becoming the operational boundary for AI governance: once agents can act, the key question is no longer whether an output sounds correct, but whether the underlying session can prove the right tool, policy, and evidence path. That is where evaluation and control design converge. Teams that cannot inspect traces will struggle to distinguish safe automation from plausible failure, which is why trace visibility should be treated as a governance prerequisite, not a monitoring luxury.

Model scoring is not a substitute for policy enforcement: the article shows that a judge can be accurate enough to label outcomes and still miss the control failure that matters. For identity-linked AI workflows, this means scorecards need to sit beside access logs, tool permissions, and approval boundaries. The practical signal for programmes is simple: if you cannot explain why the judge fired, the control is too weak to automate.

Trace-evidence mismatch: this is the failure mode where a label says resolved, but the session record shows no approved action, no supporting evidence, or no valid tool path. That pattern will matter more as AI agents become embedded in customer service, operations, and identity-adjacent workflows. Teams should prepare for judge drift, evidence gaps, and review queues that need human arbitration rather than blind automation.

For practitioners

• Separate deterministic checks from semantic evaluation Use code for schema validation, required fields, latency, and tool-name checks, then use an LLM judge only for meaning-based criteria such as grounding, safety, or task completion.
• Define the rubric before selecting the judge model Write the evaluation target, allowed labels, decision rules, and edge-case handling first, then choose the model that can apply that rubric at acceptable cost and latency.
• Store every judge result beside the trace Attach the label, explanation, prompt version, tool calls, retrieval context, and session history to the same record so reviewers can explain the decision and reproduce it.
• Calibrate against human review on production-like cases Use a labeled validation set drawn from real traces, compare judge output to human labels, and re-test after prompt, model, or retrieval changes.
• Track judge drift as a first-class risk Run canary examples over time and watch for label instability, because an evaluator can drift even when the application being measured stays constant.

Key takeaways

• LLM-as-a-Judge only works as a governance control when the rubric is explicit and the trace is inspectable.
• Production evaluators should use deterministic code for hard checks and LLM judges for meaning-based decisions.
• The real risk is not a weak score, but a confident score that hides missing evidence, skipped steps, or judge drift.

Key terms

• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Evaluation Criteria: The written contract that tells a judge what to measure, what evidence to use, which labels are allowed, and how to handle ambiguity. Good criteria turn subjective judgment into a repeatable process that can be compared across examples, models, and time periods.
• Trace Context: The execution record behind an AI response, including prompts, tool calls, retrieved documents, intermediate steps, and final output. Trace context lets reviewers see why a judge scored something a certain way and whether the score matches the actual system behaviour.
• Insufficient Evidence: A label used when the available trace or output does not provide enough proof for a reliable judgment. It is a governance outcome, not a failure to decide, because forcing a binary answer in the absence of evidence usually makes measurement worse rather than better.

What's in the full article

Arize's full blog post covers the operational detail this post intentionally leaves for the source:

• Concrete code examples for parser, schema, and trace inspection workflows
• Detailed rubric templates for resolved, partially_resolved, and insufficient_evidence labels
• Examples of judge validation against human review across production-like traces
• Phoenix Evals workflow guidance for storing results near traces and experiments

👉 The full Arize guide covers evaluation criteria, trace inspection, and judge calibration examples.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle fundamentals. It is a practical fit for practitioners who need to connect identity controls to broader AI and access risk.