LLM-as-a-judge fails when eval criteria are underspecified

At a glance

What this is: This guide explains why LLM-as-a-judge only works in production when the evaluation target, labels, and evidence rules are tightly defined.

Why it matters: IAM, NHI, and agentic AI teams need reliable evaluation controls because automated scoring without clear criteria can hide tool-use failures, weak task completion, and broken routing decisions.

Context

LLM-as-a-judge is an evaluation pattern, not a guarantee of truth. The core governance gap is that a model can produce a label while the system under review still fails to complete the real task, especially when tool use, context, or evidence are missing.

In practice, the problem is not simply model quality but evaluation design. When teams score outputs without clear criteria, anchored labels, and trace-level evidence, they risk treating a fluent assessment as operational validation for AI agents and other automated systems.

Key questions

Q: What should teams do when an LLM judge cannot see enough evidence to score an output?

A: Do not force a binary pass or fail when the trace is incomplete. Add a separate insufficient evidence or needs review state, then route those cases to human review or richer logging. That keeps the metric honest and prevents uncertain examples from being mistaken for validated automation.

Q: Why do vague judge criteria create risk in production AI systems?

A: Vague criteria let the judge invent its own standard, which can drift across prompts and models. That creates false confidence because a fluent label may not reflect task completion, grounding, or policy compliance. Clear decision rules turn the score into a controlled measurement instead of a subjective opinion.

Q: How do teams know whether an LLM judge is actually working?

A: Compare it with human labels on a representative validation set, then watch whether agreement holds across model versions, domains, and repeated runs. A useful judge is one that supports the decision it drives, whether that is a release gate, a review queue, or a routing choice.

Q: What is the difference between code-based scorers and LLM-as-a-judge scorers for agent evaluations?

A: Code-based scorers are best for binary, numeric, or structured checks such as exact match, route selection, or step limits. LLM-as-a-judge scorers are better when the question is subjective, contextual, or multi-factor, such as whether an answer feels coherent or helpful. Most mature eval programmes use both, because they measure different dimensions of agent performance.

Technical breakdown

Why underspecified criteria produce false confidence

An LLM judge can only apply the criteria it is given. If “helpful” or “correct” is not broken into observable evidence, edge cases, and decision rules, the model will infer its own standard and that standard can drift across prompts, datasets, and model versions. This is especially fragile when the judged system uses tools, retrieval, or multi-step reasoning, because the final text may look right even when the underlying workflow failed. The technical problem is not that the judge cannot score, but that it cannot distinguish plausible language from verified task completion unless the evaluation contract makes that distinction explicit.

Practical implication: Define the decision, evidence, and label set before you deploy the judge.

Why boolean and categorical labels beat open-ended scores

A judge output works best when it matches the decision you actually need to make. Boolean labels are strong for policy checks and gates, while categorical labels are better when the system can land in distinct states such as resolved, partially resolved, or insufficient evidence. Open-ended scores often look precise but hide ambiguity, especially when reviewers cannot tell whether a 4 out of 5 means slight weakness or a near-failure. For agent evaluation, fixed labels also make disagreement with human reviewers easier to inspect and calibrate, which is essential when the score will drive dashboards, routing, or release decisions.

Practical implication: Use the simplest label type that can support the operational decision.

Why trace context is part of the evaluator

A judge result is not useful on its own if the team cannot inspect the trace behind it. The real control point is the execution record: retrieved context, prompt version, tool calls, intermediate steps, and final response. Without that evidence, the judge can tell you something failed, but not where the failure happened or whether the judge itself was wrong. For agentic systems, this matters even more because the right answer may depend on tool choice, argument quality, redundant steps, or recovery behaviour rather than wording alone. Evaluation therefore has to sit next to observability, not outside it.

Practical implication: Attach judge outputs to traces so reviewers can validate the label against the underlying execution.

Threat narrative

Attacker objective: The operational objective is not theft but the creation of misleading assurance that causes teams to trust an invalid automated evaluation.

1. Entry occurs when a team adopts LLM-as-a-judge to replace manual review without first defining the evaluation target and evidence rules.
2. Credential or privilege-like misuse appears when the judge is trusted to authorise dashboards, routing, or release decisions despite underspecified labels and no human calibration.
3. Impact follows when a passing score masks a workflow that never actually completed the intended task, allowing false confidence to propagate into production operations.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Underspecified evaluation criteria create governance debt: the failure is not the judge model, but the absence of a defensible decision standard. Once teams let one score stand in for correctness, grounding, task completion, and safety all at once, they lose the ability to audit what the system actually proved. The practical conclusion is that evaluation policy must be explicit before automation is allowed to act on it.

Trace-linked evaluation is the real control surface: the judge only becomes reliable when its output can be inspected against prompts, tool calls, retrieval results, and session history. That makes the evaluation layer a governance object, not just a monitoring feature. For agentic AI programmes, the control question is whether every score can be explained and challenged in context.

Fixed labels are a better fit for production than open-ended grading: binary or categorical decisions are easier to calibrate, easier to trend, and less likely to mask ambiguity. This matters for NHI and agent workflows because task completion is often more important than stylistic quality. The practitioner takeaway is to design for reviewability, not for numerical elegance.

LLM-as-a-judge exposes the same trust problem seen in other delegated systems: a downstream decision is only as good as the evidence boundary around it. In identity governance terms, this is a form of authority without proof, where the system is allowed to decide but not forced to justify the decision. Teams should treat judge output as a claim that needs evidence, not as an answer by default.

Evaluator drift should be treated as a first-class risk: a judge can change behaviour just as the application does, especially when prompts, models, or thresholds are updated. That makes continuous calibration part of production governance, not a one-time tuning exercise. The field should expect judge maintenance to become a standing operational duty.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Evaluation debt now sits alongside governance debt in AI programmes: teams that let a judge score without a rubric, trace context, or evidence threshold will eventually confuse label production with system assurance. The control point is not the model used as judge, but the policy that constrains what a score can mean.

Agent evaluation needs to move closer to identity and execution context: when tool use, session state, and task outcome are part of the workflow, the judge has to see more than the final answer. That is why trace-linked review matters for agentic AI and for any system where delegated action changes the risk profile.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.

For practitioners

• Define the evaluation contract before deploying a judge Specify the target, allowed labels, decision rules, evidence requirements, and failure examples so the judge is scoring one thing only.
• Separate deterministic checks from semantic checks Use code for schema validity, required fields, exact matches, and tool presence, then reserve the judge for meaning-based judgments such as grounding, completion, and safety.
• Calibrate against human labels on real traces Run the judge on annotated production examples, inspect disagreements, and tighten the rubric before you let scores influence routing, dashboards, or release gates.
• Add an evidence-absent state when the trace is incomplete Use a third label such as insufficient evidence or needs review whenever the judge cannot see enough context to make a reliable call.
• Track judge stability over time Re-test fixed canary examples whenever the prompt, model, retrieval layer, or scoring thresholds change so drift is visible before it affects operations.

Key takeaways

• Underspecified judge criteria can make an AI system look successful while the underlying task still fails.
• Production evaluation works best when deterministic checks, semantic judging, and human calibration are kept separate.
• Trace context and stable labels are the difference between a useful evaluator and a misleading dashboard score.

Key terms

• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Evaluation Criteria: The written contract that tells a judge what to measure, what evidence to use, which labels are allowed, and how to handle ambiguity. Good criteria turn subjective judgment into a repeatable process that can be compared across examples, models, and time periods.
• Trace Context: The execution record behind an AI response, including prompts, tool calls, retrieved documents, intermediate steps, and final output. Trace context lets reviewers see why a judge scored something a certain way and whether the score matches the actual system behaviour.
• Insufficient Evidence: A label used when the available trace or output does not provide enough proof for a reliable judgment. It is a governance outcome, not a failure to decide, because forcing a binary answer in the absence of evidence usually makes measurement worse rather than better.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for teams that need to govern machine identities, delegated access, and AI-driven workflows with more discipline.