LLM evals in CI expose the control gap in agent governance

At a glance

What this is: Arize argues that LLM evals should be treated as CI tests, with hard assertions used for invariant failures and scored checks used for quality signals.

Why it matters: This matters because teams governing LLM applications and agents need a repeatable way to separate control-breaking failures from acceptable model variance across AI and identity-adjacent workflows.

Context

LLM evals create a governance problem when teams treat every quality issue like a release blocker or, conversely, let critical failures hide inside vague score trends. The article frames evals as a testing discipline because LLM behaviour is non-deterministic, expensive to run, and often only understandable when traces are preserved.

For IAM, NHI, and agentic AI programmes, the deeper issue is control design: some behaviours are hard invariants, while others are acceptable within a tolerance band. That distinction matters when an AI system is acting on behalf of users or services, because governance has to decide what must never happen versus what can be monitored over time.

Key questions

Q: How should teams decide which LLM evals belong in CI gates?

A: Put only hard invariants in CI gates. If a behaviour has exactly one acceptable outcome, such as a refusal rule or a permission boundary, assert on it. If the behaviour is qualitative, score it and track the trend instead of failing every run. That keeps CI focused on control breaks rather than normal model variance.

Q: Why do LLM and agent evaluations need repeated runs instead of one pass?

A: Because a single successful run does not prove stability when the underlying system is non-deterministic. Repetition reveals whether the behaviour is consistent enough to trust across prompt changes, model updates, and context shifts. Without repeated checks, teams may mistake a lucky run for a reliable control.

Q: What breaks when AI quality is only checked at the prompt level?

A: Teams miss failures that happen outside the prompt text itself. A prompt may look fine while the model selects the wrong tool, retrieves bad context, or completes an unsafe sequence of steps. Prompt-level checks are useful, but they are not enough when the delivery risk sits in the full workflow.

Q: How should organisations govern LLM-as-judge scoring in agent workflows?

A: Treat the judge as part of the assurance chain, not as a neutral bystander. Define the scoring rubric, validate it against known examples, and monitor for drift. If another model is deciding whether output is acceptable, that model needs review, calibration, and traceable criteria of its own.

Technical breakdown

Why LLM evals need both assertions and scored checks

A useful eval separates deterministic failures from probabilistic quality signals. Assertions are for behaviours with one acceptable outcome, such as a refusal rule or a permission boundary, and they belong in CI because failure means the system has broken a requirement. Scored checks are different: they measure answers that live on a spectrum, often requiring an LLM judge or other reviewer to estimate quality, groundedness, or helpfulness. In practice, the engineering challenge is not just writing the check, but deciding which class of check matches the risk. Treating both as the same thing produces brittle pipelines or weak governance.

Practical implication: Classify every eval as either a gate or a trend signal before wiring it into CI.

How versioned eval suites support agent governance

The article describes a one-to-one mapping from test suite to dataset and from run to experiment. That matters because it turns evaluation into a repeatable control, not a one-off prompt exercise. Versioned suites let teams compare behaviour over time, inspect traces when a case fails, and preserve the exact scenario that triggered a regression. For agentic systems, this is especially important because behaviour can change with prompts, model versions, tool access, and context content. The eval file becomes the source of truth, which means governance can reason about change rather than anecdote.

Practical implication: Keep eval cases versioned with the code so behavioural drift is traceable and auditable.

Why non-determinism changes what CI should enforce

LLM output is not stable in the way traditional application logic is stable. A single successful run does not prove that the behaviour is safe or repeatable, and a single weak answer does not necessarily mean the application is broken. That is why repetition, latency measurement, and trace capture matter. The article also highlights the cost of running evals, which means governance has to balance confidence against test overhead. For agent and NHI-style workflows, the control question becomes whether the system stays within acceptable behavioural bounds under repeated execution, not whether it returns one perfect answer.

Practical implication: Use repeated runs and acceptance thresholds to measure stability instead of relying on one-off success.

Threat narrative

Attacker objective: The objective is not a conventional breach but uncontrolled model behaviour that bypasses meaningful release governance and creates unsafe agent actions.

1. Entry occurs when an LLM or agent is placed into a workflow without a clear evaluation gate, so behaviour can drift undetected as prompts, models, or tools change.
2. Credential or control abuse appears when the system is allowed to act on outputs that have not been separated into hard invariants and soft quality signals, creating room for unsafe actions to pass review.
3. Impact is governance failure: teams ship behaviour they cannot reliably reason about, debug, or bound, especially when traces and repeatable checks are missing.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

LLM evals are becoming a control plane for agent governance, not just a QA practice. The article shows that the real decision is whether a behaviour belongs in a hard assertion or a scored trend line. That maps directly to governance design for AI agents and other non-human systems, where some actions must never occur and others require tolerance bands. The practitioner conclusion is to treat eval architecture as part of access control and operating policy, not as a developer convenience.

Versioned evaluation is the missing accountability layer for AI behaviour drift. When a test suite becomes the source of truth, teams can explain why a model changed and which scenario moved. That is the difference between ad hoc prompt checking and defensible operational control. For agentic AI programmes, this is the control gap that matters most: without repeatable evals, there is no stable evidence that the system remains within policy boundaries.

Non-determinism creates governance debt whenever organisations expect binary certainty from probabilistic systems. The article correctly separates invariant checks from quality scoring because LLMs do not behave like traditional deterministic software. That distinction should shape policy for AI agents, model-assisted workflows, and identity-adjacent automations that rely on consistent decisions. The practitioner implication is to design controls around bounded variance, not fantasy precision.

LLM-as-judge scoring introduces a second layer of trust that must itself be governed. If another model is deciding whether output quality is acceptable, then the judging process becomes part of the assurance chain. That matters for AI governance programmes that need reproducibility, explainability, and traceable review criteria. The conclusion is simple: evaluate the evaluator, or the assurance model becomes another opaque dependency.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Identity Maturity Model

What this signals

Evaluation debt is now a governance problem. When teams delay deciding which behaviours are hard gates and which are trend metrics, they accumulate uncertainty that surfaces later as noisy releases, unexplainable drift, or controls that fail at the wrong threshold. The right unit of control is the test case, not the prompt.

Agent governance needs evidence trails, not just policy statements. Versioned eval suites, trace capture, and repeatable judge criteria give security and AI teams a defensible record of how behaviour changed. That is the practical bridge between model operations and identity governance when agents can act on behalf of users or services.

For practitioners

• Define hard invariants before scoring quality Separate refusal rules, authorization boundaries, and other must-pass behaviours from fuzzy quality measures so CI only breaks on true control failures.
• Version eval cases with application code Keep scenarios, expected outcomes, and judge prompts alongside the test suite so behavioural changes can be reviewed like any other code change.
• Record traces for every failing run Persist the prompt, context, output, and scoring result so engineers can see why the model behaved unexpectedly instead of guessing from the final answer.
• Use acceptance thresholds for trend metrics Set pass-rate or average-score thresholds for helpfulness and latency, but do not let a single weak run fail the build when variance is expected.
• Review the judge model as part of the control chain If an LLM judge determines whether output is acceptable, define its rubric, calibrate it against known cases, and treat its drift as an assurance risk.

Key takeaways

• LLM evals become useful governance controls when teams separate hard invariants from softer quality measures instead of treating every issue as a build failure.
• Versioned test suites and trace records give practitioners a repeatable way to explain behavioural drift across model, prompt, and tool changes.
• The assurance challenge is not just whether an agent answers correctly once, but whether its behaviour stays within policy boundaries over time.

Key terms

• LLM Evaluation: LLM evaluation is the process of scoring model outputs against defined criteria so teams can judge quality before or after release. It typically uses test datasets, scorers, and review workflows to measure correctness, safety, relevance, and task success in a repeatable way.
• Hard invariant: A hard invariant is a condition that must always be true for a system to be considered safe or functional. In LLM evaluation, it is the small set of behaviours that should immediately fail CI when broken, such as a refusal rule, format constraint, or access boundary.
• Quality Signal: A scored measurement of behaviour where the result lives on a spectrum rather than a pass or fail line. Teams use quality signals to track drift, compare runs, and understand model performance without turning every imperfect response into a release blocker.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.

Deepen your knowledge

The NHI Foundation Level course covers NHI governance, agentic AI identity, machine identity security, and secrets management through the industry's only accredited NHI security programme. It helps practitioners connect control design to the operational realities of identity-driven systems.