LLM evals in CI expose the control gap in agent governance

At a glance

What this is: This is a practical guide to writing LLM evals as tests in CI, with the key finding that hard invariants and softer quality signals should be handled differently.

Why it matters: It matters because IAM, NHI, and AI governance teams need repeatable controls for agent behaviour, especially where output quality, access decisions, and runtime drift can affect security outcomes.

👉 Read Arize's guide to writing LLM evals as CI tests in Phoenix

Context

LLM applications create a governance problem that familiar software testing patterns only partially solve. Non-deterministic outputs, variable latency, scoring by another model, and context-dependent failures mean teams cannot rely on a single pass or a manual review to prove behaviour is safe. The primary challenge is not just quality assurance, but proving that the system stays within its intended operating boundary as it changes.

That challenge has a direct identity angle when LLMs act as agents, because the system under test can make decisions, call tools, and handle sensitive data in ways that resemble privileged non-human behaviour. In that setting, evals become a control surface for AI governance and NHI-style oversight, not just developer convenience. The article's starting point is typical for teams trying to operationalise LLM quality, but the governance implications are broader than the examples shown.

Key questions

Q: How should security teams test AI agents that can call tools and APIs?

A: They should test the agent's actual execution paths, not just its text responses. That means covering tool calls, protocol handling, approvals, and state transitions with adversarial scenarios that mirror real misuse. If the test cannot reach the access point where the agent acts, it does not meaningfully assess the security risk.

Q: Why do LLM evals need different controls from ordinary software tests?

A: Because LLM outputs are not fully deterministic, a single passing run cannot prove the behaviour is stable. You also have to account for model cost, variable latency, and cases that require another model to judge quality. The right control model is a blend of assertions, scoring, and trace review, not a single pass or fail check.

Q: What do teams get wrong when they treat evals as one-off checks?

A: They usually over-focus on the prompt or output and ignore the evaluation design itself. That leaves them with brittle checks, no versioned dataset, and little evidence when behaviour changes. A useful eval program treats the suite as a maintained control asset, with traceability, repeatability, and clear ownership.

Q: How do you know if agent evaluation is actually working?

A: It is working when it explains failures well enough to change the system, not just when it outputs a score. Good evaluation should point to the layer that failed, produce a reproducible trace, and feed confirmed failures into regression tests. If the score does not change engineering decisions, it is not yet operational.

Technical breakdown

Why LLM evals need both assertions and scored signals

LLM systems produce outputs that are partly deterministic and partly statistical, so one evaluation pattern cannot cover every control objective. Hard assertions are appropriate when there is exactly one acceptable result, such as a refusal condition or a strict formatting rule. Scored signals are better when quality lives on a spectrum, such as helpfulness, groundedness, or latency tolerance. The technical shift is to separate binary correctness from probabilistic quality, then record both in the same run so regressions are visible without turning every imperfect answer into a failed build.

Practical implication: gate CI only on invariants that must never break, and track quality metrics separately.

How Phoenix maps test cases to datasets and experiments

Phoenix treats a test suite as a dataset, each case as a dataset example, and each execution as an experiment. That mapping matters because it makes evaluation data versioned and reproducible instead of hidden inside ad hoc scripts. The suite becomes the source of truth, while traces, outputs, and annotations are attached to each run. Content-addressed storage means unchanged cases reuse the same example hash, which prevents duplicate drift and makes changes in behaviour easier to compare across runs. This is a testing architecture for LLM observability, not just a logging convenience.

Practical implication: keep eval cases in code so teams can diff behaviour and review trace changes as part of CI.

What CI means when model behaviour is non-deterministic

Continuous integration for LLMs is not about expecting stable text. It is about bounding variance and proving that the smallest critical behaviour still holds as prompts, models, or context change. Repetition is necessary because one successful run says very little, and latency plus cost become first-class acceptance criteria rather than incidental telemetry. The article's core architectural point is that LLM quality control needs both run-level traces and suite-level thresholds. That is a different discipline from classic unit testing, but it still uses the same test lifecycle and developer workflow.

Practical implication: define acceptance thresholds for trend metrics, then let CI fail only when a critical invariant breaks.

NHI Mgmt Group analysis

LLM evals are becoming the missing control layer for AI governance. The article shows that ordinary test frameworks can carry evaluation logic, which is useful because most organisations still lack a repeatable way to prove that agent behaviour stays inside policy. For identity and governance teams, the real issue is not the test runner itself but whether the underlying decision system is measurable, versioned, and auditable. That is where AI RMF governance expectations meet operational control, and it is where teams should treat eval design as a governance function, not a developer preference.

Unit-test thinking is useful, but it does not eliminate agent risk. Hard assertions can catch a refusal failure or a broken invariant, yet they do not explain whether an LLM agent is drifting in subtle ways that affect access, tool use, or data handling. The article correctly separates deterministic checks from scored quality signals, which is the right pattern for systems that can degrade without producing a clean failure. Practitioners should recognise this as a monitoring boundary problem: some behaviours must block release, while others require trend-based oversight and escalation thresholds.

Agent observability needs a named concept: evaluation drift debt. When teams postpone eval design, they accumulate a backlog of unmeasured behaviour that becomes hard to reason about later. That debt is not just technical. It weakens accountability because no one can show what changed, when it changed, or whether the model crossed a policy boundary. In identity-governed AI environments, that creates gaps similar to unmanaged privilege sprawl, and the fix is disciplined dataset versioning, trace retention, and explicit ownership for evaluation criteria.

LLM testing and NHI governance are converging on the same control logic. The article's emphasis on repeatable runs, traceability, and bounded behaviour mirrors the lifecycle controls identity teams already use for service accounts and other non-human identities. The difference is that AI agents can change output paths dynamically, so governance must cover both the identity of the system and the behaviour it exhibits at runtime. That aligns with OWASP Agentic AI Top 10 thinking and should push practitioners to treat agent evals as evidence of control effectiveness, not just developer hygiene.

What this signals

The practical signal for security and identity programmes is that AI governance is moving from policy statements into execution evidence. Teams that cannot show versioned evals, repeatable thresholds, and trace-backed outcomes will struggle to prove control effectiveness when agent behaviour changes across releases.

Evaluation drift debt: the longer teams delay formal evals, the harder it becomes to reconstruct what a model or agent was allowed to do at any point in time. That creates an auditability gap that looks a lot like unmanaged lifecycle risk in identity programmes.

For programmes using OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework, the next step is to treat evals as evidence of control operation, not just development quality.

For practitioners

• Define hard invariants for agent behaviour Identify the smallest set of outcomes that must never fail, such as refusal rules, tool access boundaries, or required output formats, and make those release-blocking assertions in CI.
• Separate trend metrics from build gates Record helpfulness, latency, and groundedness as monitored signals, then set pass-rate or threshold criteria so CI fails only when a critical boundary is crossed.
• Version eval cases alongside application code Store scenarios, expected behaviour, and judge criteria in the repository so teams can diff changes, reproduce failures, and tie behaviour drift to specific commits.
• Retain traces for audit and debugging Capture prompts, outputs, and evaluator annotations for each run so security, compliance, and engineering teams can review why a case passed or failed.

Key takeaways

• LLM evals become governance controls when teams separate non-negotiable invariants from softer quality signals.
• Versioned datasets and trace retention turn agent testing into auditable evidence rather than disposable QA.
• Organisations that want to govern AI agents effectively need release gates, trend monitoring, and ownership for evaluation criteria.

Key terms

• Scope drift: Scope drift is the gradual mismatch between what an integration was meant to do and what its credentials still allow it to do. It happens when permissions are not revalidated as business needs change, creating hidden over-privilege across SaaS and API-connected systems.
• Hard invariant: A hard invariant is a condition that must always be true for a system to be considered safe or functional. In LLM evaluation, it is the small set of behaviours that should immediately fail CI when broken, such as a refusal rule, format constraint, or access boundary.
• Scored signal: A scored signal is a quality measure that sits on a spectrum rather than a pass-fail boundary. It is useful for outputs that can be acceptable at different levels, such as helpfulness, groundedness, or latency, and it is usually trended over time instead of used as a release blocker.

What's in the full article

Arize's full article covers the implementation detail this post intentionally leaves for the source:

• pytest and Vitest/Jest examples that show how to structure evals as executable tests
• Phoenix logging patterns for traces, outputs, annotations, and experiment tracking
• Acceptance-criteria examples for suite-level quality thresholds and latency gating
• A full support-bot walkthrough showing invariants versus scored signals in practice

👉 The full Arize article shows the pytest and Vitest/Jest patterns used to operationalise the eval workflow.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and agentic AI identity. It helps practitioners connect identity controls to the operational evidence their programmes need.