Topic starter
19/08/2026 3:37 pm
TL;DR: Hard invariants and quality checks belong in different CI paths, according to Arize, because non-determinism, cost, latency, and LLM-as-judge scoring serve different control purposes. CI governance depends on repeatable checks that separate build-breaking failures from trend signals.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “Evals in CI: How to write your LLM evals as tests with Arize Phoenix”.
Key questions
Q: Why do LLM and agent evaluations need repeated runs instead of one pass?
A: Because a single successful run does not prove stability when the underlying system is non-deterministic.
Q: What breaks when AI quality is only checked at the prompt level?
A: Teams miss failures that happen outside the prompt text itself.
Q: How should teams decide which LLM evals belong in CI gates?
A: Put only hard invariants in CI gates.
Practitioner guidance
- Define hard invariants before scoring quality Separate refusal rules, authorization boundaries, and other must-pass behaviours from fuzzy quality measures so CI only breaks on true control failures.
- Version eval cases with application code Keep scenarios, expected outcomes, and judge prompts alongside the test suite so behavioural changes can be reviewed like any other code change.
- Record traces for every failing run Persist the prompt, context, output, and scoring result so engineers can see why the model behaved unexpectedly instead of guessing from the final answer.
Bottom line: LLM evals become useful governance controls when teams separate hard invariants from softer quality measures instead of treating every issue as a build failure.
Glossary terms: LLM Evaluation, Hard invariant, Quality Signal, LLM-as-a-judge
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 15 hours ago by NHI Mgmt Group
19/08/2026 4:17 pm
LLM evals are becoming a control plane for agent governance, not just a QA practice. The article shows that the real decision is whether a behaviour belongs in a hard assertion or a scored trend line. That maps directly to governance design for AI agents and other non-human systems, where some actions must never occur and others require tolerance bands. The practitioner conclusion is to treat eval architecture as part of access control and operating policy, not as a developer convenience.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: How should organisations govern LLM-as-judge scoring in agent workflows?
A: Treat the judge as part of the assurance chain, not as a neutral bystander. Define the scoring rubric, validate it against known examples, and monitor for drift. If another model is deciding whether output is acceptable, that model needs review, calibration, and traceable criteria of its own.
👉 Read our full editorial: LLM evals in CI expose the control gap in agent governance