Production data is exposing why generic AI evals miss the mark

At a glance

What this is: This analysis argues that AI evaluation works best when it is grounded in production trajectories, user feedback, and behaviour-specific rubrics rather than generic one-size-fits-all judges.

Why it matters: For IAM, NHI, and agentic AI programmes, the same lesson applies: governance fails when controls are detached from real runtime behaviour, so monitoring must reflect how systems actually operate.

By the numbers:

• Only 5.7% of organisations have full visibility into their service accounts.
• 97% of NHIs carry excessive privileges, increasing unauthorised access and broadening the attack surface.
• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security.

👉 Read Judgment Labs' analysis of production-based evaluation for AI agents

Context

Generic evaluation systems fail when they treat every agent, task, and user population as if quality were uniform. In practice, the behaviours that define success vary by workflow, domain, and failure mode, so static rubrics and out-of-the-box judges quickly lose alignment with production reality. For AI agent governance, that is the same structural problem identity teams face when they monitor access or privilege without observing actual runtime behaviour.

The article frames production data as the source of truth because it reveals what users correct, approve, retry, or abandon. That matters beyond model quality: agentic AI systems are increasingly taking actions through tools and delegated access, which means evaluation is becoming a control surface for AI identity, permissions, and operational risk.

The starting position described here is becoming typical for teams scaling agents quickly. The mistake is not that they measure nothing, but that they measure the wrong thing and then trust the scores.

Key questions

Q: What breaks when AI evals are not grounded in production behaviour?

A: Teams end up optimising for the wrong hill. Generic judges and static rubrics can reward outputs that look clean in testing while missing the behaviours that users actually care about in production. That creates false confidence, slower iteration, and blind spots that only emerge after the system is already deployed.

Q: Why do agentic AI systems need behaviour-based monitoring?

A: Because agent quality is not stable across tasks, users, or tool chains. Behaviour-based monitoring captures what agents actually do in production, including corrections, retries, and failures, so organisations can align controls with real risk rather than assumptions made before deployment.

Q: How can organisations tell whether an AI agent is operating outside its intended boundary?

A: Look for inconsistent classifications, premature tool calls, fabricated inputs, and responses that ignore structured guardrails. Those signals show the agent is optimising for task completion rather than respecting the workflow boundary. The safest response is to tighten the schema and review the tool path, not just rewrite the prompt.

Q: Who is accountable when production data changes an AI control model?

A: The organisation that deploys the agent remains accountable for keeping evaluation, access, and monitoring aligned with current behaviour. In practice that means product, security, and risk owners need a shared review loop so changes in usage patterns trigger control updates rather than being left to model teams alone.

Technical breakdown

Why generic LLM judges fail across tasks

A generic judge assumes that one rubric can define quality across very different workflows, but agent performance is highly context dependent. “Correct” in code review, legal analysis, customer support, and data extraction does not mean the same thing, and a prompt that scores all of them equally will flatten meaningful differences. This is not just a model-evaluation problem. It is a measurement-design problem, because the scoring system encodes the organisation’s assumptions about acceptable behaviour. When those assumptions are wrong, the system can optimise neatly for the wrong outcome while appearing consistent.

Practical implication: define task-specific scoring criteria before you trust any agent quality metric.

How production trajectories reveal drift and failure modes

Production trajectories are the sequence of actions, tool calls, responses, corrections, and outcomes that show how an agent actually behaves under real usage. They matter because static rubrics age quickly while production environments change continuously. By grouping trajectories into buckets, teams can identify repeated failure modes, distribution shifts, and places where the original evaluation set no longer represents current usage. This approach moves evaluation from theoretical coverage to behavioural evidence. For AI systems with delegated access, the same logic helps reveal when an agent is operating outside the boundaries its controls were designed for.

Practical implication: sample real trajectories continuously so your eval set does not drift away from the production environment.

Why agent behaviour monitoring becomes part of governance

Agent behaviour monitoring is the operational layer that turns raw production signals into interpretable rubrics, scores, and rewards. It closes the loop between observation and improvement by letting teams identify what users actually value, codify it, and feed it back into tuning or post-training workflows. That is especially important as agentic AI systems start using tools, APIs, and delegated privileges, because governance depends on knowing not just what the model can do, but what it repeatedly chooses to do. In identity terms, behaviour monitoring is becoming the runtime companion to access control.

Practical implication: treat production monitoring as a governance control, not only as a model-improvement feature.

Threat narrative

Attacker objective: The objective is to exploit the gap between apparent evaluation quality and real agent behaviour so the system continues operating outside intended controls.

1. Entry occurs when an AI agent is given delegated access to tools, data, or workflows without evaluation that reflects real production behaviour.
2. Escalation happens when generic scoring or static rubrics fail to detect repeated misuse, overreach, or unintended action patterns in live trajectories.
3. Impact follows when the organisation keeps trusting an agent whose measured quality no longer matches actual runtime risk, leading to unsafe decisions or exposed systems.

NHI Mgmt Group analysis

Generic evals create governance debt: When teams rely on one-size-fits-all judges, they are not just mismeasuring quality, they are building a blind spot into the control plane. The article is right that the same label can mean different things in different domains, and that reality matters even more when agent decisions map to privileges, data access, or tool execution. In agentic AI, measurement is governance, so a weak rubric becomes a weak control.

Production behaviour is the nearest thing to an operational truth source: Static rubrics assume the organisation already knows what good looks like, but production usage usually disproves that assumption. The more dynamic the workflow, the more likely it is that user corrections, retries, and approvals reveal the real policy boundary. For identity programmes, this mirrors the shift from periodic access review to continuous runtime validation, which is why behavioural evidence should shape control design.

Behaviour monitoring should be treated as identity-aware AI control: Agent systems increasingly act with inherited or delegated access, so the question is no longer only whether the model is accurate. The question is whether the system’s observed behaviour stays within the intended permission envelope. That makes agent behaviour monitoring a natural fit for NHI governance thinking, especially where AI agents consume secrets, call APIs, or trigger downstream actions.

Post-building is the right mental model for agent reliability: The article’s “post-building flywheel” captures a real shift in security architecture. Teams cannot rely on pre-launch assurance when runtime data changes the system’s risk profile after deployment. The better approach is to use production telemetry to refresh behavioural expectations continuously, which aligns with the way mature IAM and NHI programmes already treat lifecycle drift and access sprawl.

Customisation is not optional when controls touch delegated automation: The biggest mistake in this space is confusing convenience with portability. A rubric that works for one agent or one workflow may be structurally wrong for another, especially when tools, memory, and permissions vary. Practitioners should read this as a warning that agent governance needs contextual controls, not templated scoring imported from another team.

What this signals

Behavioural telemetry is becoming the missing control layer for agentic systems: as agents take on more delegated actions, teams will need monitoring that measures actual runtime conduct rather than relying on pre-launch scoring. That pushes AI governance toward the same evidence-based discipline identity teams use for access and entitlement reviews, especially where tool use and delegated credentials are involved.

Production drift is the governance concept practitioners should watch: once user expectations, workflows, and model outputs start changing together, the original rubric stops being a reliable control. That is why runtime sampling, trajectory review, and reviewable exceptions are becoming central to both AI risk management and NHI-style lifecycle oversight.

The practical signal for security leaders is simple: if you cannot explain which behaviours your agents are allowed to repeat, you cannot defend the assurance story behind those agents. The policy boundary has to be measured in production, not inferred from design-time confidence.

For practitioners

• Map production telemetry to control objectives Define which agent behaviours matter for each workflow, then tie those behaviours to explicit control objectives such as allowed tools, approved data classes, and escalation thresholds. Use real trajectory data rather than generic satisfaction scores to decide what needs monitoring.
• Build task-specific eval rubrics from live usage Sample user corrections, retries, approvals, and failures from production, then convert the repeated patterns into small rubrics that match the actual task. Revisit those rubrics whenever the workflow, model, or delegation pattern changes.
• Separate model quality from runtime privilege risk Assess whether an agent is accurate and whether it is operating safely under the access it has been given. A high-scoring agent can still be a risk if it can invoke tools, move data, or trigger actions beyond its intended scope.
• Create an agent behaviour monitoring review loop Route recurrent behavioural anomalies into a recurring governance review so product, security, and risk owners can decide whether the issue is a model defect, a policy problem, or an access design flaw. That keeps evaluation from becoming a one-time exercise.

Key takeaways

• Generic AI evals fail when they ignore how task context changes the meaning of quality.
• Production trajectories expose the behaviours, drift, and failure modes that static rubrics miss.
• For agentic AI, behaviour monitoring should sit alongside access governance as a runtime control.

Key terms

• Agent Behaviour Monitoring: Agent Behaviour Monitoring is the practice of observing how AI agents actually act in production and turning those observations into measurable controls. It focuses on trajectories, tool use, corrections, and outcomes so teams can detect drift, misuse, and policy violations rather than relying only on pre-launch testing.
• Production Trajectory: A production trajectory is the recorded sequence of actions, inputs, outputs, tool calls, and outcomes produced during real use of an agent. It is the raw evidence base for evaluation because it shows how the system behaves under genuine operational conditions, not just in synthetic test cases.
• Rubric Drift: Rubric drift is the gradual mismatch between a scoring rubric and the behaviour users now expect in production. It happens when the environment, workflow, or user preference changes faster than the evaluation criteria, causing scores to remain stable while actual quality or safety declines.
• Behavioural feedback loop: A governance pattern where the outcome of a security event is communicated back to the user in plain language so future decisions improve. In phishing defence, the loop connects detection and remediation to awareness outcomes instead of treating training as a separate activity.

What's in the full article

Judgment Labs' full blog covers the operational detail this post intentionally leaves for the source:

• How the ABM layer structures trajectory capture, bucketing, rubric mining, and reward conversion in practice
• Examples of how production user preferences are turned into stable scoring dimensions for specific workflows
• The kind of behavioural metadata teams can use to distinguish model quality problems from workflow design problems
• Implementation detail on how judge outputs can feed post-training optimisation workflows

👉 Judgment Labs' full post covers the agent behaviour monitoring workflow, rubric design process, and post-training feedback loop.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It is designed for practitioners who need to connect identity governance to broader security and operational risk.