They should treat production traces as feedback into the control system, not just incident evidence. When a new shortcut or failure mode appears, the rubric needs to change so the same behaviour is no longer rewarded. That keeps the specification aligned with real operational risk instead of historical assumptions.
How production traces should change the evaluation loop
When production traces reveal a failure mode, the evals should be updated as part of the same control loop that observed it. The point is not to preserve historical scoring, but to make the rubric reflect the real behaviour now seen in operation. If the system found a shortcut, the evaluation should stop rewarding that shortcut and start detecting it reliably.
This is how evals stay useful after deployment: they become a living specification for the behaviour you still want, the behaviour you no longer want, and the edge cases that production exposed. If the rubric does not change after a real failure, the organisation is effectively training the metric to ignore its own lessons.
A good update usually has two parts. First, encode the newly observed failure as an explicit negative case or adversarial test. Second, tighten the success criteria so a superficially correct but operationally unsafe answer no longer passes. That keeps the eval aligned with business risk instead of with whatever happened to look good in the last round of testing.
What should be changed in the rubric, not just the dataset
The essential move is to change the scoring rule, not only add a new example. If production traces show a loophole, the rubric must describe why that behaviour is wrong, what signal indicates it, and how to score it consistently across future runs. Otherwise the same pattern will reappear under a slightly different form.
In practice, this means tightening failure definitions, adding boundary cases, and revisiting any rubric language that was too permissive. Production traces are especially valuable when they show an evaluation blind spot: a case that the model can exploit while still appearing to satisfy the written task. Updating the rubric closes that gap.
Teams should also separate “interesting” failures from material ones. Not every odd trace deserves a new rule. The cases worth promoting into evals are the ones that change user trust, safety, correctness, cost, or control of the system. That discipline prevents the evaluation suite from becoming noisy or overfit to one-off quirks.
How to keep evals aligned with real operational risk
The best test is whether the updated eval would catch the same class of failure if it appeared again in a different form. If the answer is no, the rule is still too narrow. If the answer is yes, the organisation has turned an incident into reusable assurance rather than a postmortem note.
For teams using model or agent evaluations, this is closely related to hardening the control loop around observed misuse patterns, not unlike how CSA AI Agent Disclosure Accountability Gap whitepaper describes accountability gaps that surface only after real-world behaviour is observed. The same principle applies here: production evidence should change the governing test, not just the incident record.
That same operational mindset also fits broader security control design. As NIST AI Risk Management Framework and NIST SP 800-53 Rev 5 Security and Privacy Controls both imply at a control level, feedback from operations should be used to maintain governance, testing, and response effectiveness over time.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack surface, NIST AI RMF and NIST SP 800-53 Rev 5 set the technical controls, and ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | Production traces updating evals is AI risk governance and ongoing feedback management. |
| Recommendation — Use governance processes to update eval criteria from production evidence and prevent repeat failure modes. | ||
| NIST SP 800-53 Rev 5 | AU-6 — Audit Record Review, Analysis, and Reporting | Production traces are audit-like evidence used to detect and respond to failures. |
| SI-4 — System Monitoring | Operational traces reveal undesirable system behaviour that monitoring should surface. | |
| Recommendation — Review traces to identify recurring failure patterns and feed them into control improvements. Monitor production behaviour and tune evaluations when observed behaviour diverges from intent. | ||
| ISO/IEC 42001:2023 | AI management system | Updating evals from production traces is a management-system feedback loop for AI governance. |
| Recommendation — Revise evaluation procedures based on operational evidence to keep AI controls effective. | ||
| OWASP Agentic AI Top 10 | ASI08 — Cascading Failures | New failure modes in production can propagate through agentic systems and should reshape tests. |
| Recommendation — Add regression tests for observed failures that could cascade into broader system impact. | ||
Practitioner Guidance
What to prioritise: Promote only those production failures that reveal a repeatable weakness in the spec, scoring logic, or acceptance threshold. A one-off anomaly belongs in analysis; a repeatable shortcut belongs in the eval.
What to verify: Check that the new rubric change would fail the bad behaviour even if the wording, prompt, or trace details changed slightly. If the control only catches the exact same trace, it has not been tightened enough.
Decision rule: If a production trace shows the system optimising for the metric while violating intent, rewrite the rubric so that optimisation path is no longer rewarded. If it only shows a rare edge case with no broader pattern, record it but do not necessarily expand the eval suite.
Practitioner takeaway: Treat production traces as specification debt repayment, not as retrospective evidence. The value of the trace is realised only when it changes future scoring behaviour in a way that reduces repeat risk.
Related resources from NHI Mgmt Group
- How should teams investigate recurring failures in agent production traces without losing the thread across code, evals, and monitoring?
- How do organisations operationalise NHI ownership at scale?
- When should organisations treat an NHI as a high-priority risk?
- How can organisations reduce the blast radius of compromised agent identities?