They need a continuous evaluation loop. Production logs should feed new edge cases into the test set, and the scoring rubric should be reviewed whenever user behaviour, product scope, or failure patterns shift. Static evals decay quickly in dynamic AI systems.
Why This Matters for Security Teams
AI evals are not a one-time quality check. They are part of the control surface for AI governance, because a model that looked safe in lab conditions can become unreliable once user intent, prompt style, or tool access changes. NIST’s NIST Cybersecurity Framework 2.0 is useful here because it reinforces the need for continuous risk management rather than periodic review alone.
Security teams often treat evals as a model-launch gate, then stop refreshing them when the system is promoted to production. That creates blind spots around prompt injection, retrieval drift, unsafe completions, and tool misuse. For agentic systems, the issue is sharper: the eval must cover not only the LLM’s text quality, but also execution behavior, escalation boundaries, and whether the agent follows policy when context changes. Current guidance suggests that evals should track business-critical failure modes, not generic benchmark scores.
What matters is whether the test set still reflects actual use, not whether it still looks impressive in a report. In practice, many security teams encounter eval failure only after users have already found a new abuse path or product teams have expanded the workflow beyond the original assumptions.
How It Works in Practice
A continuous evaluation loop usually starts with production telemetry. Teams collect prompts, retrieval results, tool calls, refusals, hallucination reports, and human escalations, then sample them into a maintained eval set. The point is not to store everything forever, but to build a living corpus of edge cases that reflects current behavior. When the product changes, the rubric changes too.
For example, if a support assistant gains access to a new knowledge base, the eval should be expanded to test source prioritisation, stale-content handling, and citation quality. If an AI agent is allowed to trigger actions, the eval should include approval boundaries, tool-output validation, and adversarial sequences that try to bypass policy. This is where alignment to NIST AI Risk Management Framework becomes practical: the team needs defined roles, measured risk tolerance, and documented reassessment points.
- Use production logs to identify newly observed failure modes.
- Separate evals for model quality, safety, and agent execution behavior.
- Version the rubric whenever policy, scope, or user intent shifts.
- Retest after model updates, prompt changes, retrieval changes, or tool additions.
For teams operating at the LLM layer, the OpenAI evals guidance is one example of how to operationalise repeatable testing, but the control objective remains vendor-neutral: prove that the current system still behaves as intended under current conditions. These controls tend to break down when production logging is incomplete or when high-volume environments cannot reliably label failures quickly enough.
Common Variations and Edge Cases
Tighter evaluation coverage often increases operational overhead, requiring organisations to balance faster release cycles against more frequent rubric updates and review effort. That tradeoff is real, especially when multiple teams share one model or when the same model serves very different user groups.
There is no universal standard for how often to refresh evals. Best practice is evolving, but a risk-based approach is consistent: high-impact use cases, regulated workflows, and agentic systems deserve more frequent refresh cycles than low-risk drafting tools. In AI governance terms, the eval set should represent the most harmful plausible failure modes, not just the most common ones.
Edge cases appear when user behavior changes faster than the model itself. A customer-facing assistant may start receiving adversarial prompts after public launch, or an internal copilot may be repurposed for sensitive workflows without a corresponding update to the rubric. The same issue arises in retrieval-augmented generation when source content changes daily; stale documents can make yesterday’s “correct” answer unsafe today. Where agent identities, permissions, or tool scopes are involved, evals should also cover whether access boundaries still match intended control design, because model correctness alone does not prevent misuse. The gap usually appears when teams assume static benchmarks are still valid after product scope expansion.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST AI 600-1 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI risk management requires ongoing measurement as models and use cases change. | |
| MITRE ATLAS | ATLAS helps model adversarial failures like prompt injection and misuse patterns. | |
| OWASP Agentic AI Top 10 | Agentic AI evals must cover tool abuse, escalation, and unsafe autonomous actions. | |
| NIST AI 600-1 | GenAI profiles emphasize testing outputs, prompts, and system behavior over time. | |
| NIST CSF 2.0 | GV.RM-03 | Risk management should account for changing operational conditions and outcomes. |
Test the agent’s execution boundaries, approvals, and failure handling as part of each eval cycle.
Related resources from NHI Mgmt Group
- How should security teams test AI agents after prompts, models, or tools change?
- Why do AI models change the way AppSec teams should prioritise risk?
- How should security teams handle SaaS offboarding when users also use AI tools?
- How should security teams govern AI models that can call tools and access data?
