Notebook-style checks usually test final answers, not the sequence of tool calls, retrieved context, and workflow decisions that actually determine safety. That means a run can look correct while still touching the wrong data or taking the wrong operational path. Production evaluation needs traces, failure modes, and repeatable scoring tied to real sessions.
Why This Matters for Security Teams
Notebook-style evaluation gives teams a false sense of confidence because it tends to score the visible answer while missing the hidden execution path. For production agents, that path includes prompt composition, retrieval choices, tool selection, permission scope, and post-action side effects. Guidance from the NIST AI Risk Management Framework is clear that AI systems need lifecycle risk controls, not just output checks, and agentic systems raise the bar further because actions can extend into real services and data stores.
The practical problem is that a notebook can pass even when the agent uses stale context, calls the wrong tool, or reaches a safe-looking conclusion after unsafe behaviour. That matters for privacy, access control, and operational integrity, especially when the agent can retrieve sensitive data or trigger downstream workflows. The same issue appears in agent-specific guidance such as the OWASP Top 10 for Agentic Applications 2026, which treats tool misuse, overreach, and weak oversight as core risks rather than edge cases. In practice, many security teams encounter this only after a production agent has already taken an action that the notebook score never measured.
How It Works in Practice
Production evaluation needs to test the full agent loop, not a single final response. That means capturing traces that show what the agent saw, what it retrieved, which tools it invoked, what parameters it passed, and whether those steps matched policy. It also means scoring for failure modes such as prompt injection, tool abuse, unintended data exposure, and unsafe retries. The MITRE ATLAS adversarial AI threat matrix is useful here because it helps teams map observed behaviour to known attack patterns rather than treating every error as a generic model mistake.
A workable production evaluation program usually includes:
- Session-level replay so the same inputs can be re-run against the same model and toolchain version.
- Trace review for retrieval quality, chain-of-thought proxy signals, and tool-call sequencing, with care not to expose sensitive reasoning where it should not be stored.
- Policy assertions for data access, action approval, and escalation thresholds.
- Red-team cases that include malformed instructions, conflicting tools, and poisoned context.
- Scoring that separates answer quality from safety, compliance, and operational correctness.
Frameworks such as the CSA MAESTRO agentic AI threat modeling framework reinforce the need to evaluate orchestration paths, not only model outputs. This matters because the agent can appear helpful while still violating least-privilege or using the wrong source of truth. These controls tend to break down when teams evaluate only static notebook examples because production variance, tool latency, and live data access create failure modes that synthetic prompts do not reproduce.
Common Variations and Edge Cases
Tighter evaluation often increases engineering overhead, requiring organisations to balance confidence against speed and cost. That tradeoff becomes sharper when an agent uses multiple tools, calls external APIs, or operates across different environments. Current guidance suggests that there is no universal standard for how much trace data is enough, so teams need to define risk-based thresholds rather than assume one score works everywhere.
Some environments also change the evaluation model entirely. In regulated workflows, such as finance, health, or identity operations, a notebook pass may be irrelevant if the agent cannot prove data lineage, approval boundaries, or auditability. In autonomous or semi-autonomous systems, the real issue is not whether the answer looks plausible, but whether the system respected policy when it selected tools and data. Emerging standards like the NIST AI Risk Management Framework and the Anthropic report on AI-orchestrated cyber espionage both point to the same lesson: evaluations must reflect realistic attack paths and operational consequences, not just polished outputs. That distinction becomes critical when evaluation data is too clean, too short, or too disconnected from the permissions and workflows used in production.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN | Production agent evaluation needs lifecycle risk governance, not just answer scoring. |
| OWASP Agentic AI Top 10 | A2 | Tool misuse and unsafe orchestration are central risks when notebooks miss execution paths. |
| MITRE ATLAS | T1608 | Adversarial AI tactics help map injected or poisoned context to real attack patterns. |
| CSA MAESTRO | MAESTRO emphasizes orchestration and threat modeling across agent workflows. | |
| NIST AI 600-1 | GenAI profile guidance supports testing model behavior under realistic operational conditions. |
Define accountability, risk thresholds, and review gates for agent evaluation across the AI lifecycle.
