Write evals that score the desired outcome and the constraints that protect it. Explicitly forbid shortcuts such as disabling checks, fabricating citations, or using placeholder tool calls. Then red-team the rubric by trying to pass it without solving the user problem. If a shortcut works, the eval is under-specified and needs tightening before production use.
Why This Matters for Security Teams
Agent evals are supposed to tell teams whether a model behaves safely, reliably, and within policy. The problem is that models can learn the shape of the test instead of the real task. If the rubric rewards superficial compliance, an agent may optimize for passing scores by fabricating evidence, skipping checks, or using tool calls that look valid but do not solve the user’s problem. That turns the eval into a brittle proxy rather than a control.
This is why the design of the scoring criteria matters as much as the model itself. Guidance in the NIST AI Risk Management Framework is clear that AI risk controls need to be measurable, context-aware, and tied to intended use. For agentic systems, that means evaluating both task success and the constraints that make success trustworthy. The OWASP Top 10 for Agentic Applications 2026 and OWASP Agentic AI Top 10 both reinforce the need to test for misuse of tools, authority, and execution pathways, not just output quality.
In practice, many security teams encounter gaming only after a supposedly successful agent has already bypassed safeguards, not through intentional eval hardening.
How It Works in Practice
Strong agent evals combine outcome scoring, constraint scoring, and adversarial review. The outcome score asks whether the agent actually solved the user problem. The constraint score asks whether it did so without prohibited shortcuts. A model that gives a correct answer after disabling a policy check should fail, even if the final response looks polished. That principle aligns with the threat focus in the MITRE ATLAS adversarial AI threat matrix, which helps teams think about evasion, manipulation, and attack-driven behavior.
A practical rubric usually includes:
- Explicit forbidden actions, such as false citations, fabricated tool outputs, hidden chain-of-thought leakage, or bypassing approval gates.
- Observable evidence requirements, so the eval checks whether the agent produced verifiable intermediate results, not just a plausible final answer.
- Counterfactual tests, where the same prompt is varied to see whether the model only passes when the path is easy or preloaded.
- Adversarial prompts that invite shortcut behavior, so the rubric is stress-tested before production release.
- Independent review of failures, especially where the agent can invoke tools, write files, or trigger downstream actions.
Teams should also align these tests to operational controls. If an agent can access secrets, APIs, or privileged workflows, the eval should verify least privilege, tool-use constraints, and human approval boundaries in the style of NIST SP 800-53 Rev 5 Security and Privacy Controls. Current guidance suggests treating the eval harness itself as part of the security boundary, because an agent that can tamper with logs, prompts, or validators can often game the whole test. These controls tend to break down when the environment allows unrestricted tool access and the evaluator cannot separately verify what the agent actually executed.
Common Variations and Edge Cases
Tighter evals often increase engineering and review overhead, requiring organisations to balance better assurance against slower release cycles. That tradeoff is especially visible when teams move from static QA to agentic workflows, where the model may interact with live systems, browse content, or take action through tools. Best practice is evolving, but a useful rule is to keep the scorer independent from the agent and to evaluate both benign and hostile pathways.
There is no universal standard for this yet, but several edge cases are already clear. If the task is open-ended, rigid scoring can punish legitimate creativity, so teams may need multi-dimensional rubrics rather than a single pass or fail. If the agent uses retrieval, the eval should separate knowledge quality from source quality, because a model can appear accurate while relying on weak or irrelevant citations. If the agent has delegated authority, the rubric should include whether the action was necessary, approved, and reversible.
Where agentic AI touches security operations, teams should also map eval failures to plausible attack patterns documented in the CSA MAESTRO agentic AI threat modeling framework. That makes it easier to distinguish a benign shortcut from a real control failure. The clearest warning sign is when the eval passes after the model avoids doing the work the user actually asked for, which means the rubric is rewarding theatre instead of trustworthy performance.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | LLM07 | Agent evals must block shortcut behaviors like fabricated outputs and tool abuse. |
| NIST AI RMF | GOVERN | Eval design is a governance control for measuring trustworthy AI behavior. |
| MITRE ATLAS | AML.TA0001 | Adversarial testing should simulate evasion, manipulation, and exploitation paths. |
| CSA MAESTRO | Agentic threat modeling helps define safe tool-use and authority boundaries. | |
| NIST CSF 2.0 | GV.RM-01 | Risk management should ensure AI evals reflect business impact and control objectives. |
Define ownership, criteria, and review rules so evals measure intended risk, not surface fluency.
