How should organisations evaluate AI agents used for security review?

Why This Matters for Security Teams

AI agents used for security review can look persuasive while still missing high-risk issues, repeating stale findings, or overfitting to a narrow code sample. That matters because a defensive tool is only useful if it can keep pace with changing repositories, realistic review scopes, and the organisation’s actual risk profile. Current guidance suggests evaluating these systems as operational controls, not as demos, using the same discipline applied to other security automation.

For agentic systems, the question is not only whether they identify vulnerabilities, but whether they do so consistently, with traceable reasoning, and without introducing unsafe actions or false confidence. The OWASP Agentic AI Top 10 is useful here because it frames failures such as excessive autonomy, weak tool boundaries, and poor output validation in terms security teams can test. NIST’s AI governance model also reinforces that evaluation should cover performance, reliability, and accountability, not just raw capability.

In practice, many security teams discover an agent’s limits only after the first false negative becomes a production incident, rather than through intentional validation.

How It Works in Practice

Effective evaluation starts by treating the AI agent like a review analyst with a defined workflow. That means giving it realistic pull requests, multi-repository dependencies, historical commits, and repeated rescans so its output can be compared over time. The goal is to measure whether the agent can keep findings current when code changes, not whether it can produce an impressive one-off report.

A practical assessment should examine three things: coverage, quality, and operating cost. Coverage asks whether the agent inspects the right files, paths, and dependency relationships. Quality asks whether findings are technically accurate, prioritised correctly, and explained in a way a reviewer can verify. Cost asks how much time, compute, and human oversight are needed to keep the results fresh as commits change. That framing aligns well with the NIST AI Risk Management Framework, which emphasises governance, measurement, and continuous monitoring.

Operationally, teams should test the agent against a mixed benchmark set:

• Known vulnerable code with ground-truth labels.
• Clean code with no findings expected, to measure false positives.
• Incremental commit diffs, to see whether the agent updates prior conclusions.
• Multi-repository scenarios, where issues emerge only across service boundaries.
• Tool-use boundaries, to confirm the agent does not exceed approved access or action scope.

Security teams should also assess whether findings can be reproduced manually, whether the agent can explain why a file was flagged, and whether it can distinguish a likely issue from a verified defect. Where agent behaviour touches adversarial manipulation, the MITRE ATLAS adversarial AI threat matrix helps teams think about prompt injection, data poisoning, and inference-time manipulation as part of evaluation design. These controls tend to break down when the agent is allowed broad repository access without a stable test harness because the evaluation signal becomes noisy and results are no longer comparable across runs.

Common Variations and Edge Cases

Tighter evaluation often increases reviewer workload and benchmark maintenance, requiring organisations to balance deeper assurance against slower release cycles. That tradeoff becomes sharper when the agent is used for continuous security review rather than periodic assessment, because the evaluation itself must evolve with code, models, and threat patterns.

There is no universal standard for what counts as “good enough” for defensive AI review, so the threshold should be tied to risk appetite and the consequences of a missed finding. For some teams, high recall on critical issue classes matters most. For others, precision and low false alarm rates are more important because analysts cannot absorb noisy output. Best practice is evolving, but current guidance consistently points toward scenario-based testing and repeatable rescans rather than static scorecards.

Edge cases matter. An agent that performs well on single-repository application code may fail on infrastructure-as-code, secret scanning, or distributed microservices where the security signal is spread across many files. Likewise, a model that appears strong in offline testing may degrade when fed partial context, truncated diffs, or tool outputs with ambiguous provenance. The CSA MAESTRO agentic AI threat modeling framework is helpful when security review is embedded in a broader agent workflow, because it encourages teams to think about control boundaries, task decomposition, and failure containment. The right question is not whether the agent can assist review in ideal conditions, but whether it remains trustworthy when repositories, commits, and permissions are all changing at once.

Related resources from NHI Mgmt Group

• When should organisations rotate secrets used by AI agents?
• How should security teams govern browser sessions used by AI agents?
• How should security teams govern AI agents without creating a manual review bottleneck?
• How should security teams govern bearer tokens used by AI agents and SaaS integrations?