What is the difference between Attack Success Rate and Completion Under Policy for AI agent security?

How Attack Success Rate differs from Completion Under Policy

Attack Success Rate is an attack-centric measure. It asks how often a hostile prompt, tool sequence, or task causes unsafe or unintended behaviour. Completion Under Policy is a governance-centric measure. It asks how often the agent finishes legitimate work while staying inside the rules you set. The two metrics are related, but they answer different questions about agent security.

Because they measure different failure modes, neither metric is sufficient on its own. A system can look strong against adversarial prompts yet still be too restrictive to be useful. Equally, a high task-completion rate can hide fragile policy enforcement if the agent only succeeds when it is under little pressure. For that reason, teams should treat the pair as a balance between resilience and usability, not as interchangeable scores.

In practice, the comparison becomes clearer when you define the policy boundary before you measure it. Attack Success Rate depends on the attack set and the unsafe outcome you are testing for. Completion Under Policy depends on the policy rules, the task set, and what counts as compliant completion. If those definitions are vague, the numbers can move in opposite directions without telling you much about real risk.

Why both metrics matter for AI agent security

Attack Success Rate is most useful when you want to understand whether an agent can be manipulated into crossing a safety or control boundary. That makes it a good signal for red teaming, abuse testing, and comparing prompt or tool hardening approaches. It tells you how often adversarial pressure defeats the guardrails, but it does not tell you whether the agent remains practical for ordinary use.

Completion Under Policy fills that gap. It shows whether the agent can still do the intended job while respecting constraints on tool use, data handling, approvals, or action scope. That matters because security controls that are too strict can create a false sense of safety while quietly breaking workflows. A low attack rate is less meaningful if the policy blocks legitimate work so often that users route around the control.

Used together, the metrics help separate three conditions: an agent that is unsafe, an agent that is safe but unusable, and an agent that is both safe and effective. That is the core judgment in agent security, because a control that cannot support normal operations will not survive contact with production use.

What to measure when comparing the two

The most useful comparison is not just the raw percentages, but the conditions under which they were measured. You want to know which tasks were included, which policy rules were enforced, whether the agent had tool access, and whether the evaluation included adversarial prompts, ambiguous instructions, or multi-step workflows. Without that context, the same score can describe very different systems.

It also helps to look at the failure profile. A system with a slightly lower Attack Success Rate but a much higher Completion Under Policy score may be a better operational fit than one that is technically harder to trick but too brittle to use. The right choice depends on whether your environment values strict containment, broad task success, or a measured trade-off between the two.

For AI agent security, that trade-off is often the point. The best controls reduce unsafe success without turning every policy check into a productivity bottleneck. Current guidance suggests evaluating both metrics together so you can see whether improved protection is coming from genuine robustness or from excessive refusal.

Risk and Threat Considerations

These metrics can be misleading if teams optimise one while ignoring the other. Attack Success Rate can look excellent when the agent simply refuses more often, but that may just shift risk into availability and workflow failure. Conversely, a high completion score can hide a permissive policy surface that still allows an attacker to steer the agent into unsafe action.

Failure mechanism: The agent either becomes too permissive under adversarial pressure or too restrictive under normal use, so the security score no longer reflects real operating risk. That failure is usually caused by unclear policy definitions, weak evaluation sets, or controls that are not aligned to the agent’s actual tool and action boundaries.

Impact: Teams may deploy an agent that appears secure in testing but is either exploitable in practice or unusable enough that users bypass the control. In both cases, the organisation loses confidence in the policy layer and the measured security posture becomes less trustworthy.

Practitioner Guidance

What to verify: Check that the attack set, task set, and policy rules are all defined against the same operating model. If the policy can be gamed by simple refusal or by vague success criteria, the comparison is not yet decision-grade.

Decision rule: If Attack Success Rate improves but Completion Under Policy drops sharply, treat that as a control-usability regression, not a pure security win. If completion remains high while attack success stays high, tighten the policy boundary before trusting the agent in sensitive workflows.

What good looks like: The agent completes normal work reliably, refuses only where policy requires it, and shows a clear drop in unsafe success under adversarial pressure without collapsing legitimate task completion.

Practitioner takeaway: Use the pair as a balance test, not a scoreboard. The right agent is the one that stays bounded under attack while still completing enough real work to be operationally useful.

Related resources from NHI Mgmt Group

• Guide: Agentic AI Security Guide
• Why is single-provider AI agent governance not enough for enterprise security?
• How should security teams interpret jailbreak attack success rate in AI testing?
• How should security teams handle risks from AI browser extensions?
• How should security teams govern API keys used for generative AI access?