Attack success rate is not comparable without a shared method

At a glance

What this is: The post argues that attack success rate in jailbreak research is often misread because papers use incompatible measurement choices.

Why it matters: That matters to AI security and governance teams because weak measurement hygiene can make model risk, red teaming, and control effectiveness look better or worse than they really are.

By the numbers:

• We track published jailbreak research through a database of over 400 papers, which we update as new work comes out.

👉 Read Promptfoo's analysis of why jailbreak attack success rate is hard to compare

Context

Attack success rate, or ASR, is a measurement problem as much as a security metric. In jailbreak research, the same underlying attack can produce very different results depending on how many attempts are allowed, how prompts are selected, and how a judge labels outputs. For AI security teams, that means the number alone is not a reliable indicator of model robustness or red-team quality.

This matters because AI governance, model risk, and security testing programmes increasingly depend on benchmark-style evidence. If the unit of measurement is inconsistent, then comparisons across papers, vendors, or internal evaluations become misleading. The article is therefore about measurement discipline in AI security, not just about jailbreak tactics.

Key questions

Q: How should security teams interpret jailbreak attack success rate in AI testing?

A: Treat ASR as a conditional measurement, not a universal vulnerability score. Compare only evaluations that share the same attempt budget, prompt set, judge, and stopping rule. If any of those differ, the numbers may reflect methodology rather than model weakness. Decision-grade reporting needs the measurement unit, not just the percentage.

Q: Why do different jailbreak papers report such different attack success rates?

A: Because they often test different things under the same label. A one-shot test, a best-of-392 search, and a benchmark with ambiguous prompts can all produce ASR values that look comparable but are not. Judge error adds another layer of variance, so the headline number often reflects methodology more than attack quality.

Q: What do AI security teams get wrong about benchmark-based red teaming?

A: They often assume the benchmark is objective when the prompt set and rubric may be doing most of the work. If borderline prompts are scored as attacks or the judge is too loose, ASR can overstate risk. The result is a confidence problem, not just a testing problem.

Q: How can organisations make adversarial AI testing results reproducible?

A: Use fixed prompt sets, documented attempt budgets, explicit scoring rubrics, and a calibration set for the judge model. Then publish baseline refusal, attack configuration, and stopping logic together. Reproducibility depends on method disclosure, because ASR without methodology is not comparable across studies.

Technical breakdown

Why attempt budget changes attack success rate

ASR is often reported as if it were a single stable percentage, but it changes with the number of trials allowed per target. A one-shot test measures something different from best-of-K testing, where the attacker gets repeated tries and success counts if any attempt works. That is why a low per-attempt success rate can inflate into a very high reported ASR once retries, early stopping, and sampling strategy are introduced. In practice, this turns ASR into a function of both attack quality and search budget, which means two papers can be describing different threat models while using the same label.

Practical implication: Report ASR with the exact attempt budget and stopping rule so reviewers can compare like with like.

How prompt sets distort jailbreak benchmark results

A jailbreak benchmark only measures model vulnerability if the prompt set actually represents policy-violating requests. If the set includes ambiguous or debatable questions, then compliant answers may be scored as attack success even when the model is behaving reasonably. That shifts the metric from adversarial resistance to dataset construction quality. In other words, the benchmark inherits the judgement errors of whoever wrote the test prompts. For AI governance teams, this is a reminder that benchmark curation is part of the control surface, not an administrative detail.

Practical implication: Audit benchmark prompts before trusting ASR results, and separate clear policy violations from borderline cases.

Why the judge model becomes part of the control plane

Most jailbreak evaluations rely on an LLM judge, which introduces its own false positives, false negatives, and style biases. Different judges can score the same output differently, especially when outputs contain cautious preambles, partial refusals, or longer explanations. That means observed ASR is not only about the target model, but also about the scoring model and rubric. When the rubric is vague, judge variance rises. When the rubric is explicit, scoring becomes more stable. This is why reproducibility in AI testing depends on judge calibration, not just on attack generation.

Practical implication: Calibrate judges with explicit pass and fail criteria before treating ASR as a decision-grade metric.

NHI Mgmt Group analysis

ASR is a measurement construct, not a universal risk number. The article shows that identical attacks can produce wildly different results when the unit of analysis changes from one attempt to many. That means the field still lacks a common measurement language for jailbreak evaluation. For AI security governance, reproducibility is the real control objective, because unmanaged variance turns research claims into untrusted signals.

Benchmark quality is now part of AI assurance. If prompt sets include borderline or poorly classified requests, the test itself becomes the source of false success. This creates a governance gap between policy definition and evaluation design. Teams using ASR for model selection, acceptance testing, or residual risk reporting need to treat benchmark construction as a controlled process, not an ad hoc research choice.

Judge variance creates an AI governance trust gap: the scoring model can reshape observed vulnerability as much as the target model does. That is directly relevant to AI RMF, NIST-style measurement discipline, and internal model assurance programmes. The practitioner takeaway is simple: if two evaluations do not share the same rubric and judge calibration, their ASR values should not be compared as if they were equivalent.

Automated red teaming is exposing a repeatability problem in AI security. The more search, memory, and branching an evaluator adds, the more the measurement changes. This is not a reason to avoid automation. It is a reason to define the control objective as repeatable testing with explicit parameters, not leaderboard-style ranking. The field needs fewer headline percentages and more auditable methodology.

What this signals

ASR will only become more decision-useful when AI teams treat evaluation metadata as part of the control evidence. The practical shift is toward reproducible testing packets that capture prompts, budgets, judges, and baseline refusal in one record. That aligns with broader AI assurance expectations and reduces the chance that a model passes one lab and fails another for methodological reasons rather than real security differences.

Measurement governance is now a security control in its own right. For programmes that assess LLMs, the strongest signal is not a single ASR value but a stable methodology that survives reruns, reviewers, and model updates. Teams that pair calibrated judges with documented rubrics will be better placed to defend procurement, risk acceptance, and remediation decisions.

For practitioners

• Define the ASR unit before benchmarking State whether ASR is measured per attempt, per prompt, or per goal category, and record the exact attempt budget, decoding settings, and stopping rule for every run. Without that, results are not comparable across tests or teams.
• Separate clear violations from borderline prompts Review each benchmark prompt against the governing policy and mark ambiguous items explicitly. If a prompt could reasonably receive a safe, high-level answer, do not let it drive the headline ASR without a separate category.
• Calibrate judges with explicit rubrics Write pass and fail criteria that define actionable harm, refusal quality, and acceptable high-level guidance. Use the same rubric across judges, rerun a calibration set, and compare false positives and false negatives before trusting the scores.
• Report baseline refusal alongside attack results Run the same prompt set with no jailbreak strategy so you can see whether the model is already refusing appropriately. If baseline success is high, the evaluation may be measuring label noise or rubric mismatch rather than exploitability.

Key takeaways

• ASR is only meaningful when the test method is fully disclosed, because attempt budgets, prompt sets, and judges can move the result dramatically.
• Ambiguous prompts and loose rubrics can inflate jailbreak success, which turns benchmark design into part of the security problem.
• AI assurance teams should report baseline refusal, calibrated judge criteria, and fixed budgets together so ASR becomes reproducible evidence rather than a headline number.

Key terms

• Attack Success Rate: Attack success rate is the percentage of test attempts in which an injected adversarial scenario achieved its intended effect. For agentic systems, it helps distinguish a one-off anomaly from a repeatable workflow weakness that requires governance action.
• Best-of-K Testing: Best-of-K testing allows multiple attempts against the same target and counts success if any attempt works. It is a valid threat model, but it must never be compared directly with one-shot testing unless the difference in retries is made explicit and the ASR is converted back into a comparable unit.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Baseline Refusal: Baseline refusal is the behaviour of a model when exposed to a benchmark prompt without any jailbreak strategy applied. It helps reveal whether the test set is genuinely adversarial or whether the benchmark is inflating attack success through ambiguous or mislabeled prompts.

What's in the full article

Promptfoo's full article covers the operational detail this post intentionally leaves for the source:

• The full methodology checklist for reporting ASR with explicit attempt budgets, prompt sets, and judge rules.
• Worked examples showing how best-of-K testing changes the meaning of a jailbreak result.
• A reproducible evaluation setup for promptfoo red teaming runs, including baseline and attack configurations.
• The paper references and benchmark comparisons used to explain judge variance and prompt-set distortion.

👉 Promptfoo's full post includes the reproducible testing examples and evaluation checklist behind the ASR critique.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in the context of modern security programmes. It is designed for practitioners who need to connect identity controls to broader risk and assurance decisions.