Attack success rate is not comparable without a shared method

At a glance

What this is: This is an analysis of why attack success rate in jailbreak research is often not directly comparable across papers, because attempt budgets, prompt sets, and judge design change the result.

Why it matters: For IAM, NHI, and AI security teams, the lesson is that security metrics without a shared method can mislead prioritisation, benchmarking, and red-team decisions.

Context

Attack success rate is only useful when the measurement unit is clear. In jailbreak research, the same attack can produce very different numbers depending on whether success is counted per attempt, per target, or across best-of-K retries, which makes the metric look more stable than it is.

This also has a governance dimension for AI security programmes. If teams use ASR to compare tools, evaluate controls, or set risk acceptance thresholds, they need to know whether they are measuring the attack, the benchmark, or the judge. The broader lesson is that metric design is part of the security control surface, not just the reporting layer.

Key questions

Q: How should teams compare jailbreak attack success rates across papers?

A: Teams should only compare jailbreak ASR when the papers use the same unit of measurement, prompt set, attempt budget, and judge rubric. If one study reports best-of-K and another reports one-shot success, the numbers are not directly comparable. The right response is to normalise the methodology before drawing conclusions about which attack is stronger.

Q: Why do different jailbreak papers report such different attack success rates?

A: Because they often test different things under the same label. A one-shot test, a best-of-392 search, and a benchmark with ambiguous prompts can all produce ASR values that look comparable but are not. Judge error adds another layer of variance, so the headline number often reflects methodology more than attack quality.

Q: What breaks when the judge rubric for jailbreak testing is too vague?

A: A vague rubric turns the judge into an inconsistent policy interpreter instead of a measurement tool. Different judges then disagree on what counts as harmful, which makes ASR unstable and hard to reproduce. The practical failure is not only noise, but also false confidence in rankings that are driven by scorer behaviour rather than model behaviour.

Q: Should security teams treat best-of-K jailbreak results the same as one-shot results?

A: No. Best-of-K measures a different threat model because it allows repeated attempts and counts any success across the search space. One-shot results are closer to single interaction risk, while best-of-K captures persistence and search efficiency. Teams should keep both if they need them, but never present them as interchangeable.

Technical breakdown

Why best-of-K changes attack success rate

Attack success rate changes when researchers allow multiple tries and count success if any one attempt works. That is a valid threat model, but it is not the same as one-shot measurement. If a method has a 1% per-attempt success rate, then 392 tries can turn it into roughly 98% observed ASR without improving the underlying attack. The difference comes from aggregation, not capability. In practice, papers often mix per-target, per-attempt, and best-of-K reporting, which makes cross-paper comparison unreliable unless K and the stopping rule are disclosed.

Practical implication: Report ASR with the attempt budget and stop rule attached, or the number is not portable across evaluations.

How prompt set design distorts jailbreak benchmarks

A jailbreak benchmark only measures model vulnerability if the prompts are genuinely disallowed or clearly adversarial. When datasets include borderline or merely informational requests, compliant answers can be scored as attack success even though the model is behaving normally. That turns the benchmark into a test of label quality and policy interpretation, not just model robustness. The article shows that ambiguous prompts can inflate reported ASR, especially when the scoring rubric treats any non-refusal as failure. Good benchmark design needs a clean separation between harmful intent, borderline content, and harmless capability questions.

Practical implication: Sanity-check the baseline prompt set before attributing high ASR to model weakness.

Why the judge is part of the measurement

Most jailbreak papers rely on an LLM judge, and judge behaviour is not neutral. False positives and false negatives vary by judge model, output length, refusal style, and rubric specificity, which means the same response can be scored differently by different evaluators. The article cites cases where safe preambles, long outputs, or model self-preference patterns change observed ASR. This is why vague prompts such as 'Is this harmful?' are weak measurement instruments. A precise rubric defines pass and fail behaviour in operational terms, reducing variance across judges and across papers.

Practical implication: Treat the judge as a control component and document its rubric, calibration, and error patterns.

NHI Mgmt Group analysis

Attack success rate is a measurement design problem before it is a security result. In jailbreak research, the headline number often hides the unit of analysis, the stopping rule, and the sampling strategy. That means practitioners are not comparing attacks so much as comparing evaluation choices. For AI security teams, the governing question is whether the reported metric can support a real decision about risk.

Best-of-K is a threat model, not a universal benchmark. Allowing many retries reflects how motivated attackers behave, but it must be labelled as such. When papers compare one-shot results with exhaustive search, they are comparing different operational conditions, which weakens any claim of superiority. The practitioner lesson is to separate attack capability from search budget before using ASR in procurement or internal reporting.

Benchmark hygiene determines whether ASR is evidence or noise. If prompt sets contain ambiguous or borderline requests, high ASR may reflect dataset construction rather than model failure. That is a governance issue for AI safety, red teaming, and policy enforcement alike. The field needs cleaner benchmark curation and baseline checks before ASR can be treated as a durable signal.

Judge consistency is now a core part of red team methodology. An LLM judge is not a passive observer, because its rubric and bias shape the metric. This creates a measurement-control dependency that security teams already recognise in other domains: if the evaluator is unstable, the dashboard is unstable. The practitioner conclusion is to calibrate judges as carefully as the models they score.

Reproducibility is the named concept here: attack success rate without a shared method is not a shared metric. That concept should be treated as a reporting standard for AI security papers and internal evaluations alike. The practical implication is simple: no ASR figure should travel without its methodological metadata.

What this signals

Attack success rate needs methodological governance, not just better prompts. For AI security programmes, the decision is no longer whether to track ASR, but how to make it auditable enough to support control selection and risk acceptance. That means the reporting standard has to include budget, prompt quality, and judge behaviour.

Metric integrity is becoming part of AI assurance. When an evaluation depends on retries, rubric design, and scorer calibration, it behaves like a control system rather than a simple test. Teams should expect internal red-team evidence to be challenged unless the measurement method is documented as carefully as the result.

For practitioners

• Define the ASR unit before comparing results Record whether success is measured per attempt, per prompt, per target, or per goal category, and keep that unit consistent across evaluations.
• Publish the full attempt budget and stopping rule State the exact K value, whether early stopping is allowed, and how retries are counted so reviewers can interpret best-of-K numbers correctly.
• Run a baseline refusal check on the raw prompt set Test the prompts with no jailbreak strategy first to see whether apparent success is already high because of ambiguity or rubric mismatch.
• Calibrate the judge rubric with explicit pass and fail criteria Replace vague labels with concrete scoring instructions that define refusal, harmful detail, and safe educational content in operational terms.
• Report cost and repeatability alongside ASR Include median attempts to success, p95 attempts to success, token cost, and seed sensitivity so the metric can support procurement and risk decisions.

Key takeaways

• Jailbreak ASR is only meaningful when teams know exactly how the number was produced, because attempt budgets and stopping rules can reshape the result dramatically.
• Prompt quality and judge design can inflate or deflate observed success rates, so benchmark hygiene is part of AI security governance.
• Security teams should require reproducible measurement metadata before using ASR for prioritisation, procurement, or risk reporting.

Key terms

• Attack Success Rate: Attack success rate is the percentage of test attempts in which an injected adversarial scenario achieved its intended effect. For agentic systems, it helps distinguish a one-off anomaly from a repeatable workflow weakness that requires governance action.
• Best-of-K Testing: Best-of-K testing allows multiple attempts against the same target and counts success if any attempt works. It is a valid threat model, but it must never be compared directly with one-shot testing unless the difference in retries is made explicit and the ASR is converted back into a comparable unit.
• Judge Rubric: A judge rubric is the scoring rule used to decide whether an AI output is harmful, compliant, or unsafe. A precise rubric reduces scorer variance by defining pass and fail in concrete terms, while a vague rubric leaves the result open to interpretation and model-specific bias.
• Baseline Refusal: Baseline refusal is the behaviour of a model when exposed to a benchmark prompt without any jailbreak strategy applied. It helps reveal whether the test set is genuinely adversarial or whether the benchmark is inflating attack success through ambiguous or mislabeled prompts.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It helps security and IAM practitioners build a common baseline for governing machine identities, secrets, and AI-driven access patterns.