Should security teams treat best-of-K jailbreak results the same as one-shot results?

Why best-of-K and one-shot answer different security questions

Best-of-K and one-shot results should not be treated as interchangeable because they measure different interaction models. One-shot is a single attempt, so it reflects the chance that a prompt or policy fails on the first exposure. Best-of-K allows repeated attempts and records success if any trial works, which captures search efficiency, retry tolerance, and cumulative exposure across multiple tries.

That distinction matters operationally because a system that looks acceptable under one-shot testing may still fail under repeated probing. In other words, a low single-attempt failure rate does not mean the target is robust when an attacker can vary phrasing, reorder instructions, or explore the response space until one attempt lands.

For practitioners, the main takeaway is that the metric is part of the threat model, not just a reporting preference. If the control objective is resilience to persistent probing, best-of-K is the more informative measure; if the objective is first-contact exposure, one-shot remains useful, but it answers a narrower question.

What best-of-K reveals about search, persistence, and exploitability

Best-of-K is valuable because it surfaces how much effort is needed to reach a failure, not only whether a failure exists. That makes it a better proxy for attacker persistence, automated exploration, and prompt-space search. A weak system may withstand casual use yet fall quickly once an attacker can iterate, mutate inputs, or sample many candidate jailbreaks.

Best-of-K also helps distinguish brittle controls from controls that genuinely reduce the reachable attack surface. If success appears only after many tries, the defense may still be buying time, raising cost, and improving detection opportunities. If success arrives quickly, the system is easier to exploit and should be treated as materially exposed even if its one-shot score looks modest.

Red Teaming AI Agents for Identity Abuse is useful here because repeated probing is often the same practical pattern seen in jailbreak, delegation-abuse, and privilege-abuse testing: the attacker explores until one path succeeds.

Teams should interpret best-of-K as a signal about searchability and repeatability, not as a simple percentage that can be compared directly to one-shot. The same model can have very different operational meaning depending on whether the test allows retries, whether prompts are adaptive, and whether scoring counts first success or average success across attempts.

How to use both metrics without misreading the result

The cleanest practice is to keep both metrics, label them explicitly, and avoid collapsing them into a single headline number. One-shot should be used for first-attempt resilience, while best-of-K should be used for cumulative exposure under repeated probing. If you only publish one of them, you can easily overstate or understate the actual weakness.

That means practitioners should compare like with like: same model, same policy, same prompt set, same search budget, and the same success definition. Changes in any of those variables can make one result look better or worse without improving the underlying control. A fair reading depends on test design, not just the score.

FIRST matters because repeatable security testing only has value when the method is well-defined enough to support consistent comparison across runs.

MITRE ATT&CK Enterprise Matrix is also relevant as a reminder that threat assessment should reflect tactics and progression, not only isolated outcomes. Repeated attempts often change the practical risk even when the initial attempt fails.

Risk and Threat Considerations

Best-of-K can hide the true exposure if teams read it as “just another score” instead of a cumulative failure measure. The risk is that a model or control that seems acceptable under a single attempt may still be easy to break with persistence, variation, or automation, especially when the attacker can cheaply generate many candidate prompts.

Failure mechanism: repeated sampling expands the search space until one candidate bypasses guardrails, so the defender is no longer evaluating first-contact resilience but eventual reachability under sustained probing.

Impact: organisations may underestimate exploitability, overtrust a control that is brittle under iteration, and miss the need for stronger monitoring, rate limiting, or escalation thresholds.

Practitioner Guidance

What to prioritise: define which question each metric is answering before you present results. If stakeholders care about single-encounter safety, lead with one-shot; if they care about attacker persistence, automation, or repeated probing, best-of-K needs equal or greater weight.

What to verify: confirm that the test budget, retry policy, and success criteria are documented. Without those details, best-of-K can be misleading because a larger search budget will often produce worse-looking results even when the underlying control has not changed.

Common mistake: treating a good one-shot result as proof that the system is hard to jailbreak. In practice, the more useful question is whether the control still holds when an adversary can keep trying with small variations.

Practitioner takeaway: publish both metrics when possible, but interpret them as different views of the same control surface, not as interchangeable evidence of safety.

Related resources from NHI Mgmt Group

• Why are NHIs a critical concern for security teams?
• What steps should security teams take to prevent Shadow AI risks?
• Why is the abuse of NHIs a priority for security teams?
• What is the difference between SAST and DAST for security teams?