Pass@3

A repeat-evaluation method where the same model is run multiple times and a finding counts if any run detects it. It is useful when outputs vary between runs, because the pooled result can show the model’s practical coverage better than a single score.

Expanded Definition

Pass@3 describes a repeat-evaluation approach in which the same model or system is run three times, and the result is considered successful if any one run produces the expected outcome. In AI security and evaluation discussions, it is used to reduce the distortion caused by stochastic outputs, especially where a single attempt can miss a valid result even though the model is capable of producing it. The term is closely associated with generative model benchmarking, but it is not a general-purpose reliability measure for all software.

Because this method measures pooled detection across multiple attempts, it helps distinguish between occasional misses and repeatable capability. That makes it useful for comparing model behavior under variable sampling settings, temperature changes, or other non-deterministic conditions. Definitions vary across vendors and research teams on whether pass counts should be based on exact match, semantic equivalence, or human-reviewed acceptance, so the evaluation rule must be stated clearly. For broader governance context, NIST Cybersecurity Framework 2.0 is useful because it emphasizes measurable outcomes and repeatable assessment practices, even though it does not define Pass@3 itself.

The most common misapplication is treating Pass@3 as a measure of model truthfulness or security assurance, which occurs when teams infer real-world reliability from a narrow benchmark that only captures one success across three stochastic runs.

Examples and Use Cases

Implementing Pass@3 rigorously often introduces evaluation complexity, requiring organisations to weigh better coverage insight against the cost of repeated runs, scoring rules, and more careful interpretation.

• A code-generation model is tested three times on the same prompt, and if any run produces syntactically valid code that meets the task, the case counts as passed.
• An internal security assistant is evaluated for policy retrieval accuracy, and Pass@3 is used to see whether one of three sampled responses surfaces the correct control reference.
• A red-team benchmark checks whether a model can identify unsafe content, and the team records a pass when any of the three attempts flags the issue.
• A governance review uses Pass@3 to compare two model versions under identical prompts, helping analysts see whether one version has more stable practical coverage than the other.
• An evaluation pipeline aligns output scoring with NIST Cybersecurity Framework 2.0 style measurement discipline by documenting prompts, runs, and acceptance criteria before results are accepted.

These examples show why the method is often applied where exact reproduction is difficult and where sampling variability is part of the system’s normal behavior.

Why It Matters for Security Teams

For security teams, Pass@3 matters because evaluation methods shape the conclusions drawn about AI capability, risk, and operational readiness. If a model is judged on a single run, teams may underestimate useful coverage or overreact to one bad output. If Pass@3 is used without clear acceptance criteria, the opposite problem appears: weak or inconsistent behavior can look better than it really is. This is especially relevant in AI security, where teams need to understand whether a model reliably produces safe, compliant, or policy-aligned outputs under realistic variability. Pass@3 can therefore support benchmarking, but only when paired with explicit scoring rules, prompt control, and transparent reporting. It is also relevant to identity and agentic AI workflows where an AI agent may generate access guidance, remediation advice, or control mappings, because the benchmark can mask inconsistent reasoning across runs if governance is not tight.

Teams should treat Pass@3 as a measurement lens, not a security guarantee. It helps answer whether a capability can emerge across multiple attempts, but not whether the system should be trusted in production. Organisations typically encounter the limits of Pass@3 only after a model passes an evaluation yet fails during deployment, at which point the scoring method becomes operationally unavoidable to revisit.

Related resources from NHI Mgmt Group

• Why do loyalty accounts remain vulnerable after customers pass login?
• Why do organisations pass audits and still suffer identity-related breaches?
• How can organisations make access audits easier to pass?
• Why do biometric systems that pass liveness testing still create risk?