Pass K

What Pass K Measures

Pass K is a repeatability metric, not a one-shot success rate. It asks whether a task still succeeds when the same agent or workflow is run multiple times, which makes it useful for judging stability, not just best-case performance.

Why Pass K Matters

Single-run success can hide brittle behaviour, especially in agentic systems where tool use, context selection, or downstream state changes can vary from run to run. Pass K makes that variance visible by testing whether success survives repetition, rather than assuming one good run reflects reliable behaviour.

That distinction matters when teams are comparing agents, prompts, or orchestration changes. A system that passes once but fails intermittently may look acceptable in a demo and still be operationally weak in production.

How To Interpret Pass K Results

Higher Pass K values generally indicate more stable execution, but the metric should be read alongside task difficulty, randomness, and any external dependencies that can change between runs. A low score does not always mean the underlying capability is absent, only that the outcome is not robust across repeated trials.

Because the metric focuses on repetition, it is especially useful when failures are nondeterministic or hard to reproduce. That makes it a better fit for reliability assessment than for judging a single isolated attempt.

Where Pass K Fits In Evaluation

Pass K belongs in evaluation workflows that need to separate incidental success from dependable behaviour. It is most informative when used with a fixed task set, consistent scoring rules, and enough repetitions to expose instability rather than mask it.

For that reason, it works well as part of a broader evaluation suite that also looks at correctness, consistency, and failure modes. A system can score well on a single attempt and still be a poor choice if it cannot repeat that success predictably.

Risk and Threat Considerations

Repeated-run metrics matter because intermittent behaviour can become an operational risk even when isolated tests look fine. In agentic or automated systems, that instability can lead to missed actions, inconsistent decisions, or hard-to-debug failures that only appear under production conditions.

Failure mechanism: The evaluation captures one successful execution while ignoring variance caused by stochastic model output, state drift, tool timing, or hidden dependency differences across runs.

Impact: Organisations may deploy systems that appear dependable in testing but fail unpredictably in real use, increasing reliability risk and making incident triage and change control much harder.

Practitioner Guidance

Why practitioners should care: Pass K is most useful when you need evidence that a workflow is not merely capable of success, but consistently capable of success. Treat it as a guard against overconfident approval based on a single green test run.

What to watch for: Large gaps between first-pass success and repeated-pass success usually signal unstable prompts, brittle tool dependencies, or hidden state sensitivity. Those are the cases where a system deserves more scrutiny before release.

Related resources from NHI Mgmt Group

• Why do loyalty accounts remain vulnerable after customers pass login?
• Why do organisations pass audits and still suffer identity-related breaches?
• How can organisations make access audits easier to pass?
• Why do biometric systems that pass liveness testing still create risk?