Why do repeated AI verdicts over the same finding sometimes differ?

Why This Matters for Security Teams

Repeated AI verdicts that shift on the same finding create an integrity problem, not just a usability issue. If one run labels a control failure as low risk and the next calls it critical, downstream prioritisation, ticket routing, and executive reporting can all become unreliable. For security teams, the practical risk is that inconsistent outputs mask whether the model is changing its mind or the input context has changed.

This is especially important when AI is used in triage, compliance summarisation, or analyst support. The evidence may be identical in name only, while the prompt, retrieval set, model version, temperature, or hidden policy layer differs between runs. Current guidance from the NIST Cybersecurity Framework 2.0 is useful here because it emphasizes repeatable governance, assessment, and oversight rather than assuming a tool is inherently stable. In practice, many security teams discover verdict drift only after a disputed finding has already been escalated, closed, or reported as resolved.

How It Works in Practice

AI systems can return different verdicts on the same finding for several ordinary reasons. Large language models are often sampled rather than forced into a single deterministic path, so even small probability shifts can change the output. Retrieval-Augmented Generation can also vary if the evidence set changes, ordering changes, or the search layer returns slightly different passages. If the workflow includes a classifier, policy engine, or human-in-the-loop review, the final answer may reflect different context rather than model randomness alone.

To make verdicts more reproducible, teams usually need to standardise the entire evaluation path, not just the model. That means versioning prompts, freezing evidence snapshots, recording the model and parameter set, and logging any post-processing rules. For AI governance, the NIST AI Risk Management Framework is relevant because it frames reproducibility as part of trustworthy system design. For security operations, the question is whether the system can explain why a verdict changed and whether that change is attributable to new evidence, a model update, or a configuration drift.

• Lock the prompt template and note every version change.
• Capture the exact evidence set used for each verdict.
• Record model name, build, decoding settings, and policy thresholds.
• Compare outputs against a fixed test corpus before release.
• Route disputed cases to human review when the confidence signal is unstable.

Where teams also use agentic ai, the issue expands into tool choice and execution order, because an AI agent may call different tools or retrieve different context on separate runs. These controls tend to break down when retrieval is live, evidence sources update continuously, and the organisation does not preserve an immutable snapshot of what the model actually saw.

Common Variations and Edge Cases

Tighter reproducibility controls often increase operational overhead, requiring organisations to balance stable outputs against the cost of versioning, logging, and controlled releases. That tradeoff becomes more visible when the AI is being used for high-volume triage or near-real-time monitoring, where even small changes in evidence freshness can alter the verdict.

There is no universal standard for exact repeatability in generative AI yet. In some cases, “same finding” is not truly the same because the surrounding context has changed, for example a new log entry, a revised policy baseline, or a different retrieval ranking. In other cases, the model output is deterministic at the token level, but the upstream system still changes the input package. The practical answer is to separate model variance from pipeline variance.

For sensitive or regulated use cases, the NIST AI 600-1 GenAI Profile and MITRE ATLAS are useful reference points for documenting attack surfaces and evaluation discipline. If prompt injection, retrieval poisoning, or evidence tampering is possible, a different verdict may indicate an adversarial condition rather than ordinary model noise. In those environments, the right response is not to trust one “best” answer, but to prove why the system changed its mind.

Related resources from NHI Mgmt Group

• How do human-in-the-loop and human-over-the-loop differ for enterprise AI?
• What is Agentic AI and how does it differ from traditional generative AI?
• Why do AI agents increase the blast radius of over-scoped NHI tokens?
• Why do AI agents and service accounts create the same governance problem?