AI can’t universally verify behaviour, and that limits safety

At a glance

What this is: Pentera’s analysis says the universal dream of AI behaviour verification fails because computability limits block any general predictor, safety checker or containment guarantee.

Why it matters: IAM, NHI and AI governance teams should treat verification as scoped and evidence-based, because controls that assume universal certainty will fail when systems can generate unexpected behaviour.

Context

AI behaviour verification is the problem of proving that a system will always act within a desired boundary. Pentera’s analysis argues that this cannot be done universally for arbitrary programs, which means security teams cannot rely on a single control to certify every AI or software behaviour in advance.

For identity and access programmes, the practical implication is that governance has to move from abstract guarantees to scoped controls. That matters when AI systems request tools, interact with data, or operate inside workflows where IAM, NHI, and policy enforcement depend on bounded assumptions rather than universal proof.

Key questions

Q: How should security teams handle AI systems when universal verification is impossible?

A: Treat verification as scoped assurance, not absolute proof. Define the exact models, workflows, tools and data boundaries that are in scope, then monitor them continuously. The goal is not to certify every possible behaviour, but to reduce uncertainty inside a controlled operating envelope where human review and access controls still work.

Q: Why does AI inventory matter for IAM and NHI governance?

A: Because AI systems often depend on service accounts, API keys, tokens, and delegated integrations to operate. If the inventory does not capture those dependencies, identity teams cannot review access, rotate credentials, or offboard assets when they are retired. AI inventory therefore becomes the place where model governance and machine identity governance meet.

Q: What are the signs that AI safety controls are overclaiming certainty?

A: Look for policies that promise universal compliance, unconditional harmlessness or complete behavioural containment across every use case. Those claims usually hide weak scoping, vague exceptions and manual workarounds. A better signal is whether the team can state the exact operating envelope, evidence standard and escalation path for each deployment.

Q: Should organisations prioritise constrained AI architectures over broad behaviour testing?

A: Yes, when the system has direct access to tools, data or downstream actions. Broad testing still matters, but it cannot replace architectural limits on privilege, execution paths and allowed actions. Constrained design reduces the surface that verification has to cover and makes continuous oversight more realistic.

Technical breakdown

Why the halting problem blocks universal prediction

The halting problem shows there is no algorithm that can decide, for every program and input, whether execution will eventually stop. The proof works by contradiction: any supposed universal decider can be fed a self-referential program that forces the decider to be wrong. For AI, that means no system can reliably predict all possible behaviours of arbitrary code, because the problem is not a scaling issue but a logical limit. More compute, better models, or probabilistic methods do not change the undecidability boundary.

Practical implication: scope prediction to bounded classes of code, not arbitrary behaviour.

Why Rice’s theorem makes safety properties undecidable

Rice’s theorem generalises the halting limit to semantic properties of programs. Any non-trivial question about what a program does, rather than how it is written, becomes undecidable when asked universally. For AI safety, that includes questions such as whether a system will always be truthful, never cause harm, or stay within a policy under self-modification. The issue is not that these questions are unimportant. It is that no universal decider can answer them for every possible system. That pushes assurance toward restricted domains and evidence-based testing.

Practical implication: evaluate safety claims against specific systems and assumptions, not universal standards.

How Busy Beaver shows behaviour outruns computation

Busy Beaver captures how quickly program behaviour can exceed every computable predictor. For each machine size, it asks for the longest halting runtime among machines that do halt, and the resulting function grows faster than any computable function. That means even small programs can sit beyond practical or theoretical prediction once their behaviour is general enough. The lesson for AI governance is that scale does not close the gap between hard and impossible. There will always be cases where the behaviour frontier outruns formal analysis, regardless of training data or parameter count.

Practical implication: treat runtime analysis as a bounded assurance tool, not a universal containment mechanism.

NHI Mgmt Group analysis

Universal AI verification is a category error. The article’s core point is that some safety ambitions assume computation can produce certainty where logic says it cannot. That matters for governance because many assurance programmes still treat “prove the system is safe” as a default expectation. In practice, AI and software can only be verified within bounded assumptions, making scope definition the real control boundary.

Scoped verification is the only defensible model. Pentera’s framing aligns with a security reality IAM and NHI teams already know: controls work when the asset, identity, policy and time window are bounded. The same logic applies to AI behaviour checks. The question is not whether a system can be made universally safe, but whether the operating envelope is narrow enough to support reliable evidence and review.

AI governance debt grows when organisations overpromise certainty. When leaders speak as if a model can always be certified, they create expectations that no technical control can meet. That gap becomes governance debt, because exceptions, manual overrides and compensating controls quietly accumulate. The practical conclusion is to write policy around bounded assurance, monitored use and explicit escalation, not around absolute verification.

Identity controls become more important when AI acts inside workflows. As AI systems request tools, access data or trigger downstream actions, the governance problem shifts from model correctness to authorisation, accountability and containment. That is where IAM and NHI disciplines intersect with AI safety. The control challenge is to constrain what the system can touch, when it can act and how its actions are attributed.

Behavioural containment must be engineered as architecture, not expected as proof. The article is right to separate what can be verified from what can only be constrained. Security programmes should read that as a design instruction: limit privilege, narrow execution paths, monitor runtime behaviour and assume that some classes of unexpected behaviour will remain undecidable. The practitioner takeaway is to govern the boundary, not fantasise about eliminating it.

What this signals

Scoped assurance is the only durable operating model for AI safety. Security and identity teams should expect verification to remain partial, evidence-based and tied to a defined boundary of use. Once an AI system can act through real privileges, the control question shifts from “can we prove it is safe?” to “can we limit what it is allowed to do and observe when it deviates?”

AI governance debt appears when policy language outruns technical reality. If an organisation writes blanket guarantees into AI policy, it creates a gap that will later be closed by exceptions, compensating controls and manual sign-off. The cleaner pattern is to align policy, runtime controls and accountability to the narrowest defensible operating envelope.

Agentic AI identity becomes a governance issue, not a theoretical one. As soon as an AI system can request tools or invoke services, its identity, privilege and containment model matter as much as the model itself. That is why IAM, PAM and NHI discipline should sit inside AI governance from the start, not after the first incident.

For practitioners

• Define bounded verification scopes Limit assurance claims to specific models, workflows, inputs and time windows so safety evidence remains testable and auditable.
• Constrain AI tool and data access Apply least-privilege controls to any AI system that can call tools, query data or trigger actions, and review those privileges as part of change management.
• Separate safety claims from certification claims Write governance language that distinguishes monitored operation from universal proof, so policy does not promise what computation cannot deliver.
• Build continuous human oversight into runtime use Require escalation, review and exception handling for AI behaviour that falls outside the designed operating envelope, especially where identity or access decisions are involved.

Key takeaways

• Universal AI verification is blocked by computability limits, so no security programme should assume a model can be proven safe in every possible case.
• The practical alternative is bounded assurance, where teams define the exact workflows, privileges and conditions that are in scope for review.
• For AI systems that act through identities or tool access, the main control question is containment, authorisation and oversight rather than absolute behavioural proof.

Key terms

• Halting Problem: A foundational result in computability theory showing that no general algorithm can determine for every program and input whether execution will stop. It matters because it proves there are hard limits on prediction, even when the code and input are fully known.
• Rice's Theorem: A theorem stating that any non-trivial semantic property of programs is undecidable in the universal case. In plain terms, if the question is about what a program does rather than how it is written, no single algorithm can answer it for every possible program.
• Busy Beaver Function: A mathematical function that describes the maximum runtime of a halting machine of a given size. Its growth outruns every computable function, which makes it a clean demonstration that some behaviour spaces cannot be exhausted by computation, no matter how much compute is available.
• Selective Verification: A review model that assigns deeper scrutiny to high-risk changes while using automation to cover routine quality issues. It is a practical response to AI-assisted code volume because it concentrates human effort where failure has the highest security impact.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and agentic AI identity. It helps practitioners connect identity controls to the wider security decisions their programmes have to make.