AI systems are not traditional software: why security models break

At a glance

What this is: This is an analysis of why generative AI cannot be secured like traditional deterministic software, and the key finding is that model behaviour can shift with prompt phrasing, retrieved context, and configuration changes.

Why it matters: It matters because IAM, NHI, and broader security teams need controls that govern data, permissions, and runtime behaviour around AI systems, not just code defects inside them.

👉 Read Noma Security's analysis of why AI systems are not traditional software

Context

AI security starts with a simple governance problem: organisations keep applying deterministic software assumptions to systems that are probabilistic by design. That mismatch matters because model output can change with context, retrieval, and fine-tuning, which creates a risk surface that conventional appsec tooling does not evaluate well.

For IAM, NHI, and AI governance teams, the intersection is real when models can access internal systems, call tools, or act on behalf of users. In those cases, the question is not only whether the model is accurate, but whether its permissions, instructions, and runtime boundaries are tightly controlled enough to prevent unsafe actions.

Key questions

Q: What breaks when security teams treat AI like traditional software?

A: The main failure is assuming the system will behave the same way every time. Generative AI is probabilistic, so the same input can produce different output depending on context, retrieval, or configuration. That makes fixed-rule testing incomplete and can leave unsafe behaviour undiscovered until the model is already embedded in business processes.

Q: When should organisations prioritise AI-specific controls over generic appsec checks?

A: They should prioritise AI-specific controls as soon as a model influences decisions, customer interactions, or connected workflows. At that point, the risk is no longer only code quality. The organisation must govern model behaviour, data inputs, permissions, and runtime changes as a single control surface.

Q: How can organisations tell whether an AI agent is operating outside its intended boundary?

A: Look for inconsistent classifications, premature tool calls, fabricated inputs, and responses that ignore structured guardrails. Those signals show the agent is optimising for task completion rather than respecting the workflow boundary. The safest response is to tighten the schema and review the tool path, not just rewrite the prompt.

Q: Who is accountable when an AI system makes a harmful decision?

A: Accountability should follow the identity chain that authorized, configured, or triggered the action, including the human owner, the platform team, and any delegated agent or tool account. If the organisation cannot name that chain, the governance model is too weak for regulated AI use.

Technical breakdown

Why generative AI is non-deterministic

Traditional software follows fixed logic. Generative AI instead produces outputs from learned statistical patterns, which means the same input can yield different responses depending on prompt wording, retrieval results, and model state. That variability is not a defect in itself. It is the operating model. Security implications follow from that fact: testing cannot rely only on a single expected outcome, because behaviour changes as context changes. Controls have to assume that unsafe output can emerge even when the code path has not changed. Runtime monitoring, red-teaming, and policy constraints are therefore part of security design, not post-deployment extras.

Practical implication: validate AI behaviour under varied prompts and contexts, not just against fixed test cases.

How model behaviour expands the AI risk surface

AI systems are influenced by training data, retrieval sources, user prompts, and connected tools. That means the risk surface extends beyond application code into data pipelines, knowledge sources, integration permissions, and model instructions. A system can appear safe in isolation and still produce harmful, misleading, or non-compliant output once connected to live business data or internal workflows. This is where AI governance overlaps with identity and access management: if a model can invoke tools, retrieve records, or trigger actions, then its permissions become part of the threat model. The boundary of the system is no longer just the app; it is the whole operating environment around the model.

Practical implication: inventory every data source, tool, and permission an AI system can reach before it goes live.

Why traditional security tools miss AI-specific failure modes

Rules-based monitoring and conventional vulnerability scanning are built to find known flaws, misconfigurations, and deterministic abuse patterns. They are much less effective at spotting hallucination risk, prompt-sensitive policy bypass, or behaviour changes caused by a model update. AI security therefore needs controls that evaluate reasoning quality, unsafe instruction following, and runtime drift. This is especially important when AI outputs influence customer support, compliance decisions, or automated actions. A model that is technically online but behaviourally unstable is a governance issue, not just a tuning issue. Security teams need a way to measure whether the model is still operating inside its intended boundary after each change.

Practical implication: pair AI-aware testing with runtime controls that can detect drift, unsafe output, and policy bypass.

NHI Mgmt Group analysis

Deterministic security assumptions are the first control failure in AI programmes. The article is right to frame generative AI as a different system category, because fixed-rule security models do not map cleanly to probabilistic output. That difference matters for governance, because risk review processes that assume stable behaviour will understate the probability of unsafe responses. For AI programmes, the practitioner conclusion is simple: security evidence must cover behaviour variance, not just code integrity.

AI governance debt is created when teams defer model-specific controls until after deployment. The article describes a common pattern where organisations move fast and promise to tighten security later, but AI systems can already be influencing decisions, customer interactions, and internal workflows. That creates governance debt: the longer controls wait, the more business processes depend on an unbounded model. The practitioner conclusion is to treat AI security as a lifecycle discipline from design through runtime.

Identity and permission boundaries become more important when models can act through tools. The article notes that AI systems may connect to internal systems or take actions on their own, which makes access control part of model security. This is where NHIMG’s lens intersects directly with AI governance, because tool access, delegated permissions, and runtime authorisation determine whether a model can exceed its intended role. The practitioner conclusion is to govern AI systems with the same rigour used for privileged services.

AI-aware threat modelling should replace generic appsec assumptions for model-facing systems. Traditional scanners can detect input validation problems or exposed endpoints, but they cannot reliably assess prompt injection, hallucination impact, or context-sensitive policy failure. That means threat modelling has to account for the model, its data sources, its tools, and the business decisions it influences. The practitioner conclusion is to classify model behaviour as an operational control surface, not an isolated feature.

Runtime drift is the named concept security leaders should track. A model can remain unchanged in code while its behaviour changes because of retrieval, fine-tuning, or prompt context. That makes runtime drift the practical governance problem behind many AI incidents, and it is why output monitoring and change control belong together. The practitioner conclusion is to measure behaviour continuously, not just validate the initial release.

What this signals

AI programmes will increasingly be judged on whether they can prove behavioural control, not just model performance. That pushes teams toward continuous testing, tighter change management, and explicit governance of the data and tools that shape model output. For practitioners, the practical question is whether the AI system still behaves inside its approved boundary after every prompt, retrieval, or configuration change.

Runtime drift: when a model changes behaviour without a code change, it becomes a security and governance problem rather than a pure engineering issue. That means AI risk reviews need to track not only release events but also data freshness, prompt policy changes, and tool permission scope. Teams that cannot measure drift will struggle to defend trust in production AI.

For practitioners

• Map AI control dependencies before deployment Identify every model, retrieval source, tool connection, and permission that can influence or extend the AI system. Treat these as part of the security boundary, not as separate implementation details.
• Add behaviour-based testing to release gates Test prompts, contexts, and retrieval combinations that can drive unsafe or misleading output. Include scenarios where the same request is phrased differently to expose variability before users do.
• Constrain delegated tool access tightly Limit what an AI system can read, write, or trigger through connected services. Review whether the model needs standing access at all, or whether task-scoped access is sufficient.
• Monitor for output drift after model changes Track whether the model starts producing different answer patterns after retraining, prompt changes, or data updates. Escalate drift when it affects customer guidance, compliance, or operational actions.
• Brief governance teams in risk language Explain that AI security is not just software hygiene. Frame the issue as control over behaviour, permissions, and trust boundaries so non-technical leaders can make informed decisions.

Key takeaways

• Generative AI cannot be governed like deterministic software because its output changes with context, retrieval, and configuration.
• The real risk surface includes data, permissions, tools, and runtime behaviour, not just application code.
• Security teams need AI-aware testing and runtime controls before models are embedded in business workflows.

Key terms

• Deterministic Software: Software that produces the same output when given the same input under the same conditions. Security controls for deterministic systems rely on predictable execution, fixed logic, and repeatable test results, which is why they do not fully capture how generative AI behaves in production.
• Runtime Drift: Runtime drift is the gap between an AI agent’s approved authority and its actual behaviour as conditions change. It appears when the agent adapts to new context, new integrations, or new instructions and begins acting outside the scope that governance originally defined.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• Tool-Use Boundary: The approved set of systems, actions, and data sources an AI model may access. This boundary is central to security because once a model can call tools or act on records, access control and permission scope become part of AI risk management, not just integration design.

What's in the full article

Noma Security's full article covers the operational detail this post intentionally leaves for the source:

• The article's leader-friendly explanation of why probabilistic model behaviour changes the security model.
• The article's examples of how prompt phrasing, retrieval, and fine-tuning can alter output and risk.
• The article's board-level framing for explaining AI governance to non-technical stakeholders.
• The article's suggested way to map AI controls to the broader lifecycle of design, build, and runtime.

👉 The full Noma Security article expands on the risk surface, runtime behaviour, and governance message for senior leaders.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity in a way that supports broader identity and security decision-making. It is designed for practitioners who need to connect identity control to modern operational risk.