AI threat modeling for GenAI now needs model and agent controls

At a glance

What this is: Protecto argues that GenAI threat modelling must extend beyond classic software assumptions because model behaviour, retrieval pipelines and agent actions create attack surfaces that STRIDE and PASTA do not fully capture.

Why it matters: IAM, NHI and AI security teams need a threat model that governs what the model can see, what it can retrieve and what an agent can do before security gaps become live operational exposure.

By the numbers:

• Gartner projects that by 2028, 25% of enterprise GenAI applications will face five or more security incidents per year, nearly triple the 9% recorded in 2025.
• OpenText and the Ponemon Institute found that 79% of organisations have not yet reached full AI maturity in cybersecurity.

Context

GenAI creates a security problem that classic threat modelling only partly covers: the model itself behaves like an attack surface, not just the application wrapped around it. Prompt injection, retrieval poisoning, output leakage and agent actions all sit outside the assumptions that older software methods were built to test.

That is why threat modelling guide needs to be adapted for AI systems rather than copied into them unchanged. Where GenAI is tied to personal data, enterprise access or delegated tool use, the governance question becomes who controls context, who controls action and who can prove the model stayed within its intended boundary.

Key questions

Q: How should security teams model GenAI systems differently from traditional applications?

A: Security teams should model GenAI around the prompt, retrieval, model and agent layers rather than around static application boundaries. The key difference is that the model can be influenced by retrieved content and can also take actions through connected tools. That means threat modelling must cover context, permissions and output handling together.

Q: When does AI threat modelling become an access control problem?

A: It becomes an access control problem when the model can reach sensitive data or operational tools that the original user should not control directly. At that point, the question is not only what the model says, but what it can see and do. Runtime context and task scope matter as much as identity.

Q: What are the warning signs that GenAI governance is failing in production?

A: Common signs include rising token spend without clear business growth, poor traceability of prompt flows, inconsistent routing between models, and backend services that still contain long-lived API keys. If teams cannot explain who is calling what and why, governance is not working.

Q: Should organisations prioritise AI data governance before scaling AI adoption?

A: Yes. Organisations that scale AI before establishing discovery, classification, monitoring, and policy enforcement are effectively expanding the attack surface faster than they can govern it. AI adoption should be matched with controls that follow the data lifecycle, otherwise compliance, exposure, and misuse risks compound as usage grows.

Technical breakdown

Why classic threat models miss GenAI attack surfaces

Traditional frameworks such as STRIDE and PASTA assume predictable software paths, fixed trust boundaries and stable outputs. GenAI changes all three. The prompt, retrieval layer, model and agent layer can each be exploited independently, and a successful attack does not need to break the host application first. In practice, the model can be manipulated through retrieved content, context windows or the permissions attached to downstream tools. That makes the model, not just the app, part of the security boundary.

Practical implication: Model the AI system as a chain of trust boundaries and test each boundary separately, not as a single application security object.

How retrieval and context poisoning turn data into an exploit path

Retrieval-augmented generation expands the attack surface because external content is pulled into the model’s working context at runtime. If malicious instructions are hidden in documents, webpages or vector stores, the model can ingest them as if they were legitimate task context. Sensitive data creates a second risk: once it enters the context window, it can reappear in generated output even when it was never meant to be exposed. This is a governance issue as much as a technical one, because the control point is before retrieval and before prompt assembly, not after the answer is produced. See the LLM Security Guide.

Practical implication: Classify and mask sensitive content before vectorisation and prompt construction so poisoned or private data never reaches the context window.

Why agent permissions change the threat model

AI agents are not just text generators. When they can call APIs, write files, query databases or trigger workflows, they inherit operational authority that can be abused through a compromised prompt or poisoned context. That makes privilege scope central to AI security. An over-permissioned agent is effectively a delegated insider with machine speed, and traditional role-based assumptions are too coarse if the action itself is the security event. Governance has to shift toward context-based decisions, per-task limits and approval gates for high-risk actions. See the Agentic AI Security Guide.

Practical implication: Treat every agent tool call as a privileged action and constrain it to the minimum permissions needed for that task.

Threat narrative

Attacker objective: The attacker wants the model or agent to reveal sensitive data, execute unsafe actions or silently corrupt business decisions through trusted AI workflows.

1. Entry occurs through prompt injection, indirect injection in retrieved content, shadow AI use or other unsanctioned GenAI access paths.
2. Credential or context abuse follows when the model consumes poisoned retrieval data or inherited permissions that let it act beyond its intended scope.
3. Escalation happens when the agent or model reaches APIs, databases or workflows with more authority than the originating request should have had.
4. Impact is data leakage, corrupted outputs, unauthorised actions or regulatory exposure across enterprise AI workloads.

Breaches seen in the wild

• CISA Private-CISA GitHub leak 2026: A CISA contractor's public GitHub repo exposed AWS GovCloud admin keys, Artifactory credentials and plaintext passwords for six months.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI threat modelling has become a governance control, not a design workshop. The article’s core point is that GenAI introduces new trust boundaries around retrieval, inference and delegated action. That means security teams are no longer just validating software behaviour, they are governing how data, context and action flow through a probabilistic system. Practitioners should treat AI threat modelling as a standing control in the architecture lifecycle.

Shadow AI creates a governance blind spot that traditional inventory processes were never built to catch. If most enterprise usage happens through personal accounts and unsanctioned tooling, then the control problem is not just model risk, it is identity and access visibility. The organisation cannot secure what it cannot inventory, and AI access often escapes the same review paths used for standard application estates. Practitioners should connect AI inventory to identity governance and AI security maturity model thinking.

Context-based access is the right idea because static roles do not describe AI risk well enough. A model or agent may have the right identity but still be unsafe if it can reach the wrong data at the wrong moment. That is why runtime context, purpose and sensitivity should influence access decisions for AI systems. Practitioners should define the access decision at inference time, not just at enrolment time.

Data privacy and AI security are now the same operational conversation. The article shows that tokenisation, auditability and deletion controls are not privacy add-ons, they are part of the AI threat model. When a model can expose PII through retrieval or output, privacy failures become security failures. Practitioners should align AI governance, privacy and IAM so the same data cannot enter, persist and reappear without trace.

From our research library:

• Organisations that rely heavily on static credentials reported a 20-percentage-point increase in security incidents compared with those with low reliance, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Security Guide

What this signals

AI governance debt: teams that scale GenAI before they can inventory interactions, classify data and constrain agent authority accumulate risk faster than they can review it. The operational problem is not merely model quality, it is that security teams inherit a system whose actions are only partly human-directed, so governance must move upstream to context and issuance time.

The next phase of AI security will be measured by whether organisations can prove control over retrieval, tool use and sensitive data handling in real time. That is where identity governance intersects with AI governance: if the system can act, then access, purpose and approval need to be machine-enforceable rather than policy-only.

For practitioners

• Inventory every AI interaction point Map chat interfaces, API endpoints, retrieval pipelines, agent tool calls and training data sources so hidden GenAI usage does not bypass governance.
• Classify sensitive data before model access Apply detection and masking before data reaches a vector database or prompt builder so PII, PHI and payment data never become model context.
• Define explicit trust boundaries for agents Set clear rules for what data an agent can access, what tools it can call and when human approval is required for high-risk actions.
• Move controls to the point of risk Enforce masking, context filtering and access decisions at inference time rather than relying on downstream review after the model has already used the data.

Key takeaways

• GenAI threat modelling has to cover model behaviour, retrieval paths and agent permissions because classic software assumptions do not account for those controls.
• The biggest governance gap in the article is visibility. Shadow AI and hidden data flows make it impossible to secure what the organisation cannot inventory.
• Security teams need to move controls to inference time, where context, data sensitivity and tool access can be evaluated before the model acts.

Related NHI Mgmt Group resources: Threat Modelling Guide · AI Security Maturity Model · Agentic AI Security Guide · LLM Security Guide

Key terms

• GenAI Threat Modelling: The practice of identifying how generative AI systems can be manipulated, leaked from, or made to act unsafely. It extends conventional threat modelling to cover prompts, retrieval pipelines, model behaviour and autonomous tool use, because each layer can fail differently and produce distinct security outcomes.
• Context Window Poisoning: A technique that hides malicious instructions inside content an AI assistant automatically reads, such as comments, documentation, logs, or metadata. The model treats the text as part of the task context, which can steer code generation or reasoning without any direct prompt injection from the user.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Context-based access control: A policy model that authorizes access using the request’s identity, payload, origin, and intended action together. It is designed for systems where risk changes at runtime, especially AI agents and other NHI workloads that move through multiple tools and data sources.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity and machine identity security. It helps practitioners connect identity controls to the AI systems and access paths that now shape enterprise risk.