TL;DR: Five-in-four enterprise GenAI apps may face five or more security incidents a year by 2028, with 79% of organisations still short of full AI maturity, Protecto says. GenAI systems need runtime checks on prompts, retrieval paths, model behaviour and agent permissions to keep actions within scope.
Editorial analysis by NHI Mgmt Group, based on content published by Protecto: “AI Threat Modeling: A Practical Guide for Enterprise GenAI Security”.
By the numbers:
- Gartner projects that by 2028, 25% of enterprise GenAI applications will face five or more security incidents per year, nearly triple the 9% recorded in 2025.
Key questions
Q: When does AI threat modelling become an access control problem?
A: It becomes an access control problem when the model can reach sensitive data or operational tools that the original user should not control directly.
Q: What are the warning signs that GenAI governance is failing in production?
A: Common signs include rising token spend without clear business growth, poor traceability of prompt flows, inconsistent routing between models, and backend services that still contain long-lived API keys.
Q: How should security teams model GenAI systems differently from traditional applications?
A: Security teams should model GenAI around the prompt, retrieval, model and agent layers rather than around static application boundaries.
Practitioner guidance
- Inventory every AI interaction point Map chat interfaces, API endpoints, retrieval pipelines, agent tool calls and training data sources so hidden GenAI usage does not bypass governance.
- Classify sensitive data before model access Apply detection and masking before data reaches a vector database or prompt builder so PII, PHI and payment data never become model context.
- Define explicit trust boundaries for agents Set clear rules for what data an agent can access, what tools it can call and when human approval is required for high-risk actions.
Bottom line: GenAI threat modelling has to cover model behaviour, retrieval paths and agent permissions because classic software assumptions do not account for those controls.
Glossary terms: GenAI Threat Modelling, Context Window Poisoning, Shadow AI, Context-based access control
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
AI threat modelling has become a governance control, not a design workshop. The article’s core point is that GenAI introduces new trust boundaries around retrieval, inference and delegated action. That means security teams are no longer just validating software behaviour, they are governing how data, context and action flow through a probabilistic system. Practitioners should treat AI threat modelling as a standing control in the architecture lifecycle.
A few things that frame the scale:
- Organisations that rely heavily on static credentials reported a 20-percentage-point increase in security incidents compared with those with low reliance, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should organisations prioritise AI data governance before scaling AI adoption?
A: Yes. Organisations that scale AI before establishing discovery, classification, monitoring, and policy enforcement are effectively expanding the attack surface faster than they can govern it. AI adoption should be matched with controls that follow the data lifecycle, otherwise compliance, exposure, and misuse risks compound as usage grows.
👉 Read our full editorial: AI threat modeling for GenAI now needs model and agent controls