GenAI threat modeling: are your controls keeping up?

TL;DR: Gartner projects 25% of enterprise GenAI applications will face five or more security incidents a year by 2028, while 79% of organisations still have not reached full AI maturity in cybersecurity, according to Protecto. That gap means threat modelling now has to cover prompts, retrieval, model behaviour and agent permissions, not just conventional application boundaries.

Editorial analysis by NHI Mgmt Group, based on content published by Protecto: “AI Threat Modeling: A Practical Guide for Enterprise GenAI Security”.

By the numbers:

• Gartner projects that by 2028, 25% of enterprise GenAI applications will face five or more security incidents per year, nearly triple the 9% recorded in 2025.

Key questions

Q: How should security teams model GenAI systems differently from traditional applications?

A: Security teams should model GenAI around the prompt, retrieval, model and agent layers rather than around static application boundaries.

Q: When does AI threat modelling become an access control problem?

A: It becomes an access control problem when the model can reach sensitive data or operational tools that the original user should not control directly.

Q: What are the warning signs that GenAI governance is failing in production?

A: Common signs include rising token spend without clear business growth, poor traceability of prompt flows, inconsistent routing between models, and backend services that still contain long-lived API keys.

Practitioner guidance

• Inventory every AI interaction point Map chat interfaces, API endpoints, retrieval pipelines, agent tool calls and training data sources so hidden GenAI usage does not bypass governance.
• Classify sensitive data before model access Apply detection and masking before data reaches a vector database or prompt builder so PII, PHI and payment data never become model context.
• Define explicit trust boundaries for agents Set clear rules for what data an agent can access, what tools it can call and when human approval is required for high-risk actions.

Bottom line: GenAI threat modelling has to cover model behaviour, retrieval paths and agent permissions because classic software assumptions do not account for those controls.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →