How should security teams model GenAI systems differently from traditional applications?

Why GenAI threat models have to move beyond app boundaries

Traditional application threat models usually stop at the request, service and data layers. GenAI adds a model layer that can reshape the response based on prompts and retrieved context, so the security boundary is no longer just the application code path. That means teams need to model how inputs, retrieval sources, model behaviour and downstream actions interact as one system.

The practical shift is that a safe application can still produce unsafe outcomes if the model is induced to follow malicious instructions or if retrieved content is hostile, stale or overly broad. Security review therefore has to ask not only “can the app be exploited?” but also “can the model be steered, confused or overtrusted?”

GenAI also changes the trust model for outputs. A model response may be treated as advice, a generated artifact, or a trigger for another workflow, so output handling becomes part of the security design rather than a post-processing concern. The right question is whether the system can contain bad content, reject bad context and prevent unreviewed actions from becoming operational actions.

Where prompt, retrieval, model and agent layers create distinct failure points

The prompt layer is exposed to direct manipulation, indirect prompt injection and content that tries to override the intended task. The retrieval layer adds risks from untrusted documents, poisoned knowledge bases, excessive context inclusion and leakage through overbroad search or ranking. The model layer can amplify those inputs through hallucination, overconfident synthesis or policy bypass behavior.

The agent layer is the biggest architectural difference from a traditional app because the system may have tool access, API access or workflow permissions that let it do things, not just say things. Once an agent can call tools, retrieve data, send messages or update records, security teams have to reason about permissions, command scope, approval gates and blast radius in the same model as the prompt and context risks.

That is why GenAI threat modelling should separate what the model can LLM Security Guide calls prompt, context, output and action risks. It is also why the OWASP Top 10 for LLM Applications Guide is useful here, because it frames the attack surface around the full LLM application stack rather than a single application endpoint.

How to model controls around context, permissions and output handling

The most useful control model is to treat context as input material that must be curated, permissions as capabilities that must be bounded, and outputs as potentially unsafe until verified. In practice, that means validating which sources may enter the prompt, which tools the model may invoke, and which outputs may be displayed, stored or executed automatically.

Security teams should also model retrieval as a trust decision, not just a search feature. If the model can retrieve internal documents, tickets, chat history or web content, the control question becomes whether each source is appropriate for the task, whether it can be poisoned, and whether the system can prove where a specific answer came from.

That operating model is consistent with NIST AI 600-1 GenAI Profile, which emphasizes governance, provenance, testing and incident handling for generative systems. It also aligns with the NIST AI Risk Management Framework and the CSA MAESTRO agentic AI threat modeling framework, both of which push teams to think in terms of governance, risk and multi-step behavior rather than isolated prompts.

Risk and Threat Considerations

GenAI systems create a larger and more fluid attack surface than conventional apps because untrusted content can enter through prompts, retrieval and tool outputs, then influence later decisions. The main risk is not only data leakage, but also unsafe action execution when a model is trusted to act on text that it should have treated as adversarial.

Failure mechanism: Attackers or low-trust content can poison retrieved context, steer the model through prompt injection, or exploit excessive tool permissions so that a model response becomes an unsafe action path.

Impact: The result can be disclosure, fraud, unauthorized workflow changes, incorrect decisions, or a wider blast radius than a normal application exploit because the system combines language interpretation with execution authority.

Practitioner Guidance

What to verify: Confirm that your threat model explicitly separates prompt injection, retrieval poisoning, model misbehavior and tool abuse, because a single “application risk” box is too coarse for GenAI. If the model can call tools, the permission review has to be as strict as the model review.

Decision rule: If an output can trigger an external action, treat it as a controlled release point, not as harmless text. Require review, validation or policy checks before the output reaches a user, system or workflow that can change state.

Practitioner takeaway: The key modelling mistake is to treat GenAI as a smarter application; it is better understood as a system where trust, context and authority can all be influenced at runtime.

Related resources from NHI Mgmt Group

• How should security teams pentest agentic systems differently from web applications?
• How should security teams prevent model denial of service attacks in GenAI applications?
• How should security teams approach penetration testing for APIs differently from traditional web applications?
• What should security teams do first when GenAI is outside traditional model risk management scope?