GenAI Threat Modelling

What GenAI Threat Modelling Covers

GenAI threat modelling asks where a generative AI system can be bent off course, exposed, or induced to produce unsafe outcomes. It treats the model, prompts, retrieval layer, tools, and surrounding orchestration as one attack surface rather than as isolated parts.

The method is useful because generative systems fail in layered ways. Prompt injection, unsafe retrieval results, manipulated context, and tool misuse can each produce different consequences, so the threat model has to follow the path from input to output to action.

That is why a general threat modelling process is still the starting point, but the objects being analysed are AI-specific. The question is not only “what can be attacked?”, but also “what can be persuaded, poisoned, or overruled?”

Why GenAI Needs a Different Threat Lens

Traditional application threat modelling focuses heavily on request validation, authorisation, data handling, and abuse of backend logic. GenAI adds probabilistic behaviour, generated text as an execution influence, and model responses that may be treated as trusted even when they are not.

MITRE ATLAS adversarial AI threat matrix is a strong companion for this work because it catalogues AI and ML attack techniques such as prompt injection, context poisoning, tool misuse, and agent hijacking. Those patterns help teams move from vague concern to concrete attack paths.

GenAI systems also create new trust boundary problems. Retrieval pipelines can surface untrusted content, system prompts can be manipulated indirectly, and autonomous tool use can turn a bad instruction into a real-world action. The threat model should therefore trace influence, not just data flow.

Core Attack Surfaces to Model

The most important surfaces are the prompt channel, the retrieval layer, the model itself, and any tool or API the system can invoke. Each layer can fail independently, which means a single control is rarely enough.

Prompt injection matters because attacker-controlled text can reshape the model’s instructions or steer it away from policy. Retrieval poisoning matters because bad documents can enter context and be amplified by the model as if they were reliable evidence.

Tool use is especially sensitive when the model can act on behalf of a user. Once the system can send emails, query internal systems, or modify records, unsafe generation becomes an unsafe action path. A practical model of the system has to include those downstream effects.

CISA cyber threat advisories are useful when translating these patterns into current operational awareness, because they help teams connect abstract AI abuse cases to broader adversary behaviour and active exploitation trends.

How Teams Use the Threat Model

A useful GenAI threat model identifies where trust is assumed, where content can be influenced, and where a model output can trigger a side effect. That makes it easier to decide which risks need architectural controls, which need monitoring, and which need policy constraints.

NIST AI 600-1 GenAI Profile helps structure that work because it frames generative AI risk management around governance, testing, provenance, and incident handling. It is especially useful when a team needs to connect a threat model to an operating programme.

For systems that behave more like autonomous agents than chat interfaces, Threat Modelling AI Agents and the broader MITRE ATLAS Guide help extend the analysis from model output risk to agent behaviour, tool chaining, and attack surfaces created by autonomy.

Risk and Threat Considerations

GenAI threat modelling is most valuable when the system can ingest untrusted content, retain conversational context, or act through tools. Those conditions create realistic paths for prompt injection, data leakage, policy bypass, and unsafe downstream execution.

Failure mechanism: Attackers can shape the model’s context or inputs so that the system discloses sensitive information, follows malicious instructions, or performs an action the operator did not intend.

Impact: The result can be confidential data exposure, integrity loss in model-driven decisions, unsafe automated actions, or a compromise of connected systems that trust the model’s output.

Practitioner Guidance

What to watch for: Threat models should be revised whenever the system gains a new retrieval source, tool, plugin, memory layer, or autonomy step. Those additions often change the attack surface more than a model version change does.

Governance implication: Treat the GenAI threat model as a living design artifact, not a one-time review. Owners should be able to explain which prompts, contexts, tools, and outputs are in scope and which controls are expected to stop unsafe influence.

Practitioner takeaway: The most useful GenAI threat models are specific about where influence enters, where trust is granted, and where a generated output can become a real action.

Related resources from NHI Mgmt Group

• How can organisations tell whether their threat modelling is actually improving security?
• Who is accountable when AI components are added without threat modelling?
• How should security teams use API security events to improve governance and threat modelling?
• What is the difference between AI threat detection and GenAI-powered security assistants?