How should security teams apply threat modeling to AI systems?

Why This Matters for Security Teams

Threat modeling AI is not just about prompt injection or model output abuse. Security teams need to model the entire AI system as an attack surface: training data, retrieval layers, inference endpoints, plugins, agents, and the workflows that consume outputs. That framing is consistent with the OWASP NHI Top 10 and the CSA MAESTRO agentic AI threat modeling framework, both of which push teams to treat AI behaviour as an operational security problem, not a purely data science problem.

The practical risk is that AI can read widely, generate convincingly, and trigger downstream actions faster than review processes can keep up. NHI risk research from The State of Non-Human Identity Security shows that over-privilege, weak monitoring, and missing rotation remain common attack drivers, which matters because AI systems often inherit the same patterns through API keys, service accounts, and third-party connectors. Current guidance from the CISA cyber threat advisories also reinforces that chained abuse across systems is a realistic threat pattern, not an edge case.

In practice, many security teams encounter AI abuse only after an output has already caused data exposure, a bad action, or a downstream privilege jump, rather than through intentional threat review.

How It Works in Practice

Effective AI threat modeling starts by decomposing the system into assets, trust boundaries, and action paths. The model itself is only one component. Teams should map what the system can ingest, what it can store, what it can retrieve, what it can generate, and what those outputs can cause elsewhere. That includes human review queues, agent tool calls, ticketing systems, code deployment pipelines, and any external APIs that accept model-generated content.

A useful pattern is to classify threats by stage:

• Data ingress: poisoning, sensitive-data leakage, malicious retrieval content, and untrusted file or document inputs.

• Inference: prompt injection, jailbreak attempts, model evasion, and context manipulation.

• Output handling: hallucinated actions, unsafe recommendations, unvalidated code, and policy bypass through automated execution.

• Connected workflows: abuse of agents, tokens, connectors, or orchestration layers to move from a model response into real-world impact.

At the control layer, current best practice is to combine access restriction, content validation, logging, and approval gates. For higher-risk workflows, teams should isolate sensitive retrieval sources, constrain tools by purpose, and require explicit review before the model can act. The threat model should also identify where identity and secrets are present, because exposed credentials are often the bridge from model misuse to broader compromise. NHIMG’s LLMjacking research illustrates how attackers exploit compromised NHIs, while the Anthropic report on AI-orchestrated cyber espionage shows that automation can compress attacker timelines.

Threat scenarios should be tested with realistic prompts, poisoned inputs, connector abuse, and simulated tool chaining. These controls tend to break down when the AI system has broad tool access and no enforced approval boundary, because the model’s output can immediately become an operational action.

Common Variations and Edge Cases

Tighter AI threat modeling often increases review overhead, so organisations must balance speed against the cost of missing a high-impact abuse path. That tradeoff is especially visible in agentic systems, where autonomy expands the number of actions the model can attempt.

One common edge case is retrieval-augmented generation. The model may be safe, but the retrieved content may not be. Another is delegated automation, where a model only “recommends” an action but a downstream workflow executes it without adequate validation. A third is multi-agent orchestration, where one agent’s output becomes another agent’s instruction, making trust boundaries hard to maintain.

Guidance is still evolving on how much autonomy is acceptable by default. Current guidance suggests modelling not only compromise scenarios, but also misuse of legitimate capabilities, because many AI incidents are policy failures rather than classic vulnerabilities. For broader risk mapping, The 52 NHI breaches Report remains useful for understanding how identity, secrets, and privilege exposure often become the real blast radius. For adversarial taxonomy, teams should also reference MITRE ATLAS adversarial AI threat matrix alongside MAESTRO.

Threat models also need refresh cycles, not one-time sign-off, because model capabilities, connectors, and business workflows change quickly. Where the AI can take action without a human approval step, the model should be treated as a privileged workload, not just a content generator.

Related resources from NHI Mgmt Group

• What steps should security teams take to prevent Shadow AI risks?
• How should security teams apply NIST 800-53 to AI systems with autonomous actions?
• Why are NHIs a critical concern for security teams?
• How should teams reduce the risk of exposed AI credentials being abused?