Who is accountable when AI components are added without threat modelling?

Why This Matters for Security Teams

When AI components are added without threat modelling, the risk is not just a missed document. It is a missed control decision about data flow, trust boundaries, failure modes, and who can trigger external actions. That gap matters because AI features often inherit application permissions, connect to sensitive repositories, and introduce new paths for prompt injection, model misuse, and unsafe tool execution. Guidance from NIST AI Risk Management Framework makes clear that governance and mapping risk are not optional extras; they are part of responsible AI deployment.

Accountability should therefore be explicit before release. Product ownership decides the business purpose, engineering owns the implementation choices, and security owns the review gate that should have forced a threat model. If AI is embedded into an existing product without that review, the organisation often assumes the prior control set still applies, even though the attack surface has changed. Current guidance suggests treating each AI-enabled workflow as a new system boundary, not a feature toggle.

In practice, many security teams encounter the missing threat model only after a new AI path has already touched sensitive systems, rather than through intentional design review.

How It Works in Practice

A workable accountability model starts by assigning one named owner for the AI-enabled change, then separating that role from the people who build and approve the underlying services. The owner is responsible for ensuring the threat model exists, is current, and covers both the AI component and the surrounding workflow. For agentic or tool-using systems, that review should include whether the model can call APIs, retrieve internal data, write back to systems, or trigger downstream automation.

Practitioners should anchor the review in known attack patterns and control baselines. MITRE ATLAS adversarial AI threat matrix is useful for mapping model poisoning, prompt injection, evasion, and extraction scenarios. For operational response, CISA cyber threat advisories help teams connect emerging attacker tradecraft to practical defensive updates. A good review usually covers:

• Data inputs, training sources, and whether any untrusted content can influence outputs.
• Privileges granted to the AI component, including secrets, tokens, and API keys.
• Human approval points before the system can take irreversible action.
• Logging, monitoring, and rollback paths if the model behaves unexpectedly.
• Ownership for ongoing tuning, retraining, and re-validation after changes.

Security review should also map AI behaviour to established controls such as NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where access control, audit logging, and configuration management are involved. For autonomous workflows, the CSA MAESTRO agentic AI threat modeling framework gives a useful structure for separating model risk from orchestration risk. These controls tend to break down when AI is bolted onto legacy applications with shared service accounts and no explicit approval path, because accountability is then dispersed across teams with no single gatekeeper.

Common Variations and Edge Cases

Tighter AI governance often increases delivery overhead, requiring organisations to balance faster feature release against the cost of a deeper review cycle. That tradeoff becomes more pronounced when teams are experimenting with prototypes, using third-party models, or embedding AI into customer-facing flows where speed is commercially important. Best practice is evolving, but current guidance suggests that “temporary” AI features still need an accountable owner and a documented threat review before production use.

There are a few important edge cases. If a third-party vendor supplies the model but the organisation exposes it through its own product, accountability does not disappear into the supplier contract. The internal product owner still owns the business decision to adopt it, while security owns the approval criteria. If the AI only summarizes non-sensitive text, the threat model may be lighter, but it is not absent. If the system can access internal knowledge bases, ticketing tools, or production APIs, the review must be much stricter because the AI now sits inside the trust boundary.

The same principle applies to agentic AI. Where the model can act, not just generate text, the accountability question expands to include the human who authorised that autonomy, the engineer who exposed the tool interface, and the security function that should have validated the guardrails. In AI incidents, organisations usually discover ownership gaps after the workflow has already been exploited, not at the moment the code was merged.

Related resources from NHI Mgmt Group

• How should security teams use AI for browser threat hunting without creating false confidence?
• Who is accountable when an AI vendor changes an agent's capabilities without notice?
• Who is accountable when an AI workflow touches CUI without a distinct identity?
• What breaks when AI agents are added to an IAM programme without new controls?