Who is accountable for LLM data leakage in an organisation?

Why This Matters for Security Teams

LLM data leakage is rarely just a “model problem.” It is usually an identity, access, and governance failure that shows up in the model layer because the organisation did not control who could submit prompts, what data could be included, or which tools the model could reach. That makes accountability harder than traditional data loss cases, especially when developers, analysts, and AI platform teams all touch the same workflow.

Current guidance from NIST AI RMF and the OWASP Agentic AI Top 10 points toward shared governance, but shared does not mean ambiguous. The control owners must be identifiable before leakage occurs, not after an incident review. NHIMG’s LLMjacking: How Attackers Hijack AI Using Compromised NHIs shows how quickly exposed credentials become usable in practice: when AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes. In practice, many security teams encounter LLM leakage only after prompts, files, or secrets have already been copied into an external system, rather than through intentional governance.

How It Works in Practice

Accountability for leakage should be mapped to the control plane that governs identity, data, and AI access paths. IAM owns authentication and entitlement boundaries. Security owns detection, policy enforcement, and incident response. Data governance owns classification, retention, and approved handling rules. AI platform or engineering teams often operate the runtime, but they should not be the only line of defence.

The practical model is to treat each LLM interaction like a sensitive data transfer request. Before data reaches the model, the organisation should know: who is requesting access, what data classes are involved, whether the prompt contains secrets or regulated data, and whether the target model is approved for that classification. Policy-as-code can enforce this at request time, while logging and DLP controls provide evidence afterward. This is where workload identity, short-lived credentials, and context-aware authorisation matter more than static role assignment.

For agentic or tool-using systems, the risk expands because the model may call APIs, query repositories, or chain tools without a human seeing each step. That is why NHIMG’s 52 NHI Breaches Analysis and the external CSA MAESTRO agentic AI threat modeling framework both emphasise runtime control, not just policy documents. A mature operating model usually includes:

• data classification rules tied to prompt and file submission paths
• central approval for which models can receive which data types
• temporary credentials for tools and connectors used by the LLM
• audit logs that link user identity, model request, and downstream action
• incident ownership that is assigned before a leak, not negotiated after one

These controls tend to break down when shadow AI tools bypass the approved submission path, because no single team can see the request, the model, and the destination at the same time.

Common Variations and Edge Cases

Tighter leakage controls often increase friction for developers and analysts, requiring organisations to balance speed against assurance. That tradeoff is real, especially where teams use external LLM services, multi-agent workflows, or embedded copilots inside business applications. Best practice is evolving, and there is no universal standard for this yet, but current guidance suggests that accountability should follow the decision point where data is allowed into the AI workflow.

Edge cases create ambiguity. In a vendor-hosted SaaS copilot, the application owner may control configuration while the security team controls policy. In a custom agent stack, platform engineering may own the runtime, while data governance sets the boundaries for what can be sent. If the organisation cannot answer who approved the model, who classified the data, and who can revoke access, accountability has not been operationalised.

NHIMG’s McKinsey AI platform breach and the NIST AI 600-1 Generative AI Profile reinforce the same operational lesson: the organisation needs named owners for approval, monitoring, and response across the full data path, not just a general “AI team” label.

Related resources from NHI Mgmt Group

• Who is accountable when an LLM leaks data after following malicious instructions?
• Who is accountable when a cloud misconfiguration exposes production data?
• Who is accountable when an agentic browser changes data or permissions?
• Who is accountable when unsanctioned SaaS stores sensitive business data?