Who should be accountable for LLM security and governance?

Why This Matters for Security Teams

LLM security and governance break down when accountability is assigned only to the team that trained or integrated the model. In practice, the real risk sits in operational behaviour: which data the model can reach, which tools it can call, who reviews outputs, and how incidents are contained. That is why security, compliance, data, legal, and business owners all need named control ownership, especially when LLMs are embedded into workflows that handle sensitive information.

Current guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework points toward shared governance, but shared responsibility only works when each control is explicitly assigned. NHIMG research shows the consequence of weak ownership: in AI Agents: The New Attack Surface report, 80% of organisations said their AI agents had already acted beyond intended scope. That is not just a model issue, it is an operational governance failure.

In practice, many security teams discover ownership gaps only after an LLM has accessed data, exposed credentials, or produced an incident that nobody was formally empowered to contain.

How It Works in Practice

Effective LLM accountability starts by separating model development from production governance. The team that builds the model may own training data, evaluation, and release engineering, but the team that runs the service should own runtime access, policy enforcement, logging, and incident response. That distinction matters because the highest-risk decisions happen at execution time, not in the lab.

A workable operating model usually assigns control ownership across a few functions:

• Security owns prompt, tool, and secret-access controls, plus detection and response.
• Data owners define which datasets the LLM may use, retain, or summarize.
• Compliance and legal approve use cases, retention rules, and review obligations.
• Business owners define acceptable outputs, escalation paths, and human approval thresholds.

Operationally, the governance layer should use policy-as-code and runtime checks rather than static approvals alone. The NIST AI 600-1 Generative AI Profile and CSA MAESTRO agentic AI threat modeling framework both reinforce the need for documented controls, monitoring, and traceability across the lifecycle. For teams mapping these duties to NHI practice, the Top 10 NHI Issues is a useful reminder that identity, secret handling, and auditability are governance problems, not just engineering problems.

Best practice is evolving toward explicit ownership matrices, where every high-risk action has a named approver, a named detector, and a named responder. These controls tend to break down when LLMs are embedded across multiple business units because no single team sees the full access path from prompt to tool execution to downstream data exposure.

Common Variations and Edge Cases

Tighter accountability often increases coordination overhead, requiring organisations to balance faster delivery against stronger approval, review, and escalation discipline. That tradeoff becomes visible in multi-team deployments, where a central platform team runs the LLM stack but product teams define use cases and data owners govern the content. In those environments, a RACI chart is useful, but it is not enough unless it is tied to real controls and incident playbooks.

There is no universal standard for this yet, but current guidance suggests three edge cases need special treatment. First, vendor-hosted LLMs still require an internal owner for data classification and output risk, even when the provider manages infrastructure. Second, retrieval-augmented systems need clear accountability for source quality and access to connected repositories. Third, agentic workflows should be treated more strictly than chat interfaces because tool use, chaining, and autonomous retries expand the blast radius.

NHIMG research on AI agents shows why this matters: 92% of organisations say governance is critical, yet only 44% have implemented policies, which leaves a gap between intent and execution. Security leaders should treat accountability as an operational control, not a policy statement, and should align it with standards such as the NIST Cybersecurity Framework 2.0 and the AI Agents: The New Attack Surface report for evidence of how quickly scope can drift.

Related resources from NHI Mgmt Group

• What is the difference between role-based access and API key governance for NHI security?
• How should security teams use IAST and RASP in NHI governance?
• Why is single-provider AI agent governance not enough for enterprise security?
• How should security teams prepare data access governance before enabling GenAI tools?