Because the risk is no longer confined to the model itself. LLMs are now part of user workflows and application paths, so security teams must govern access, data movement, auditability, and accountability around the interactions, not just the model endpoint.
Why the governance problem expands beyond the model
GenAI stops being a model-only issue once it is embedded in workflows that read, transform, or generate business content. That means the governance question shifts from “Is the model secure?” to “Who can use it, what data can it see, what actions can it trigger, and how do we prove those actions were appropriate?” The control surface is the interaction layer, not just the endpoint.
That shift is why NIST AI 600-1 GenAI Profile matters here: it treats GenAI risk as a lifecycle and governance problem, not a narrow model-hardening exercise. In practice, the organisation needs policy around permitted use, data handling, review points, and escalation when GenAI output influences decisions or customer-facing actions.
For teams building an operating model, Identity Security Programme Guide is useful because genai governance often lands in the same decision space as access governance: ownership, RACI, exception handling, and control accountability. The practical issue is not whether the model is clever, but whether the organisation can assign responsibility for the surrounding access and data controls.
Why data movement and auditability become governance issues
Once users can paste prompts into an application, or an application can call a model on their behalf, data starts flowing across boundaries that were not part of traditional model security. Sensitive content may enter prompts, be reflected in responses, or be routed into downstream systems. Governance therefore has to define what data is allowed, where it can persist, and what records must exist to reconstruct the interaction later.
That is why auditability is not a “nice to have” feature. If the organisation cannot prove which user, workflow, or system submitted the prompt, what context was supplied, what response was returned, and what happened next, it cannot reliably investigate misuse, support accountability, or prove policy compliance. Logging must be designed for reviewability, not just observability.
NIST AI Risk Management Framework and ISO/IEC 42001:2023 AI Management System Standard both reinforce this broader governance view. Together, they point practitioners toward documented controls for transparency, traceability, accountability, and ongoing risk treatment rather than one-time technical approval of a model.
Why accountability has to cover people, processes, and connected systems
GenAI creates governance complexity because its outputs are often mediated by other systems. A chatbot may answer a user, but a workflow may also route the answer into ticketing, CRM, code generation, customer communications, or approvals. That means the organisation must govern not only model behaviour but also the consuming application, the permissions behind it, and the humans who can invoke it.
This is where the boundary between “AI use” and “business process” disappears. A prompt can become a decision input, a generated response can become an operational action, and a low-friction interface can hide the fact that the organisation has effectively delegated part of the process to software. Governance must therefore establish which actions remain advisory, which require human review, and which are prohibited altogether.
NIST Cybersecurity Framework 2.0 is a useful umbrella here because it frames the issue across govern, identify, protect, detect, respond, and recover. For GenAI, the practical move is to map each workflow to an owner, define approval thresholds, and make sure the organisation can detect when a model interaction has crossed into business-impacting action.
Risk and Threat Considerations
GenAI governance fails when organisations treat the model as the only asset worth securing. The bigger exposure is that prompts, outputs, and tool calls can move sensitive data, create unauthorized actions, or blur accountability across systems and teams.
Failure mechanism: A user-facing or embedded GenAI capability can inherit access from the surrounding application, then use that access to expose data, alter records, or trigger actions without a clear audit trail or approval boundary.
Impact: The result is governance failure, not just model compromise: data leakage, unreviewed decisions, weak non-repudiation, and difficulty proving which action was taken by whom, through which workflow, and under what authority.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST CSF 2.0 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | GenAI governance requires lifecycle risk management beyond model security. |
| Recommendation — Establish governance, map GenAI risks, and monitor controls across the system lifecycle. | ||
| ISO/IEC 42001:2023 | AI management system | The question is about organisational AI governance and accountability. |
| Recommendation — Implement an AI management system with defined roles, controls, and review processes. | ||
| NIST CSF 2.0 | GV.RM-01 — Risk Management Strategy | GenAI introduces cross-workflow risk that must be governed as part of enterprise risk. |
| PR.AA-05 — Identity Management, Authentication, and Access Control | GenAI governance depends on controlling who can use the system and what it can access. | |
| DE.CM-01 — Security Continuous Monitoring | Auditability and monitoring are essential for GenAI interactions and downstream actions. | |
| Recommendation — Define risk tolerance and apply it to GenAI workflows, data access, and approvals. Restrict GenAI access to approved users, data, and tools. Continuously monitor GenAI prompts, outputs, and tool use for policy breaches. | ||
Practitioner Guidance
What to prioritise: Start with the workflows that can expose sensitive data or trigger downstream action, not with the model itself. If a GenAI feature can read internal content, create customer text, or call tools, it needs governance as a business process.
What to verify: Confirm that each GenAI use case has a named owner, a defined data boundary, a retention rule for prompts and outputs, and a review path for high-impact actions. If you cannot reconstruct the interaction, you cannot govern it reliably.
Decision rule: If the GenAI output can change a record, send a message, approve a request, or inform a regulated decision, require explicit human or policy control before release. If it is only advisory, keep it tightly scoped and logged.
Practitioner takeaway: GenAI governance is about controlling the interaction surface, because that is where access, data, accountability, and business impact converge.