How should teams govern GenAI systems under the EU AI Act?

What “govern the whole system” means under the EU AI Act

Under the eu ai act, governance is broader than model selection or prompt policy. Teams need a system view that includes the base model, orchestration layer, retrieval components, tools, data sources, outputs, and the people who own each control. That is where EU AI Act regulatory framework becomes the practical reference point, because it ties obligations to how the AI system is deployed and used, not just what model sits underneath.

This is especially important when a GenAI application can call tools, retrieve external information, or trigger downstream actions. Those capabilities change the system’s actual behaviour and may change its risk classification, documentation needs, and oversight obligations. Governance therefore has to track the living system, not a frozen design document.

For teams building operational controls, the useful question is whether the system still behaves as originally assessed. If the answer is no, the governance record is already stale. Ownership, intended purpose, data flows, and human oversight must be versioned so they can be reviewed when the system changes, not only at launch.

Which controls matter most in practice?

The most important controls are the ones that let you explain and constrain the system end to end. That means documenting who owns the application, what the system is allowed to do, which data it can ingest, what tools it can invoke, what logs are retained, and what humans can override or stop it. The NIST AI Risk Management Framework and the NIST AI 600-1 GenAI Profile are useful because they push teams toward governance, mapping, monitoring, and incident handling for GenAI as an operating system of controls.

Teams should also treat retrieval and tooling as part of the governed surface. A model with no tool access is very different from the same model connected to databases, ticketing systems, payment actions, or code deployment. Once retrieval or agentic tooling is added, the question is no longer only about content quality. It is about delegated authority, action boundaries, and whether the system can be safely observed when it takes steps on its own.

That is why ISO/IEC 42001:2023 AI Management System Standard is a strong governance companion for this question. It reinforces repeatable ownership, documented controls, and management review, which are essential when a GenAI service evolves faster than a traditional application.

Why compliance stays continuous after launch

GenAI governance cannot be a one-time approval. New data sources, model swaps, prompt changes, retrieval updates, or agent integrations can all alter output behaviour and user impact. The NIST Cybersecurity Framework 2.0 is relevant here because it reinforces an ongoing govern, identify, protect, detect, respond, and recover cycle, which mirrors how AI systems need to be monitored in production.

The continuous-compliance issue is strongest where the system’s permissions expand over time. A harmless assistant becomes a materially different system when it can see internal records, call external services, or generate actions that affect customers or operations. Teams should therefore re-run reviews whenever capabilities, data, or authority change, not only when the model provider releases a new version.

For organisations that need a control baseline for system changes, the NIST SP 800-53 Rev 5 Security and Privacy Controls is useful for anchoring access, audit, configuration, and monitoring expectations around the application environment that hosts the AI system. The point is not to treat the model in isolation, but to govern the full stack that enables its behaviour.

Risk and Threat Considerations

GenAI risk under the EU AI Act often grows after deployment because the live system can accumulate new permissions, new data sources, and new workflows. If those changes are not re-reviewed, the organisation can end up with an AI service that behaves outside the conditions that were originally assessed, which creates compliance, governance, and operational exposure.

Failure mechanism: teams assess the model once, but fail to track changes to retrieval, tools, data access, or human oversight; the system then drifts into a materially different operating state without updated controls or documentation.

Impact: the organisation may lose visibility into what the system can do, where its inputs came from, and who is accountable for the resulting outcomes, which can increase regulatory, security, and business risk at the same time.

Practitioner Guidance

What to prioritise: establish a system inventory that records owner, intended use, connected data sources, tool access, monitoring, and human override points. If you cannot describe those elements clearly, the system is not ready for stable governance.

What to verify: verify that every material change, including retrieval updates, agent enablement, or connector additions, triggers a fresh review of risk, oversight, and logging. The most common mistake is assuming the model version is the only thing that matters.

What good looks like: the compliance record, technical controls, and operational ownership all match the live behaviour of the deployed system, and there is a defined path for re-approval when that behaviour changes.

Practitioner takeaway: under the EU AI Act, govern the AI system as a living service with changing authority, not as a static model artifact.

Related resources from NHI Mgmt Group

• How should security teams implement continuous AI security testing for high-risk systems under the EU AI Act?
• How should security teams govern API keys used for generative AI access?
• How should security teams govern AI assistants that can act inside IAM systems?
• How should security teams govern AI systems that can act without human approval?