Enterprise generative AI security needs governance, not just scale

At a glance

What this is: Fiddler’s article argues that enterprise generative AI creates governance gaps around emergent behaviour, explainability, and misuse, not just a scaling challenge.

Why it matters: IAM, security, and AI governance teams need to treat generative AI as a controlled system with model, data, and lifecycle boundaries, or risk losing oversight of how outputs are produced and used.

Context

Generative AI is not just a larger version of older automation. As models become more capable, the governance problem shifts from basic deployment to whether the organisation can define control boundaries, explain outputs, and constrain use across the AI lifecycle.

That creates a genuine intersection with identity and access management when AI systems consume data, expose functionality, or operate with delegated rights. The article is mainly about AI governance, but the identity question is whether humans, services, and emerging AI agents are all operating under enforceable access and audit rules.

Key questions

Q: What breaks when generative AI is allowed to execute security actions without governance?

A: The organisation loses traceability, consistent decision boundaries, and clear accountability for remediation steps. A model can appear helpful while still taking actions that exceed intent or policy. Without scoped permissions and review points, incident handling becomes fast but difficult to audit or reverse.

Q: Why do explainability and provenance matter for generative AI risk?

A: They let teams trace how a model reached a result, which is necessary for investigation, audit, and control validation. Without that traceability, security and AI teams cannot reliably test whether the model behaved safely, whether unsafe data influenced the output, or whether a downstream decision used a flawed response.

Q: How should security teams govern generative AI once it becomes part of daily operations?

A: Treat generative AI as an access-bearing workflow, not a standalone tool. Map what data it can reach, who owns the permissions behind it, and where human review still matters. If the AI is drafting, translating, or analysing sensitive material, the governance focus should be on entitlements, accountability, and monitoring rather than the model itself.

Q: When does function creep turn generative AI into a higher-risk system?

A: When a model moves from a narrow internal task to customer-facing, decision-support, or automated action without a matching increase in controls. Risk rises when the organisation expands the model’s role faster than it expands logging, approval, review, and containment.

Technical breakdown

Emergent model behaviour changes the governance problem

Large language models do not scale capability in a straight line. As parameter counts and training data grow, models can display emergent behaviour, meaning new abilities appear at thresholds that are hard to predict from size alone. That makes traditional approval logic weak, because a model can behave differently after retraining, prompt changes, or context expansion even when the deployment label stays the same. Governance therefore has to follow the model’s operating conditions, not just its version number or release date.

Practical implication: track model changes as control changes, not cosmetic updates.

Explainability and provenance are control requirements, not reporting extras

Explainability is about being able to trace why a model produced a specific output, while provenance records where the model, data, and prompt context came from. In enterprise generative AI, those are operational controls because they determine whether a team can investigate bad outputs, identify unsafe training data, and understand how a result was shaped. Without that traceable path, monitoring becomes reactive and model assurance depends on guesswork. This is especially important where LLMs influence business decisions or support user-facing workflows.

Practical implication: require output traceability before allowing AI into decision-support paths.

Misuse and data poisoning are lifecycle threats

The article highlights data poisoning, dual use, and function creep as core security concerns. Data poisoning corrupts the model or its surrounding context by injecting harmful or misleading inputs, while misuse happens when a model is repurposed for harmful or unintended tasks. Function creep is the gradual expansion of use beyond the original approval scope. Together, they show that AI security is not just model hardening. It is lifecycle governance over training data, prompts, deployment scope, and permissible use, with human control still required at key decision points.

Practical implication: bind model access, use cases, and data sources to explicit governance approvals.

Threat narrative

Attacker objective: The attacker aims to distort model outputs, degrade trust, or abuse the model as a scalable generator for harmful content and misinformation.

1. Entry occurs when malicious or untrusted data is introduced into the model training set, retrieval layer, or prompt context.
2. Credential or control abuse is not the main mechanism here; the exploit path is manipulation of model behaviour through poisoned inputs or unsafe reuse.
3. Impact follows when the model produces unreliable, toxic, or misleading outputs that can be used for fraud, misinformation, or unsafe decisions.

Breaches seen in the wild

• Hugging Face Spaces breach 2024: Unauthorised access to Hugging Face Spaces may have exposed secrets users stored for AI apps; tokens were revoked and org tokens removed.
• Gemini AI Breach — Google Calendar Prompt Injection: Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Governance debt is now an AI security problem. Enterprise generative AI exposes a gap between what teams approve and what models can actually do once they are retrained, prompted, or embedded into workflows. That gap grows when organisations treat model scale as progress but leave provenance, approval scope, and output accountability informal. The result is a control environment that cannot reliably explain or constrain model behaviour, and practitioners need governance that follows runtime reality.

Explainability is part of security engineering, not just model transparency. If a team cannot trace why a model produced an output, it cannot confidently investigate harm, validate reliability, or separate model failure from data failure. That matters for AI governance, but it also intersects with IAM when model outputs trigger downstream access, workflow execution, or agentic action. Practitioners should treat traceability as an operational control that supports audit and containment.

Function creep is the quiet failure mode in enterprise AI adoption. A generative AI use case often starts as a narrow internal assistant and expands into customer, decision-support, or content-generation roles without equivalent governance. That expansion changes the risk profile more than the model architecture does. The real issue is not whether the model is large enough, but whether the organisation has defined where AI is allowed to act and where human approval still matters.

Model abuse should be governed as an access problem as much as a safety problem. When prompts, tools, data sources, and outputs are loosely connected, the AI system can become a high-throughput interface to sensitive information or operational actions. That creates an identity and privilege issue even when the article is framed as AI safety. Practitioners need to govern who can invoke the model, what it can reach, and which actions remain off limits.

From our research library:

• According to Gartner, worldwide spending on generative AI is set to reach $644 billion in 2025, a nearly 77% year-over-year increase.

What this signals

AI governance debt will accumulate faster than model risk registers can absorb it. The next control gap for most enterprises is not access to a model, but the lack of a documented boundary around what the model may read, recommend, and trigger. When those permissions are spread across prompts, connectors, and downstream workflows, security teams need to govern the system as a privileged runtime, not a chatbot.

Trust in enterprise generative AI will increasingly depend on lifecycle evidence. Organisations will need to prove what data influenced a model, which version produced the output, and who had authority to connect that model to sensitive systems. That is where AI governance meets identity governance, because delegated access without lifecycle review turns a model into an uncontrolled decision layer.

For practitioners

• Define model governance boundaries Document approved training data, allowed prompts, permitted use cases, and prohibited outputs before deployment expands beyond a pilot.
• Require traceable model outputs Log prompts, retrieved context, model version, and downstream action so investigators can reconstruct why a response was produced.
• Treat AI access as a privileged pathway Limit who can connect models to tools, data stores, and customer workflows, and review those rights as privileged access rather than general application access.
• Test for poisoning and misuse scenarios Include poisoned data, prompt abuse, and function creep in red-team and assurance testing so unsafe behaviour is found before scale amplifies it.

Key takeaways

• Enterprise generative AI introduces governance risk because model behaviour can change in ways that are not obvious from scale alone.
• Explainability, provenance, and lifecycle controls are necessary if organisations want to investigate, constrain, and trust model outputs.
• Security teams should govern prompts, tools, and data access around AI systems as privileged pathways rather than assuming the model itself is the only risk.

Key terms

• Emergent Behaviour: Emergent behaviour is when a model shows new capabilities that are not obvious from its size or training recipe alone. In generative AI, this creates assurance problems because performance can change in unexpected ways as scale increases, making static testing an incomplete control.
• Model Provenance: Model provenance is the evidence chain showing where an AI artefact came from, how it was modified, and whether the version in use is the one that was approved. For AI security teams, provenance is the control that turns trust from assumption into verification.
• Function Creep: Function creep occurs when data collected for one purpose is gradually reused for broader or unrelated purposes. It often appears during project expansion, new integrations, or changing business objectives. In a DPIA, teams should document purpose limits and controls that prevent processing from drifting beyond the original lawful basis and scope.
• Data Poisoning: The deliberate or accidental contamination of a data source that influences system behaviour. In AI environments, poisoned content can alter retrieval results, generated answers, or downstream decisions, which makes write access, change monitoring, and source integrity part of the identity control problem.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management in the context of modern control design. It helps identity and security practitioners connect delegated access, runtime trust, and lifecycle oversight to the systems they govern.