GenAI security risks are outpacing enterprise governance controls

At a glance

What this is: This analysis argues that GenAI introduces new security risk categories, from prompt-based data leakage to deepfakes, hallucinations, prompt injection, and over-reliance on AI output.

Why it matters: IAM, security, and governance teams need to treat GenAI as an access, data, and decision-control problem because unmanaged model use can bypass existing approvals, validation, and least-privilege assumptions.

By the numbers:

• 88% of organizations now use AI in at least one business function.
• Only 24% have robust AI risk governance frameworks.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read Proofpoint's analysis of GenAI security risks and practical mitigation steps

Context

GenAI is no longer a fringe experimentation layer. It is entering day-to-day business workflows, which means security teams now have to govern prompts, outputs, and AI-connected systems as part of the enterprise attack surface. The primary issue is not the model itself, but the way it inherits trust from users, documents, APIs, and collaboration tools without strong controls.

For identity and governance teams, the important shift is that GenAI can act as an access amplifier. When users paste sensitive material into unmanaged tools, or when AI systems connect to internal resources with broad permissions, the model becomes a new path for data exposure, policy bypass, and decision error. That makes AI governance an identity, access, and data-control problem, not just an AI operations issue.

Key questions

Q: What fails when organisations let GenAI read and act on untrusted content?

A: Untrusted content can steer the model into leaking data, bypassing policy, or producing instructions that look legitimate but are operationally unsafe. The failure is not just technical, it is a control failure caused by treating inputs as trusted. Organisations need content sanitisation, human verification, and connector scoping before AI is allowed to influence decisions.

Q: Why do AI tools create governance risk even when humans stay in charge?

A: AI tools create risk when they reshape the real decision path without changing formal ownership. Teams may rely on output that is faster, more persuasive, or less scrutinised than human work. The result is weaker accountability, not because AI is autonomous, but because the control process stops matching how decisions are actually made.

Q: How do security teams know whether AI access is actually working safely?

A: Look for three signals: complete discovery of the AI estate, clear mapping of source data to each system, and logs that prove what was accessed and why. If any of those are missing, the control environment is incomplete. Safe AI access is evidenced, not assumed.

Q: Who is accountable when an AI agent triggers a banking error or compliance breach?

A: Accountability sits with the institution that granted the agent access, defined its scope, and failed to govern its actions. Banking regulators will focus on whether the bank can prove effective oversight, traceability, and control over both human prompts and autonomous actions.

Technical breakdown

Prompt injection and indirect instruction hijacking

Prompt injection is the practice of embedding malicious instructions inside content that an AI system ingests, such as emails, PDFs, web pages, or documents. The model may treat those instructions as part of the task, which can cause it to reveal data, ignore policy, or perform unintended actions. Indirect prompt injection is especially difficult because the attacker does not need direct access to the assistant interface; they only need the AI to read hostile content through normal business workflows. Practical implication: treat all ingested text as untrusted input and sanitize content before it reaches AI-connected workflows.

Practical implication: Sanitize and classify content before it reaches AI-connected workflows.

Hallucinations, validation gaps, and control failure

Hallucinations are plausible but incorrect model outputs. In enterprise settings, the risk is not only factual error. It is the decision chain that forms around the output. If a model drafts a policy, suggests a configuration, or summarises a compliance requirement and a human accepts it without verification, the error can become an operational control failure. This matters most where AI output influences access decisions, financial approvals, incident triage, or regulatory reporting. Practical implication: AI output must remain advisory until it passes human or automated verification.

Practical implication: Require verification before AI output influences access, finance, or compliance decisions.

Shadow AI, data leakage, and access scope

Shadow AI appears when staff use unmanaged GenAI tools outside approved governance. The risk is not just that sensitive data leaves the organisation. It is that the organisation loses visibility over where prompts, outputs, and attachments go next, including retention, reuse, or exposure through connected plugins. Once AI tools are linked to internal systems, access scope becomes the decisive issue. Broad connector permissions can turn a simple productivity tool into a high-impact data access channel. Practical implication: map every AI tool to its data sources, permissions, and retention behaviour.

Practical implication: Map every AI tool to its data sources, permissions, and retention rules.

Threat narrative

Attacker objective: The attacker wants to extract data, manipulate decisions, or trigger unauthorized actions through trusted AI workflows.

1. Entry begins when an employee, contractor, or attacker places sensitive text, poisoned content, or malicious instructions into a GenAI workflow.
2. Escalation occurs when the model or its integrations inherit trust from the input and act on data, policy, or system access that should have remained constrained.
3. Impact follows when sensitive information leaks, decisions are corrupted, or impersonation and social engineering succeed at scale.

NHI Mgmt Group analysis

GenAI governance debt is now a control problem, not a policy problem. The article’s core point is that adoption has raced ahead of oversight, and that mismatch creates a durable governance deficit. Security programmes that treat AI as an application feature miss the fact that prompts, outputs, and connectors create new trust boundaries. Practitioners should anchor AI governance in access control, validation, and data handling rather than standalone guidance documents.

Prompt injection is the AI equivalent of untrusted content execution. When a model reads hostile instructions embedded in routine business content, the system can be steered without a visible exploit chain. That makes provenance, sanitisation, and content inspection part of the AI security model, not optional hardening. Teams should map this to NIST AI RMF and, where the workflow is adversarial, to MITRE-ATT&CK behaviours around execution, credential access, and exfiltration.

Identity controls are the missing layer in most GenAI deployments. AI systems that can reach internal data stores, collaboration suites, or APIs become non-human access consumers, even when the vendor presents them as productivity tools. That means entitlement scope, approval lifecycle, and auditability must be applied to AI integrations just as they are to service accounts and bots. Practitioners should govern GenAI connectors as non-human identities with bounded privilege and reviewable access.

Deepfakes turn trust in identity signals into a fraud exposure. Voice and video impersonation weakens controls that were built for human recognition, especially in finance, HR, and executive approval workflows. The governance response is not to rely on a single stronger control but to recompose verification around out-of-band confirmation, policy-based approval, and fraud-aware escalation paths. Practitioners should treat synthetic identity as a standing exception to informal approval processes.

AI safety and NHI governance are converging around the same question: who, or what, is allowed to act? GenAI creates runtime behaviour that looks autonomous even when the underlying model is not. That blurs the boundary between model governance and identity governance, especially when AI tools can call other systems. Teams should decide whether their AI programme is governed by usage policy alone or by enforceable identity and privilege constraints.

What this signals

Shadow AI becomes an identity problem as soon as a model or assistant can reach internal resources. The programme signal is clear: map AI connectors the same way you would map service accounts, API keys, and privileged bots. Where AI systems can call other systems, least privilege, logging, and approval lifecycle become the difference between managed adoption and silent access expansion.

The operational pattern to watch is runtime trust expansion. Once prompts, files, and chat interfaces are allowed to trigger actions, governance has to move from policy statements to enforceable controls, especially in workflows that touch finance, HR, customer data, or incident response. NIST AI RMF and NIST SP 800-53 both support this shift toward accountable, auditable control design.

For practitioners

• Define prohibited prompt content Publish a data-classification policy that explicitly bans source code with secrets, customer records, internal strategy, and regulated personal data from public or unmanaged GenAI tools.
• Inventory AI tools and connectors Build a live register of sanctioned and shadow AI services, then map each one to its data sources, plugins, and retained prompt or output records.
• Apply least privilege to AI integrations Limit every AI connector to the minimum systems and datasets required, and remove broad access to mailboxes, document stores, and cloud APIs where not essential.
• Require human verification for high-risk outputs Block AI-generated recommendations from directly influencing access decisions, financial approvals, compliance text, or configuration changes until they are validated.
• Test for prompt injection and synthetic fraud Red-team AI workflows with malicious documents, hidden instructions, and deepfake scenarios to measure whether controls stop content-driven manipulation before business impact.

Key takeaways

• GenAI is creating new security risk classes because models process untrusted content, generate non-deterministic output, and connect to sensitive systems.
• The evidence points to a governance gap: AI adoption is widespread, but oversight, auditability, and access discipline remain weak.
• The right response is to govern AI like a non-human access layer, with least privilege, input sanitisation, output verification, and clear accountability.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Hallucination: An AI-generated response that is fluent and plausible but incorrect, unsupported, or fabricated. For identity and governance teams, hallucination is a control issue because users may act on it as if it were trusted system output, especially when the chatbot sits inside an operational workflow.
• Synthetic Identity: A synthetic identity is a software-based actor that can authenticate, request access, and execute actions without being a human user. In practice, this includes AI agents, bots, service accounts, tokens, and other machine identities that need clear ownership, scope, and revocation.

What's in the full article

Proofpoint's full blog covers the operational detail this post intentionally leaves for the source:

• Detailed best-practice guidance for data controls around prompts, including acceptable-use policy language and DLP enforcement patterns.
• Concrete examples of monitoring signals for shadow AI, prompt submissions, output records, and plugin activity across collaboration tools.
• Stepwise recommendations for least-privilege access to AI integrations, including where to audit connector permissions.
• The 30-day rollout plan with week-by-week actions for AI inventory, monitoring, and governance testing.

👉 Proofpoint's full post covers threat examples, governance actions, and a 30-day rollout plan for GenAI controls.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity with a practitioner focus. It helps security teams apply identity controls to service accounts, bots, and AI-connected systems with clearer accountability.