GenAI security risks expose the gap in AI agent governance

At a glance

What this is: The article maps the main attack vectors against generative AI systems and argues that AI agent access, monitoring, and governance are now core security controls.

Why it matters: IAM, PAM, and security teams need to treat AI agents and their tokens as governed identities because compromise can create direct paths into sensitive data and downstream systems.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.
• 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes, and as quickly as 9 minutes in some cases.

👉 Read Obsidian Security's analysis of genAI security risks and attack vectors

Context

Generative AI security risk is not just a model integrity problem. It is a governance problem created when AI systems are allowed to interpret untrusted input, reach into sensitive data sources, and act with identities that are not monitored like other privileged accounts.

The article focuses on where traditional cybersecurity assumptions fail: prompt filtering alone does not stop instruction hijacking, and generic monitoring does not reveal whether an AI agent has crossed its intended access boundary. That intersection with IAM and NHI governance is what makes the topic operationally relevant for security teams.

Key questions

Q: How should security teams govern AI agents that can access enterprise systems?

A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring. The control set should include inventory, task-bound credentials, audit trails, and revocation paths. If an agent can call tools or touch production systems, it belongs in the same governance model as service accounts and other machine identities.

Q: Why do AI agents create access risk even when the model is accurate most of the time?

A: Because the risk is not only incorrect reasoning, it is incorrect action. A model that reads untrusted content and can act on it can turn a small mistake into an external email, database write, or escalation, which is why containment matters more than prediction quality.

Q: What do organisations get wrong about prompt injection?

A: They often treat it as a purely content-filtering problem. In practice, prompt injection is an instruction-trust problem that becomes serious when the chatbot can act on behalf of the organisation. The fix is not just blocking bad text. It is constraining what the model can access, what it can call, and what it can change.

Q: When does zero trust fail for AI-enabled data environments?

A: Zero trust fails when verification stops at login and does not continue through data use. In AI-enabled environments, identities can remain authenticated while still copying, transforming, or exposing information that should not leave the governed workflow. Continuous policy enforcement and audit linkage are what keep the model credible.

Technical breakdown

Prompt injection and instruction hijacking in genAI systems

Prompt injection works because large language models are designed to follow instructions inside text, even when those instructions are maliciously embedded in user content, documents, or retrieved context. Attackers can override system intent, steer model outputs, or trigger actions inside downstream workflows. The risk grows when the model is connected to tools, APIs, or internal knowledge sources, because the injected prompt is no longer just a content manipulation problem. It becomes a control-bypass path into enterprise systems.

Practical implication: separate user content from system instructions and apply policy checks before any tool call or data retrieval.

Data poisoning and model inversion as integrity and privacy failures

Data poisoning introduces corrupted or biased records during training or fine-tuning so the model learns the wrong patterns, while model inversion uses crafted queries to reconstruct information from training data or model outputs. Both attacks exploit the fact that model behaviour is shaped by data, not just code. In enterprise settings, this means the attack surface includes datasets, labelling pipelines, embedding stores, and feedback loops. A compromised training pipeline can produce a durable security problem that survives deployment.

Practical implication: treat training and fine-tuning data as security-controlled assets with validation, provenance, and review.

Why AI agents need identity and access governance

AI agents often operate with broad credentials so they can reach mailboxes, SaaS apps, databases, and internal APIs. If those credentials are excessive, stolen, or reused across workflows, the agent becomes a high-impact access path rather than a simple automation layer. This is where NHI governance matters. An AI agent is a non-human identity when it has runtime access to systems, and its entitlements need lifecycle management, monitoring, and revocation just like any other privileged account.

Practical implication: inventory AI agents as governed non-human identities and constrain their access by task, environment, and data sensitivity.

Threat narrative

Attacker objective: The attacker aims to turn a trusted AI workflow into a path for data access, output manipulation, or broader operational abuse.

1. Entry typically begins with malicious input, poisoned data, or a compromised third-party integration that reaches the AI system through a trusted channel.
2. Escalation occurs when the model follows injected instructions or an exposed token grants the attacker access to downstream data sources and connected tools.
3. Impact is data disclosure, manipulated decisions, or unauthorised system actions that can spread across integrated enterprise workflows.

NHI Mgmt Group analysis

AI agent governance debt is now a security issue, not a future concern. The article shows that AI systems can be manipulated through input, data, and access, which means the control problem sits at the identity and workflow layer as much as the model layer. When agents hold broad credentials, their runtime behaviour becomes a privileged access problem. Security teams should stop treating agent oversight as experimental and start treating it as governed production access.

Prompt filtering alone does not solve the trust boundary problem. Prompt injection succeeds because the model cannot reliably distinguish malicious instructions from legitimate content once both are rendered as input. That makes the boundary between user data, retrieved context, and system instructions the real control point. Practitioners should design for data separation, policy enforcement, and tool-call mediation rather than hoping the model will self-police.

Identity-first protection is the missing layer in genAI security programs. The article correctly ties AI risk to access controls, but the deeper point is that AI agents behave like non-human identities with live entitlements. Without inventory, entitlement review, and revocation discipline, organisations cannot tell whether an agent is operating within scope or abusing its own access. Teams should align AI governance with NHI lifecycle control, not just application security reviews.

Supply chain exposure matters because AI systems inherit trust from every dependency. Third-party models, datasets, connectors, and APIs all widen the attack surface and weaken visibility. A poisoned or compromised dependency can bypass local controls because it arrives already trusted by the application. Security leaders should extend supplier risk and software composition practices into AI pipelines, because the model is only as governed as the components feeding it.

Continuous monitoring must shift from model performance to model behaviour. The article emphasises behavioural analysis, which is the right direction because abuse often shows up as unusual query patterns, access bursts, or data movement before it appears as overt compromise. That means security operations needs AI-specific telemetry, not just infrastructure logs. Practitioners should build detection around anomalous action sequences, not just failed authentication or obvious malware signals.

What this signals

AI agent governance will increasingly be judged by access containment, not by model accuracy alone. As soon as agents can reach live systems, the programme risk shifts to entitlement scope, token lifetime, and the auditability of actions. Security leaders should assume that any agent with broad data access becomes a high-value NHI and govern it accordingly.

Only 52% of companies can track and audit the data their AI agents access, leaving nearly half of deployments without basic investigative coverage. That gap will push more organisations toward inventory-first controls, tighter logging, and policy gates around tool execution. The practical test is whether your team can reconstruct an agent's full action chain after an incident.

AI security programmes now need a named control boundary for autonomous access. The question is no longer whether to adopt AI agents, but where to place the line between allowable inference and governed action. Teams that align NHI lifecycle practices, zero trust segmentation, and behavioural monitoring will be better positioned to contain abuse without slowing legitimate automation.

For practitioners

• Inventory AI agents as governed non-human identities Record every agent, connector, token, and service account that can reach enterprise data. Assign an owner, define the allowed scope, and revoke anything that lacks a clear business purpose. This is the only way to distinguish managed automation from shadow AI.
• Block direct prompt-to-tool trust Insert policy checks between model output and downstream tool execution so malicious instructions cannot trigger actions automatically. Separate user content, retrieved context, and system prompts, and require explicit authorization for sensitive operations.
• Validate training and retrieval inputs Apply provenance checks, dataset validation, and change control to training corpora, embeddings, and retrieval indexes. Poisoning often enters through trusted pipelines, so the control needs to start before model training and continue through runtime retrieval.
• Monitor AI behaviour as an access signal Baseline normal agent activity, then alert on unusual tool calls, data access bursts, repeated retries, and access outside expected business hours or workflows. Security teams need telemetry that reveals abuse before sensitive data leaves the environment.
• Apply least privilege and short-lived access to agent tokens Issue the smallest viable permissions for each AI workflow and rotate or expire tokens quickly. Where a task can be completed without standing access, prefer ephemeral credentials and re-authentication at the point of use.

Key takeaways

• Generative AI risks are operational when prompts, data, and access converge inside one workflow.
• AI agents become a governance problem the moment they hold reusable credentials or broad system access.
• Security teams need identity-first controls, behavioural monitoring, and input separation to reduce abuse.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Data Poisoning: The deliberate or accidental contamination of a data source that influences system behaviour. In AI environments, poisoned content can alter retrieval results, generated answers, or downstream decisions, which makes write access, change monitoring, and source integrity part of the identity control problem.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.
• AI Agent Lifecycle Governance: The set of controls that assigns, constrains, monitors, and retires autonomous agents across their full operating life. It extends IAM practice to software that can act on its own, making ownership, scope, auditability, and revocation mandatory rather than optional.

What's in the full article

Obsidian Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Threat examples for prompt injection, model inversion, and poisoning that show how each attack behaves in real environments
• Practical mitigation patterns for integrating AI monitoring with SIEM and incident response workflows
• Implementation guidance for zero-trust access controls, input validation, and behavioural baselining in AI systems
• Additional context on how the vendor frames SaaS and AI security operations around these threats

👉 Obsidian Security's full post covers the attack patterns, defensive controls, and operational blueprint in more detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It gives security and identity practitioners a common control model for governing AI agents and other non-human identities.