Generative AI security depends on identity-aware controls and monitoring

At a glance

What this is: Obsidian Security argues that generative AI security failures come from model behaviour, data flow, and access paths rather than conventional code vulnerabilities.

Why it matters: IAM, PAM, and AI security teams need to treat AI agents and GenAI apps as governed identities with monitored privileges, not as ordinary application components.

By the numbers:

• Organizations with comprehensive AI threat management programs report 60% lower mean time to recovery from security incidents than those relying on reactive approaches.

Context

Generative AI security is fundamentally a governance problem because the main failure mode is not code exploitation but manipulation of model behaviour, data inputs, and runtime trust. Traditional security controls were built to inspect software logic, network traffic, or static infrastructure, but AI systems make decisions through probabilistic inference and external data access that can be steered by adversarial prompts or poisoned inputs.

That creates a genuine identity bridge. AI agents, APIs, and model-driven workflows increasingly act with delegated access, which means security teams have to govern who or what is allowed to prompt, retrieve, write, or trigger actions, and under what conditions. The article's starting position is typical of current enterprise deployments: AI is being adopted faster than the control model around it.

The result is a broader control gap across authentication, authorisation, monitoring, and incident response. In practice, GenAI security now sits at the intersection of application security, cloud governance, and identity governance, because the model is only one part of the attack surface; the permissions and data pathways around it are what turn manipulation into impact.

Key questions

Q: What breaks when generative AI can access tools and data without identity controls?

A: The main failure is that model behaviour becomes an access path. If the system can retrieve data, call APIs, or use delegated credentials, then a prompt-level attack can turn into unauthorized action, data exposure, or policy bypass. The control boundary has to move from output filtering to execution authority, token scope, and revocation.

Q: Why do AI agents with excessive privileges create more risk than ordinary chatbots?

A: AI agents can act across tools, APIs, and data sources, so excessive privilege turns one bad interaction into a broad compromise path. A chatbot may misstate an answer, but an over-permissioned agent can retrieve records, alter workflows, or expose secrets. The risk rises with delegated access scope, not model sophistication.

Q: How can teams know whether GenAI monitoring is actually working?

A: Look for prompt-level telemetry, retrieval traces, output logs, and downstream action records that can be joined into a single event chain. If the organisation can only see model uptime or platform health, it is not seeing the security-relevant layer. Effective monitoring answers who used the tool, what data entered it, and what happened next.

Q: Should organisations prioritise least privilege or zero trust for AI systems first?

A: Least privilege should come first because it limits how far a compromised prompt or agent can go. Zero trust then adds verification at each interaction, but verification is far less effective if the underlying permissions are already excessive. The practical sequence is to scope access tightly, then verify continuously.

Technical breakdown

Prompt injection and data poisoning change the threat model

Prompt injection is a form of input manipulation where an attacker supplies text that overrides system instructions or steers the model toward unsafe outputs. Data poisoning attacks corrupt training or fine-tuning data so the model learns the wrong associations, which can remain hidden until the model is used in production. These are not classic code exploits. They target the inference and learning process itself, which is why static scanning and perimeter controls miss them.

Practical implication: Treat model inputs and training pipelines as security boundaries, not just data flows.

Identity controls for AI agents and model access

AI agents often interact with multiple tools and data sources, so they inherit risk from whatever credentials, tokens, or service accounts they can reach. If authorisation is too broad, a successful prompt injection can become a privilege problem, not just a content problem. Identity-aware control means binding access to the specific model, workflow, or task, then limiting what that identity can retrieve or execute. This is where AI governance meets NHI governance.

Practical implication: Scope model and agent permissions to the smallest possible task boundary and monitor delegated access continuously.

Behavioural monitoring and zero-trust runtime assumptions

Because AI models do not behave deterministically, security teams need monitoring that looks for abnormal output patterns, drift, and unexpected tool use rather than only known indicators of compromise. Zero trust in this context means every model interaction is treated as potentially compromised until verified through policy, context, and telemetry. That shifts detection from signatures to behaviour baselines, and from trusting the model boundary to verifying each interaction as it happens.

Practical implication: Instrument runtime telemetry for anomaly detection and verify each AI interaction before allowing downstream action.

Threat narrative

Attacker objective: The attacker aims to steer the AI system into revealing data, producing harmful outputs, or executing actions that should never have been authorised.

1. Attackers begin with prompt injection, data poisoning, or adversarial inputs that target the model's behaviour rather than its code.
2. If the model or agent has excessive delegated access, the manipulated interaction can expose sensitive data or trigger unauthorised actions across connected systems.
3. The impact is compromised outputs, data leakage, or business process abuse that conventional security tooling may not detect in time.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Identity-aware AI security is now a governance requirement, not an optional control layer. Generative AI systems are not protected adequately by code-centric assurance alone because their main risks arise from manipulated prompts, poisoned data, and delegated runtime access. Once an AI system can retrieve data or trigger actions, its identity and privilege model becomes part of the attack surface. Practitioners should treat AI access governance as a first-class control domain.

AI security and NHI governance are converging around delegated access. The article shows why service accounts, API tokens, and agent permissions matter as much as the model itself. If an AI agent can act on behalf of a user or workflow, the question is no longer whether the model is accurate but whether the identity behind it is constrained, monitored, and revocable. That makes NHI lifecycle control central to GenAI risk management.

Prompt injection is really a trust-boundary failure. The named concept here is the verification trust gap: the gap between assuming a model prompt is benign and proving it is safe before action is taken. Traditional security assumes intent is outside the system, but AI systems ingest untrusted instructions continuously. Security teams need policy enforcement at interaction time, not just post-event review.

Runtime monitoring matters because model misuse is often silent until impact. Model inversion, adversarial inputs, and subtle output drift can all appear normal at first glance. That means the control problem is continuous assurance, not one-time hardening. Organisations that cannot observe model behaviour, tool calls, and downstream effects will detect AI compromise too late to limit exposure.

The market is moving toward behavioural controls around AI, not just model governance. That shift aligns with how identity programmes matured around cloud and SaaS: ownership, scope, telemetry, and revocation eventually matter more than architecture diagrams. For security leaders, the implication is clear. GenAI controls need to sit inside IAM, PAM, cloud, and detection workflows rather than as a separate AI-only programme.

From our research library:

• Systems with least-privileged AI access had a 17% incident rate vs 76% for over-privileged systems. Organisations failing to scope AI access properly are 4.5x more likely to experience a security incident, according to the 2026 Infrastructure Identity Survey.
• Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
• Read next: AI Agent Authorisation Guide

What this signals

Verification trust gap: GenAI programmes fail fastest where teams assume a prompt, model output, or tool call is safe until proven otherwise. The better pattern is to treat every interaction as untrusted input and every delegated permission as time-bounded, observable, and revocable.

Security leaders should expect AI governance to merge with identity governance. That means the control questions move from model quality alone to who can prompt, which data can be retrieved, which tools can be invoked, and how quickly the resulting access can be withdrawn if behaviour changes.

For practitioners

• Map every AI system to an accountable identity Inventory each GenAI application, agent, API key, and service account, then assign ownership for approval, rotation, and revocation.
• Enforce least privilege on model-connected access Limit what each agent can read, write, retrieve, or trigger, and separate production data access from experimentation paths.
• Add runtime detection for prompt and output anomalies Monitor for unusual prompt patterns, tool usage, output drift, and repeated access to sensitive datasets through the model interface.
• Test AI workflows with red team scenarios Include prompt injection, data poisoning, and unauthorized tool invocation in adversarial exercises before deployment expands.
• Connect AI telemetry to existing incident response Ensure alerts from model activity, token misuse, and delegated access failures feed the same response path used for identity incidents.

Key takeaways

• Generative AI security fails when teams rely on code-focused controls against threats that manipulate model behaviour, data, and runtime trust.
• Identity-aware access and behavioural monitoring are the two controls that turn AI systems from opaque risk multipliers into governable services.
• The practical priority is to constrain delegated access, observe model interactions continuously, and integrate AI incidents into existing response processes.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Data Poisoning: The deliberate or accidental contamination of a data source that influences system behaviour. In AI environments, poisoned content can alter retrieval results, generated answers, or downstream decisions, which makes write access, change monitoring, and source integrity part of the identity control problem.
• Model Inversion: Model inversion is an attack technique that tries to recover sensitive information from a model by querying it in a targeted way. The risk matters because information hidden in training data, prompts, or internal behaviour can sometimes be inferred from outputs rather than directly accessed.
• Behavioral Monitoring: Behavioral monitoring is the practice of detecting misuse by comparing current activity to established patterns of normal use. For NHI governance, it is essential because valid API credentials can look authentic even when they are stolen and being used for exfiltration or lateral movement.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps security and identity teams build the control model that AI-driven access now requires.