Top 10 AI security incidents show why prompt-layer controls matter

At a glance

What this is: Akto surveys 10 AI security incidents from 2025-2026 and finds that copilots, agents, MCP servers, and RAG pipelines are being attacked through prompt injection, authentication bypass, and command execution paths.

Why it matters: This matters because IAM, AI governance, and security teams now have to treat prompts, tool access, and agent identity boundaries as part of the attack surface, not as soft application inputs.

Context

AI security incidents now look less like model failures and more like governance failures around what AI systems trust, what they can invoke, and where their outputs can reach. In this article, the primary problem is the collapse of the boundary between untrusted content and privileged execution inside copilots, agents, MCP servers, and RAG-enabled workflows. For identity and access teams, the important question is no longer only how a model behaves, but what identities, tools, and permissions it can reach once prompt-layer trust breaks down.

The incidents Akto groups together show a common pattern: the AI layer is increasingly connected to enterprise data, developer tooling, and automation privileges. That creates a genuine identity and authorization problem because AI systems often inherit access from users, service accounts, or platform defaults without the same controls applied to human sessions. For practitioners, this is an IAM and NHI governance issue as much as an AI security issue.

Key questions

Q: What breaks when an AI system cannot separate instructions from data?

A: The trust boundary breaks first, then the policy boundary follows. A retrieved document, email, or webpage can be interpreted as an instruction instead of evidence, which lets adversaries influence model behavior without ever touching the user interface. Once that happens, traditional keyword filters and prompt rules become incomplete because they are defending the wrong layer.

Q: Why do agentic AI workflows require stronger identity controls than a single-chatbot model?

A: Agentic workflows can take actions across systems, not just generate text, so they inherit real operational risk. When agents can search, retrieve, and book on behalf of users, over-permissioning or weak scoping can expose sensitive data and APIs. Identity controls must therefore govern runtime access, prompt boundaries, and revocation, not just model output.

Q: How do security teams know if an AI workflow is too exposed?

A: Security teams should look for three signals: the assistant can read untrusted free text, it can call tools that touch sensitive systems, and its permissions exceed the narrow task it needs to complete. If those conditions overlap, the workflow is already exposed. The risk rises further when logs, tags, or comments feed back into model context.

Q: Should organisations treat MCP-connected AI systems like privileged workloads?

A: Yes. MCP-connected systems can invoke tools, reach data, and trigger actions, which means they function more like privileged workloads than passive applications. They need least privilege, logging, access review, and clear ownership. If a model can act on behalf of a user or service, its permissions deserve the same scrutiny as any elevated account.

Technical breakdown

Why prompt injection behaves like an authorization failure

Prompt injection is not just a content problem. It is a trust-boundary failure where untrusted text is allowed to influence privileged system behavior. In AI applications, the model may be asked to summarise email, ingest a CRM record, or follow instructions embedded in a document, but the runtime does not reliably distinguish user intent from attacker-controlled instruction. That makes prompt injection structurally similar to broken authorisation in application security: the system executes actions or reveals data outside the user’s intended scope. When agents can call tools, query connected services, or forward outputs, the blast radius expands beyond the model itself.

Practical implication: Treat prompt sources, retrieved context, and tool instructions as separate trust zones with enforced policy checks before execution.

How agent and MCP permissions expand the blast radius

Agentic systems and MCP servers turn an LLM from a responder into an actor that can trigger external actions. Once the agent can access files, APIs, shells, or downstream agents, the real risk shifts from text generation to delegated privilege. This is where identity matters: the agent is operating through a non-human identity or a proxy access path that may have broader rights than the immediate task requires. If those permissions are inherited loosely, hardcoded, or shared across instances, one compromised prompt can become lateral movement across tools and workflows. The control issue is not model quality alone, but whether delegated access is tightly scoped and isolated.

Practical implication: Map every agent and MCP integration to explicit entitlements, short-lived credentials, and per-task access boundaries.

Why input sanitization alone does not stop AI supply-chain abuse

Several of the incidents Akto describes show attackers using indirect channels such as poisoned records, branch names, headers, and hidden text. These are supply-chain-style attacks because the malicious payload arrives through content the system already trusts. Traditional sanitization helps, but AI workflows often consume content from many sources before policy enforcement occurs, which means the exploit lands earlier in the chain than developers expect. In practice, the vulnerable point is frequently the connector, framework, or runtime bridge, not just the prompt box. That is why AI security needs the same discipline used for software supply chains and service-to-service authentication.

Practical implication: Validate and constrain every upstream data source before it enters an AI workflow, not just the final model prompt.

Threat narrative

Attacker objective: The attacker wants to turn trusted AI automation into an execution path for data theft, privileged action, or arbitrary command execution.

1. Entry typically begins when attackers place malicious instructions in content the AI system already trusts, such as an email, support ticket, branch name, webpage, or connected record.
2. Credential access or abuse follows when the AI runtime, agent, or connector acts on that content and uses inherited permissions, shared secrets, or delegated access to reach systems it should not control.
3. Escalation occurs when the AI tool chains into other agents, APIs, or internal workflows, turning a single poisoned input into broader data exposure or command execution.
4. Impact is unauthorised disclosure, workflow manipulation, or remote code execution through the AI system's own privileges rather than through a traditional user session.

Breaches seen in the wild

• CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
• SalesBleed Salesforce Agentforce 2026: Three fixed Agentforce flaws let poisoned web leads make AI agents leak CRM data with zero clicks and send phishing under the agent's identity.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt-layer control is now a governance domain, not a model tuning issue: once AI systems ingest untrusted text and then act on it, the security boundary has already moved upstream of the model. The control problem is no longer only output safety, but the integrity of instructions, context, and tool invocation. For identity teams, that means prompt handling must be governed with the same seriousness as privileged application logic.

AI agents create a non-human privilege problem that standard IAM models were not built to absorb: the agent often operates through shared runtime credentials, delegated APIs, or connector accounts that are broader than the task itself. That makes identity scoping, credential isolation, and tool-level authorisation central controls. This is where agent identity and NHI governance converge: the actor may be software, but the rights it inherits still need lifecycle control.

Prompt injection is the AI-era equivalent of trust boundary collapse, but the failure mode is broader than classic application injection: the malicious payload can arrive through content, metadata, retrieval sources, or downstream agents. AI trust boundary collapse: when systems cannot reliably separate instructions from data, every connected tool becomes part of the attack surface. Practitioners should treat any AI workflow with external inputs as an identity-governed execution environment, not a passive assistant.

The most serious AI incidents are now exposing process weaknesses, not just technical flaws: weak inventory, permissive defaults, shared secrets, and unreviewed connector behaviour repeatedly show up as the enabling conditions. That means security leaders should measure AI risk by how well access, tool use, and runtime behaviour are governed across the full stack. The practical conclusion is to manage AI systems as production services with controlled identity, not experimental chat interfaces.

AI security and NHI governance are converging faster than most programmes have adjusted for: once an agent can read, decide, and act, it behaves like a machine identity with business impact. That creates a need for lifecycle controls over credentials, permissions, and isolation boundaries. Practitioners who still treat AI access as a UI concern will miss the governance layer that determines real exposure.

From our research library:

• Half of companies using generative AI will deploy agentic AI by 2027, according to Deloitte's 2025 Technology Predictions.
• Read next: Agentic AI Identity Guide

What this signals

AI incidents are increasingly exposing a governance gap: organisations have wrapped model output controls around systems that can already read, call, and act inside enterprise environments. The security problem now sits at the intersection of application trust, delegated access, and identity control, which is why prompt-layer controls must be designed as part of the runtime boundary rather than added after deployment.

AI trust boundary collapse: when a model cannot distinguish instructions from data, the resulting risk is not just hallucination but unauthorised action. That matters for IAM and NHI teams because 69% of security leaders agree identity management must fundamentally shift to address agentic AI systems, according to the 2026 Infrastructure Identity Survey. The practical response is to govern AI access as a privileged execution problem, not a chat safety problem.

For practitioners

• Map AI workflows to explicit trust boundaries Separate untrusted content, retrieved context, model instructions, and tool execution paths so each boundary has its own policy check and logging point.
• Inventory every agent and MCP integration List each AI agent, connector, and MCP server with its associated account, secret, scope, and downstream system access.
• Replace shared or static credentials Use short-lived, task-scoped credentials for AI runtimes and remove hardcoded secrets that let one compromise spread across instances.
• Harden upstream data sources Sanitize and constrain email, web, CRM, ticketing, and repository inputs before they can influence summarisation, retrieval, or command execution.
• Continuously red team AI toolchains Test prompt injection, indirect injection, authentication bypass, and command execution across the full AI stack, including connectors and runtime dependencies.

Key takeaways

• AI security incidents are now being driven by attacks on the AI stack itself, not just by using AI as an attacker tool.
• The recurring failure pattern is trust boundary collapse across prompts, connectors, retrieved data, and delegated tool access.
• Security programmes need identity-aware control over AI workflows, because the blast radius is defined by what the system can reach, not only by what it can say.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads, causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions, including calling APIs, writing code, and orchestrating other agents, with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to govern machine identities, delegated access, and AI-driven workflows in the same programme.