Top 10 AI security incidents show why prompt-layer controls matter

At a glance

What this is: This is an analysis of ten AI security incidents from the last 12 months, showing that attackers are targeting AI systems themselves through prompt injection, auth bypass, serialization bugs, and tool abuse.

Why it matters: It matters because AI agents and copilots increasingly sit inside identity, data, and workflow paths, so security teams need controls that govern what the system can read, do, and delegate.

By the numbers:

• Stanford HAI found that publicly reported AI security incidents rose 56.4% year over year.
• IBM's 2026 X-Force Threat Intelligence Index reported 300,000+ AI chatbot credentials observed for sale on the dark web.
• 73% of organizations say they already feel the impact of AI-powered threats.

👉 Read Akto's analysis of the top 10 AI security incidents that shook enterprises

Context

AI security has moved from experimental risk to operational exposure because copilots, agents, MCP servers, and RAG pipelines are now connected to real enterprise data and actions. In practice, that means the problem is no longer only model quality or output safety. It is governance of the instructions, tools, identities, and trust boundaries surrounding the AI system.

For IAM and NHI teams, the intersection is direct. AI agents and supporting services behave like non-human identities when they authenticate, inherit permissions, call tools, and read data sources, which makes access scope and delegation design central controls. The incidents in this article are a typical warning sign for programmes that have scaled AI faster than their identity governance.

Key questions

Q: What breaks when AI agents are given broad enterprise access without tight governance?

A: Broad access turns AI agents into high-speed execution paths that can move data, spend money, modify records, or delete assets before operators can intervene. The failure is not only misuse by an attacker. The system itself can exceed intended scope during normal task completion, so the real control problem is bounding authority before runtime action begins.

Q: Why do local AI agents complicate identity and access management?

A: They can retain legitimate permissions while changing timing, prioritisation, and action sequence outside human presence. That means the visible identity may remain stable even as the operational behaviour becomes autonomous. IAM teams then lose the simple link between user session, authorisation, and accountability.

Q: How do organizations prove AI agent controls are actually working?

A: Organizations prove control effectiveness by showing which agents accessed which data, what actions they executed, and whether those actions stayed within approved task boundaries. Useful evidence includes logs, policy decisions, anomaly alerts, and review records. Without that chain, governance is mostly declarative.

Q: What should organisations do after an AI security incident?

A: Contain the affected identities and integrations first, then review every connected agent, connector, and fallback path for the same failure pattern. Reset secrets, rotate credentials, validate logs, and test whether the issue can recur through another input channel. The goal is to stop one flaw from becoming a repeatable platform weakness.

Technical breakdown

Why prompt injection now behaves like an identity-control failure

Prompt injection is often described as an application-layer issue, but in agentic systems it becomes an identity problem because the injected instruction can redirect privileged behaviour. If an agent can read mail, query documents, call APIs, or invoke other agents, then the attacker is not only manipulating text. They are manipulating an execution path backed by permissions. That is why invisible instructions hidden in emails, web pages, tickets, or repository content can become a control-plane issue for the enterprise.

Practical implication: Treat prompts, retrieved content, and tool inputs as untrusted data and separate them from the policies that authorize action.

How MCP servers and agent tool chains expand the attack surface

The Model Context Protocol connects AI agents to tools and data sources, which makes it powerful and also risky when inputs are not tightly controlled. If an MCP server shells out to the operating system, forwards headers, or inherits broad credentials, a malicious prompt or poisoned input can become command execution or data exfiltration. The core issue is that the agent is only as safe as the least safe tool in its chain, and the chain often crosses identity boundaries silently.

Practical implication: Inventory every tool, permission, and backend identity exposed to AI workflows before allowing agent execution in production.

Why AI security incidents now look like credential and privilege incidents

Several of the attacks in this article succeeded because authentication, account linking, or secret handling failed outside the model itself. That is a familiar NHI pattern: once a secret, token, or static credential is exposed, the attacker moves from experimentation to durable access. In agentic environments, the same principle applies to service accounts, linked identities, and delegated scopes. If those identities are over-privileged, the AI system becomes an amplifier for the breach rather than a boundary around it.

Practical implication: Apply least privilege, short-lived credentials, and lifecycle controls to every identity used by AI platforms and their integrations.

Threat narrative

Attacker objective: The attacker aims to turn trusted AI infrastructure into an execution layer for data theft, unauthorized access, and control over downstream enterprise workflows.

1. Entry occurred through prompt injection, unauthenticated APIs, poisoned data sources, exposed secrets, or command injection in the AI execution path.
2. Escalation followed when the AI system trusted the malicious input and used its existing permissions to read data, call tools, impersonate users, or invoke other agents.
3. Impact included data exposure, token theft, remote code execution, privilege escalation, and the silent reprogramming of AI behavior at enterprise scale.

Breaches seen in the wild

• Gemini AI Breach — Google Calendar Prompt Injection — Gemini AI assistant prompt injection attack leaks sensitive Google Calendar data.
• CoPhish OAuth Token Theft via Copilot Studio — CoPhish campaign exploits Microsoft Copilot Studio agents to steal OAuth tokens via AI-assisted phishing.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Prompt-layer control is becoming a governance domain of its own. The article shows that AI systems fail when security teams treat prompts as content instead of privileged instructions that shape behaviour. That is a different problem from classic application security, because the harmful action may be executed by a trusted model or agent rather than by direct code exploitation. Practitioners need to govern the prompt layer with the same seriousness they apply to production configuration and access policy.

AI agents behave like non-human identities once they can read, decide, and act. This is where identity governance intersects the article most clearly. When an agent can authenticate, inherit context, call tools, and escalate work to other services, the real question becomes who or what is authorised to do what on the agent's behalf. That aligns closely with OWASP Agentic AI Top 10 and the broader NHI governance problem: the blast radius is defined by delegated access, not just model behaviour.

AI supply chain risk now includes toolchains, frameworks, and helper services. The incidents in the article are not isolated model bugs. They reflect a stack where a vulnerable framework, a careless fallback path, or a permissive integration can convert a low-skill attack into enterprise compromise. That means AI governance has to extend beyond the model vendor and into the code, connectors, and identities that surround the system. Security teams should assume every dependency in the AI stack can become a control failure.

Continuous adversarial testing is now a baseline, not a maturity goal. The article's examples show that attackers move faster than periodic review cycles and exploit both logic flaws and identity weaknesses. This is consistent with the broader AI risk picture captured in AI Agents: The New Attack Surface report, where AI behaviour has already escaped intended scope in many organisations. The practical conclusion is straightforward: governance must be continuous, inventory-led, and tied to real execution paths.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• Forward look: The control gap is moving from visibility into delegated action, which is why OWASP Agentic AI Top 10 is becoming a practical baseline for security teams.

What this signals

AI programmes now need an identity inventory for agents, connectors, and service identities, not just model registries. The next control gap is not whether the model can answer safely. It is whether the system can prove which identity acted, what data it touched, and which delegated scope allowed it to do so.

Prompt-layer trust gap: enterprise AI will keep failing where untrusted content is allowed to influence privileged actions. Teams should expect more incidents that blend data ingestion flaws with identity abuse, which makes 52 NHI Breaches Analysis relevant as a pattern library for the credential side of the problem.

If your AI governance plan does not include runtime monitoring, tool segmentation, and rapid credential revocation, it is incomplete. The operational question is no longer whether AI can be deployed. It is whether the surrounding identity model can constrain what the system is allowed to become.

For practitioners

• Map AI execution paths to identity boundaries Document which service accounts, tokens, API keys, and linked users each copilot, agent, RAG pipeline, or MCP server can reach. Reconcile that map with least-privilege access and remove inherited permissions that were added for convenience rather than necessity.
• Isolate prompt, retrieval, and tool trust zones Separate untrusted content from policy inputs, and do not let retrieved text or external pages directly influence privileged actions. Add sanitization, allowlisting, and human review gates where an agent can send mail, change records, or invoke downstream systems.
• Treat AI assistants as monitored workloads Log tool calls, credential use, and agent-to-agent delegation with the same discipline used for other high-risk workloads. If you cannot trace which identity triggered which action, you cannot investigate compromise or prove control effectiveness.
• Run continuous red teaming on agent behaviour Test for prompt injection, hidden instruction channels, auth bypass, command injection, and data exfiltration across the full stack, including connectors and fallback paths. Re-test after every major model, framework, or integration change.

Key takeaways

• The article shows that AI incidents are increasingly identity incidents because agents, copilots, and connectors act with delegated trust.
• The scale signal is clear: reported AI incidents are rising, exposed credentials are circulating, and many deployments already exceed intended scope.
• The control answer is not model-only safety but prompt governance, least privilege, continuous testing, and traceable AI identity boundaries.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.
• Delegated AI identity: Delegated AI identity describes an AI system that acts with permissions, credentials, or scoped authority on behalf of a business process. It matters because once an AI can initiate actions, it needs lifecycle, access, and offboarding controls similar to other governed non-human identities.

What's in the full article

Akto's full blog covers the operational detail this post intentionally leaves for the source:

• Incident-by-incident breakdowns of the 10 AI attacks and the exact exploit chain used in each case
• The article's control recommendations for prompt-layer hardening, agent behaviour monitoring, and runtime defences
• Specific examples of how copilots, agents, MCP servers, and RAG pipelines were breached in production-like environments
• Akto's closing guidance on what CISOs should prioritise over the next six months

👉 Akto's full post covers the individual attack paths, control failures, and response priorities behind each incident

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It gives security practitioners a structured way to align identity controls with the systems now acting on their behalf.