Google Gemini incident exposes the AI identity governance gap

At a glance

What this is: Google Gemini’s calendar vulnerability showed how hidden instructions in normal invite metadata can turn legitimate AI permissions into unintended data exposure.

Why it matters: It matters because IAM, IGA, and PAM teams now have to govern contextual intent and cross-service behavior, not just access rights and credential scope.

👉 Read Unosecur's analysis of the Google Gemini incident and AI identity risk

Context

Indirect prompt injection is a manipulation technique where hidden instructions are embedded in content that an AI system later reads as part of normal processing. In the Google Gemini incident, that meant a calendar invite could carry instructions that changed how the assistant handled schedule data, even though no credentials were stolen.

The governance gap is that traditional identity controls validate entitlement, but not intent, context, or downstream action chain. For NHI and AI governance programmes, that means a system can remain technically authorised while still violating the security boundary the business assumed it would preserve.

This is the kind of failure that becomes visible only when AI services are treated as governed identities with observable behavior, not just as application features. The article reflects a broader and increasingly typical pattern in enterprise AI adoption, not a one-off edge case.

Key questions

Q: What breaks when an AI assistant can access private data and untrusted content at the same time?

A: When an assistant can access private data and ingest untrusted content, a small injected instruction can become a data-exfiltration path. The usual assumption that content is passive fails, because the model can interpret it as an operational command and move it into tool execution.

Q: Why do traditional IAM controls miss indirect prompt injection attacks?

A: IAM was built to answer who can access what, not whether an AI should act on a specific piece of content. Indirect prompt injection exploits that gap by keeping every permission legitimate while changing the meaning of what the system reads. The result is authorized access with unsafe intent, which legacy controls are not designed to detect.

Q: How should security teams govern AI-assisted data movement across endpoints?

A: Security teams should govern AI-assisted data movement by starting at the endpoint, where content is opened, copied, transformed, and redistributed. They need lineage-aware policy that tracks how information moves across applications and identities, including non-human actors. Without that sequence, teams can neither distinguish normal use from risky propagation nor enforce controls before exposure spreads.

Q: Who is accountable when an AI assistant overshares sensitive content?

A: Accountability sits with the team that owns the policy, the attribute feeds, and the enforcement points, because ABAC only works when all three are managed together. If any one of them is missing, the organisation has not built a defensible control path, even if the model itself appears constrained.

Technical breakdown

Why contextual control matters more than simple authorisation

Authorisation answers whether an identity can reach a resource, but not whether the action is safe in the current context. In AI systems, untrusted text can become part of the decision process, which means the effective input is not just user intent but any embedded instruction the model can parse. That creates a semantic attack surface. The system may faithfully follow policy while still producing an unsafe outcome because policy did not constrain meaning, only access.

Practical implication: security teams need context-aware governance for AI actions, not only entitlement checks.

How indirect prompt injection becomes a cross-service identity problem

The risk appears when an AI service can read from one source and write to another under legitimate permissions. A hidden instruction in source data can steer the model to summarize, copy, or expose sensitive content into a downstream system, creating a new trust path without a traditional exploit. The identity issue is that the service is operating as a non-human actor whose output becomes an access path in itself.

Practical implication: map every read-to-write path where AI output can propagate data into another system.

Why behavioral monitoring is part of AI identity governance

Static permission models miss the fact that AI behavior can change based on the content it processes. That is why identity telemetry, behavioral baselines, and anomaly detection matter for AI services and other NHIs. The control objective is not to infer what the model thinks, but to detect when its actions diverge from normal context, scope, or data handling patterns.

Practical implication: treat unusual cross-service writes and data propagation as identity events, not only application events.

Threat narrative

Attacker objective: The attacker’s objective was to exfiltrate private calendar and meeting data without stealing credentials or triggering a conventional alert.

1. Entry occurred through a legitimate-looking calendar invite that carried hidden instructions inside the event description.
2. Escalation happened when Gemini processed the malicious text as contextual input and used its existing permissions to read and write calendar data.
3. Impact followed when private meeting information was written into a new event that exposed sensitive data beyond the intended trust boundary.

Breaches seen in the wild

• MongoBleed breach — MongoBleed exposed secrets across 87K MongoDB servers.
• IOS app secrets leakage report — iOS apps leaking hardcoded secrets and credentials endangering user privacy.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Contextual intent governance is the missing control layer for AI identities. Traditional IAM proves that an AI service is allowed to read and write, but it does not prove that the action is safe in context. That distinction matters because indirect prompt injection weaponises legitimate permissions rather than stealing them. Practitioners should treat intent as a governance dimension, not an application nuance.

AI services behave like non-human identities only when their outputs create downstream trust decisions. The Gemini case shows that a model can remain inside its assigned permissions and still generate an insecure data path. That makes it an NHI governance problem, not a model-only problem. Identity teams need to assess where AI outputs are consumed as if they were trusted operator actions.

Semantic attack surface should be a named control concept in enterprise AI programmes. The hidden instruction was not code, but it was still executable influence over system behavior. This shifts the governance boundary from authenticated action to interpreted meaning, which is exactly where conventional IAM stops. Practitioners should use this concept to map where language can trigger privileged workflows.

Static access review cycles were designed for stable entitlements, not content-triggered action chains. That assumption fails when an AI service can transform untrusted input into a privileged downstream action within a single interaction. The implication is not just better review frequency. It is a rethink of what actually constitutes an access event in AI-enabled environments.

Visibility, monitoring, and accountability become the only workable control set when language is an execution surface. If an identity can be manipulated through content, the programme has to detect behavior, not merely entitlement. That is why AI governance now belongs alongside NHI oversight, IGA, and PAM rather than beside them. Security teams should build around observed action paths, not declared permissions.

From our research:

• 80% of identity breaches involved compromised non-human identities such as service accounts and API keys, according to the Ultimate Guide to NHIs.
• Only 5.7% of organisations have full visibility into their service accounts, which is why AI services hidden inside application workflows are so difficult to govern.
• 52 NHI Breaches Analysis shows how unmanaged machine identities repeatedly turn legitimate access into incident pathways.

What this signals

Semantic trust drift: AI programmes now need a control model for content that changes action, not just code that changes state. When a model can turn untrusted language into downstream writes, the security boundary shifts from authentication to interpretation, which current IAM programmes rarely map cleanly.

For practitioners, this means AI identity governance has to sit alongside the wider NHI programme, including access reviews, visibility, and offboarding. The organisations that already treat service accounts as governed identities will have a shorter path to governing AI services that behave the same way but with more dynamic input paths.

For practitioners

• Map AI read-to-write trust chains Identify every AI service that can read one system and write into another, then document where its outputs can expose sensitive data or create a new record. Prioritise calendar, email, ticketing, and collaboration workflows where hidden instructions can travel with content.
• Add context checks to privileged AI actions Require a second layer of policy for actions that summarize, copy, publish, or forward sensitive material. The control should evaluate source trust, content origin, and downstream exposure before the AI output is committed.
• Treat AI service accounts as governed identities Inventory the identities behind AI integrations, classify them by data access, and include them in access reviews, least-privilege checks, and offboarding workflows. The goal is to see AI services as accountable actors rather than invisible application features.
• Instrument anomaly detection for cross-service writes Baseline normal output patterns for AI services and alert when a model writes unexpected events, messages, or records after processing untrusted input. Focus on cross-service writes, not just login anomalies, because the abuse occurs after authentication.

Key takeaways

• The Gemini incident shows that legitimate AI permissions can still produce unsafe data exposure when hidden instructions shape the model’s output.
• The evidence points to a governance gap, not a model failure: IAM validated access, but it did not validate intent, context, or downstream action.
• Practitioners should govern AI services as non-human identities with read-to-write path visibility, behavioural monitoring, and context-aware control points.

Key terms

• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Contextual control: The governance layer that decides whether an AI action makes sense in the current situation, not just whether it is technically allowed. It matters when legitimate permissions still produce unsafe outcomes because the model acted on untrusted context.
• Semantic Attack Surface: Semantic attack surface is the set of readable inputs that can influence how an AI system behaves, even when those inputs are not executable code. In agentic systems, documentation, repository structure, comments, and retrieved context can all become security-relevant because they affect decisions.
• Read-to-write trust chain: A path where an identity reads content from one system and writes the result into another system. In AI environments, that chain can create data exposure if untrusted input influences a privileged downstream write.

What's in the full article

Unosecur's full blog covers the operational identity controls this post intentionally leaves at the governance level:

• How Unosecur structures AI service visibility across identity, access, and behavior signals.
• The checklist for contextual oversight and auditability in environments where AI services can write to downstream systems.
• The specific monitoring and remediation workflow the vendor recommends for AI-driven identity misuse.
• The role of access reviews and compliance reporting when AI services create unexpected data exposure.

👉 Unosecur's full post covers the AI identity control gaps, behavior signals, and remediation checklist in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.