Google Gemini XSS exposes the limits of AI code execution controls

At a glance

What this is: Appknox’s analysis shows that Google Gemini’s code canvas can turn generated output into instantly executed code when validation and sandboxing are too weak.

Why it matters: For IAM and security teams, the issue matters because AI tools are now acting like execution environments, which means trust, authorisation, and runtime controls have to extend beyond classic application review.

By the numbers:

• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read Appknox's analysis of the Gemini XSS vulnerability and AI code execution risk

Context

AI code generation changes the security problem from review-only risk to execution risk. When a system renders untrusted output in real time, the boundary between code creation and code use disappears, and that creates a new control gap for application security, IAM, and AI governance teams.

In this case, the primary issue is not just cross-site scripting in the traditional sense. It is the treatment of AI-generated output as trusted enough to execute automatically, which means organisations need runtime authorisation, tighter sandboxing, and stronger control over what the tool is allowed to do in a live preview environment.

Key questions

Q: How should security teams govern AI coding assistants that can execute commands?

A: Treat them as delegated non-human identities with bounded execution authority. Require human approval for destructive commands, keep command scopes narrow, and log every tool action. The key control question is not whether the assistant is helpful, but whether it can be prevented from acting outside intended scope when prompts, context, or rules are manipulated.

Q: Why do AI-generated code changes increase application security risk?

A: AI-generated code can increase risk because it accelerates output faster than review, testing, and secret hygiene can keep up. The issue is not only flawed logic. It is also the possibility that tokens, credentials, unsafe dependencies, or insecure defaults are reproduced at scale across the delivery pipeline.

Q: What do teams get wrong about sandboxing autonomous AI agents?

A: Teams often confuse containment with trust. A sandbox can limit blast radius, but it does not automatically prevent the agent from using allowed tools against its own environment, especially when package installs, runtime scripts, and configuration files are all within reach. The wrong assumption is that policy compliance equals benign intent.

Q: What should organisations do when GenAI is embedded in code and workflows?

A: Apply secure SDLC and third-party dependency controls to AI outputs, including review for bugs, secrets, prompt-injection artifacts, and unsafe API use. AI-assisted code is not exempt from normal engineering governance. The practical standard is the same as any external input: inspect, test, approve, and monitor before it reaches production.

Technical breakdown

Why auto-rendered AI output becomes an execution path

Modern AI code assistants often combine generation, preview, and execution in one workflow. That design is efficient, but it removes the pause where validation normally occurs. If HTML, JavaScript, or embedded content is rendered immediately, the system is no longer just suggesting code. It is handling untrusted input as if it were safe output. The security model breaks when the browser or preview engine becomes the enforcement layer instead of a controlled review step.

Practical implication: teams should separate generation from execution and require explicit approval before any AI-produced content runs.

How sanitisation and sandboxing fail in AI preview environments

The article describes a familiar pattern in modern web exploits: parsing differences, encoded payloads, and over-permissive sandbox settings. SVG parsing, legacy attributes such as xlink:href, and entity obfuscation can all evade simplistic filters if controls do not recurse through nested structures. Sandbox boundaries also matter because an iframe is not safe by default. If pop-ups, redirects, or script-linked actions remain available, the preview environment can still be abused.

Practical implication: harden preview sandboxes and validate all embedded content types, not just standard HTML.

Why this is an appsec and identity governance problem

AI-generated code risks are often discussed as a coding quality issue, but the real concern is control of runtime behaviour. Once an AI system can execute code, it becomes part of the trusted computing chain and needs governance like any other privileged system. That includes access boundaries, execution policy, logging, and accountability for what the system is allowed to render or trigger. In practice, this is where application security starts to overlap with identity governance and privileged access control.

Practical implication: treat AI execution features as privileged capabilities and govern them with explicit policy and auditability.

Threat narrative

Attacker objective: The attacker wants to turn trusted AI preview functionality into a code-execution path that supports phishing, redirection, or further compromise.

1. Entry occurs when a crafted payload is introduced through AI-generated or AI-edited code in the preview workflow.
2. Escalation happens when sanitisation gaps and permissive parsing allow the payload to survive into the rendering layer.
3. Impact follows when the preview environment executes malicious code automatically, enabling redirects, phishing, or session abuse.

NHI Mgmt Group analysis

Zero-click execution is the new control failure in AI-assisted development: the risk is no longer only whether code is malicious, but whether the system is allowed to execute it before scrutiny. Traditional AppSec assumes human review sits between creation and runtime. AI preview workflows collapse that boundary, so governance must focus on execution permission, not just code quality. Practitioners should treat auto-render features as a privileged pathway and control them accordingly.

Execution control now matters more than static code scanning: tools that inspect code before release cannot fully address a workflow where the system renders and runs output instantly. The article illustrates a broader gap in AI governance: the trust boundary has moved from source code to runtime behaviour. That makes sandbox design, allowlisting, and event logging critical parts of the control set. Practitioners should extend security review to the runtime layer, not stop at pre-commit analysis.

AI preview surfaces need identity-style governance: when a platform can trigger browser actions, redirects, or script execution, it should be governed like a powerful service account or automation identity with tightly scoped permissions. The conceptual mistake is to see the feature as a convenience layer rather than an actor with authority. That mindset creates the same kind of governance drift seen in NHI programmes. Practitioners should define explicit permissions for AI execution paths and review them as part of privilege management.

AI governance debt is building inside developer workflows: every feature that compresses generation and execution without independent controls adds hidden risk that later teams inherit. This is not a Gemini-only problem. It is a pattern that will reappear anywhere code can be generated, previewed, and executed inside the same trust boundary. Practitioners should assume the category will expand and design policy now, before the operational blast radius grows.

Runtime trust must be measured, not assumed: the presence of a secure-looking preview panel does not prove the workflow is controlled. The relevant question is whether untrusted output can cross into execution without a deliberate enforcement step. That is the governance threshold organisations need to test. Practitioners should make runtime approval, logging, and sandbox constraints auditable controls rather than informal defaults.

What this signals

AI-assisted development will increasingly be judged by whether it can prove runtime control, not just code quality. That means security programmes need a way to assess execution boundaries, logging, and approval flow in the same way they already assess identity and privilege controls. For teams building AI-heavy pipelines, the operational question is whether untrusted output can still cross into production behaviour without a hard gate.

Execution trust gap: the meaningful risk is not only malformed code, but the organisational assumption that generated output can be allowed to run because it came from a trusted tool. That assumption will fail in more places as AI features expand across development platforms. Practitioners should expect preview surfaces, code canvases, and low-code execution paths to become recurring audit points.

Security leaders should prepare for AI tooling to be reviewed like any other privileged automation layer. That means access boundaries, policy enforcement, and incident logging need to be defined before the next workflow surprise forces the issue. The programmes that mature fastest will be the ones that treat AI execution paths as governed assets rather than experimental features.

For practitioners

• Separate generation from execution Require a distinct review step before any AI-generated HTML, JavaScript, or embedded content is rendered in a live preview or browser context.
• Harden preview sandbox settings Disable unnecessary iframe capabilities such as pop-ups, redirects, and cross-window actions, and test the sandbox against encoded and nested payloads.
• Treat AI execution features as privileged Assign explicit owners, access rules, and audit logging to tools that can execute code automatically, just as you would for a sensitive automation identity.
• Add runtime detection for preview abuse Monitor for suspicious redirects, window.open abuse, SVG-based payloads, and unexpected script execution inside AI-assisted workflows.
• Map controls to security frameworks Use NIST Cybersecurity Framework 2.0 and NIST SP 800-53 Rev 5 Security and Privacy Controls to anchor approval, logging, and sandbox hardening requirements.

Key takeaways

• AI-generated output can become a live execution path, which means the security problem is now runtime trust, not just code review.
• The strongest evidence from this case is the collapse of the pause between generation and execution, which makes weak sanitisation and sandboxing materially more dangerous.
• Practitioners should govern AI preview features as privileged capabilities, with explicit approval, logging, and sandbox constraints.

Key terms

• AI Code Execution Vulnerability: A flaw where AI-generated output can run as executable code without sufficient validation or approval. The risk is not limited to malicious prompts. It arises when preview, rendering, and runtime controls fail to keep untrusted content from behaving like trusted application logic.
• Zero-Click Execution: A condition where malicious code runs without the victim taking a deliberate action such as clicking or confirming. In AI environments, this often happens when generated output is auto-rendered in a live preview and the security boundary is assumed to be a review step when it is really an execution step.
• Execution Control: The set of technical and governance measures that decide when code is allowed to run, in what environment, and with what permissions. It is stronger than static validation because it governs runtime behaviour, sandbox constraints, logging, and explicit approval before an action reaches production-like execution.

What's in the full article

Appknox's full blog post covers the operational detail this post intentionally leaves for the source:

• The full exploit chain showing how SVG parsing, entity obfuscation, and xlink:href were combined to bypass sanitisation.
• A clearer breakdown of the preview sandbox weaknesses that allowed malicious actions to escape intended boundaries.
• The article's practical prevention guidance for teams evaluating AI code execution and preview isolation controls.
• The source also expands on why traditional AppSec tooling struggles when generation and execution happen in the same workflow.

👉 Appknox's full post covers the exploit path, sandbox failure points, and runtime control implications.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and workload identity. It helps security practitioners connect privilege control and runtime trust to real operational programmes.