Secure AI code generation still needs runtime scanning and guardrails

At a glance

What this is: This article reports that AI-generated code still produced vulnerabilities even in a security-focused tournament, with runtime scanning revealing repeated flaws such as path traversal and insecure command handling.

Why it matters: It matters because teams using coding agents need control points at generation time, not just review time, or insecure output can move straight into development workflows.

By the numbers:

• 31.4% of all submissions contained at least one vulnerability.
• 53 players competed across 6 rounds and 102 evaluated submissions.
• Path traversal accounted for 56.5% of all vulnerabilities found.

Context

AI code generation changes the timing of control failure. The risk is no longer just whether a developer notices insecure code during review, but whether the system that generates the code can be constrained before vulnerable patterns appear at all.

That matters for secure software delivery because coding agents can produce plausible code that still embeds classic flaws such as path traversal, CSRF, command injection, XSS, and authorization bypass. The identity and access angle is indirect but real: once generated code reaches production, its permissions, secrets handling, and API access patterns become part of the attack surface.

Key questions

Q: What should teams do first when AI-generated code affects security tooling?

A: Start by treating the code as an untrusted change until a human reviewer confirms the control objective, the test coverage covers failure cases and the release gate requires explicit approval. Security tooling should never inherit the lower assurance often tolerated in prototype work, because defects there can distort detection or enforcement.

Q: Why do AI coding tools still create security risk even when developers use security-aware prompts?

A: AI coding tools can still produce insecure output because prompt discipline does not guarantee secure implementation. The model may generate vulnerable patterns, omit controls, or reproduce unsafe defaults. Organisations should treat generated code as untrusted until it has passed policy checks, vulnerability verification, and functional revalidation inside the workflow.

Q: How do security teams know whether generated code is actually safe to ship?

A: They should look for policy compliance before commit, not just scan results after commit. A safe-enough threshold requires secure generation rules, dependency provenance checks, and runtime reachability validation so teams can see whether a flaw is theoretical or reachable in the live stack.

Q: What happens when a coding agent is allowed to generate code with broad access to repositories and secrets?

A: The agent becomes a non-human identity with a direct path into code and infrastructure decisions. If its permissions are broad, any vulnerability it generates can carry privileged access paths, making the blast radius much larger than a normal code defect would create.

Technical breakdown

Why runtime scanning matters during AI code generation

AI-generated code is not inherently safer because the user asked for security. The core problem is that code generation happens before human review, so insecure patterns can be emitted, copied, and accepted as working output. Runtime scanning inserts an automated control at the point where the code is created, letting static analysis or policy checks reject dangerous constructs before they are committed. That is materially different from scanning only in CI, where the code may already have influenced local testing, downstream prompts, or developer judgment.

Practical implication: treat generation-time scanning as part of the control plane for coding agents, not as a downstream quality gate.

Why secure prompts do not eliminate classic web vulnerabilities

The tournament data shows the model repeatedly produced familiar weaknesses such as path traversal, CSRF, command injection, XSS, and authorization bypass. That pattern matters because it means the failure mode is not exotic model weirdness, but repeated reproduction of standard application security defects under prompt pressure. In practice, the model optimises for plausible code completion, not for threat-aware design. Prompting can reduce obvious mistakes, but it does not reliably enforce secure defaults across changing tasks, languages, or frameworks.

Practical implication: pair prompting with application security verification and secure coding rules, especially for file handling, command execution, and authorization logic.

How relative scoring can hide insecure output

A head-to-head challenge can reward the better answer, not the safe answer. That creates a dangerous illusion: code can win a contest, satisfy functionality, and still contain vulnerabilities that would be unacceptable in production. The article’s data also shows that some zero-vulnerability submissions lost on task fit or completeness, which means security alone is not enough and functionality alone is not enough. For practitioners, that is the same trade-off seen in real engineering: unsafe code can be 'good enough' to ship unless a guardrail blocks it earlier.

Practical implication: enforce security and functionality as separate acceptance criteria so that one cannot compensate for failure in the other.

Threat narrative

Attacker objective: The practical attacker objective is to get insecure code accepted into the software supply chain so the weakness reaches production and becomes exploitable.

1. Entry occurs when a coding agent produces insecure code in response to a prompt that appears valid but encodes risky behaviour, such as shell execution or unsafe path handling.
2. Credential or access abuse is not the main mechanism here; instead, the model repeatedly generates exploitable logic, including path traversal, command injection, CSRF, XSS, and authorization bypass.
3. Impact follows when vulnerable code is accepted into the development workflow, where it can reach review, deployment, and eventually production systems with the flaw intact.

Breaches seen in the wild

• PocketOS database deletion incident: An AI coding agent found an over-privileged Railway API token in the codebase and deleted PocketOS production data and backups in nine seconds.
• Firebase misconfiguration exposure 2024: Missing Firebase security rules on 916 websites exposed 125 million user records and 19.87 million plaintext passwords; a quarter were fixed.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI code generation creates a new control point problem, not just a code quality problem. The article shows that vulnerabilities can appear even when users are trying to be secure and the generated code is being scanned automatically. That means the governance question shifts from 'can reviewers catch it later' to 'what is blocking unsafe output at generation time.' For teams using coding agents, the security model has to move upstream into the creation loop.

Runtime scanning is now part of secure software governance for AI-assisted development. Static review after the fact remains useful, but it is too late to be the only line of defence when insecure patterns are repeatedly generated. This is especially relevant where AI-generated code touches secrets, authentication, authorization, or API calls, because those paths determine blast radius. The practitioner conclusion is straightforward: secure generation needs continuous verification, not occasional inspection.

Secure prompting is a weak substitute for secure constraints. The tournament demonstrates that even knowledgeable participants, given explicit security incentives, still produced vulnerable output at scale. That points to a named failure mode: prompt-dependent security assurance. If the security posture depends on the user remembering the right phrasing, the control is brittle by design. The practitioner response is to treat prompts as guidance and scanners as enforcement.

AI coding agent governance now intersects with identity and privilege decisions. Once generated code can call internal services, read secrets, or execute privileged actions, the issue is no longer just code safety but non-human identity control. The relevant governance boundary is who or what is allowed to generate, submit, and execute code paths that carry access. For IAM and PAM teams, that means coding agents need the same scrutiny applied to other machine actors.

Path traversal remains a durable pattern because it sits at the boundary between input handling and file-system trust. The repeated appearance of the same flaw across different prompts suggests that secure-by-default application logic is still not emerging automatically from current coding agents. That reinforces the need for secure coding standards, policy checks, and runtime scanning in the same workflow. The practitioner takeaway is to enforce controls where the model is weakest, not where the review cycle is strongest.

From our research library:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap, according to the State of Secrets in AppSec.
• Read next: AI Coding Agents Security Guide

What this signals

Generation-time control is now the decisive boundary for AI-assisted development. Teams should assume that prompts alone will not suppress recurring vulnerability classes, especially where the model keeps reproducing path traversal and command injection patterns. The practical shift is toward enforcement at output time, backed by secure coding policy and automated scanning.

Non-human identity governance matters once a coding agent can touch code, secrets, or deployment paths. At that point the agent is not just a productivity tool, it is a machine actor with access decisions attached to it. That means inventory, ownership, and least-privilege review need to extend to the development workflow itself.

For practitioners

• Implement generation-time static analysis for coding agents Scan AI-generated code before it is committed so insecure patterns are blocked while the agent is still producing output, not after a pull request is opened.
• Add security policy checks to prompt and response workflows Define forbidden patterns for shell execution, path traversal, unsafe file access, and unauthorised network calls, then reject outputs that violate those rules.
• Separate security scoring from functional acceptance Require code to meet both secure coding thresholds and problem-completion thresholds so a safe but incomplete answer cannot silently replace a complete but vulnerable one.
• Review coding agent access to secrets and repositories Treat the agent as a non-human identity with bounded access, explicit ownership, and short-lived permissions whenever it can touch source code or deployment resources.
• Track recurring flaw classes in AI-generated code Measure how often the same vulnerability types reappear across prompts so you can tune controls around file handling, authorization, and input validation rather than chasing isolated defects.

Key takeaways

• AI code generation still emits exploitable patterns even when users are trying to write secure code, which makes downstream review alone an insufficient safeguard.
• The repeated flaws in this tournament were familiar application security issues, not exotic model failures, which points to a control gap in generation-time enforcement.
• Teams that want safe AI-assisted development need scanning, policy controls, and access governance in the workflow that produces the code.

Key terms

• Generation-time Scanning: Security scanning that runs while code is being produced, rather than after commit or in a downstream pipeline. This approach shortens feedback loops and reduces the chance that vulnerable AI-generated code will move into review, integration, or release stages before defects are caught.
• Secure Coding Tournament: A secure coding tournament is a controlled evaluation where participants try to produce secure, functional code under scoring rules that penalise vulnerabilities. It reveals how well people, prompts, and tooling perform under pressure, but it does not replace real-world engineering controls.
• Coding Agent: A coding agent is a software system that can plan, generate, and modify code with limited human prompting. In governance terms, it is not just a tool but an actor whose permissions, tool access, and rollback path must be managed as part of the delivery process.
• Prompt-Dependent Security Assurance: Prompt-dependent security assurance is the brittle practice of relying on careful instructions to prevent insecure output from AI systems. It fails when the model reproduces risky patterns despite good intentions, so security has to move from wording to enforcement.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and identity practitioners build the control discipline that AI-assisted development now depends on.