AI-generated code expands appsec risk faster than human review

At a glance

What this is: This is an analysis of how AI-generated code, coding assistants, and LLM agents are expanding application security risk across prompt injection, supply chain exposure, and secret leakage.

Why it matters: It matters to IAM and security practitioners because AI tooling is creating new trust boundaries around credentials, approvals, and third-party access that existing appsec and identity controls do not yet govern well.

By the numbers:

• Through 2029, over 50% of successful cybersecurity attacks against AI agents will exploit access control issues, using direct or indirect prompt injection as an attack vector.
• By 2027, at least 30% of application security exposures will result from usage of vibe coding practices.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read LEGIT Security's analysis of AI-generated code risk in application security

Context

AI-generated code is changing application security because it increases the amount of code that ships faster than review and governance processes can keep up. The core issue is not simply that the code is machine-written, but that AI-assisted development broadens the attack surface through prompt injection, unvetted libraries, hidden dependencies, and secret exposure in ways traditional appsec controls were not built to absorb.

That creates a direct governance problem for IAM, PAM, and secrets management teams because AI code workflows often depend on credentials, tokens, and third-party integrations that are not always scoped or monitored as rigorously as human-built systems. In practice, the line between application security and identity security gets thinner when AI tools can generate, recommend, or execute code with access to sensitive environments.

The article's starting position is typical of the current market: many organisations are adopting AI development tooling before they have a clear control model for it.

Key questions

Q: How should security teams govern AI code assistants that have repository and cloud access?

A: Security teams should govern AI code assistants as privileged non-human identities with explicit ownership, least privilege, and continuous logging. The important control is not whether the assistant is allowed to exist, but whether its access is scoped to a defined task and can be revoked quickly when its role changes.

Q: Why do AI coding agents increase supply-chain risk?

A: AI coding agents can choose tooling, install packages, and edit lock files in ways that may bypass the controls humans expect in a managed environment. That widens the blind spot around untracked software and makes package provenance harder to enforce. The practical response is to restrict agent permissions, inventory their workspaces, and review their install behaviour like any other privileged automation.

Q: What breaks when indirect prompt injection is not controlled in AI systems?

A: Indirect prompt injection breaks the assumption that retrieved content is safe to use as instruction material. Once malicious text enters the model context, the system may alter responses, leak data, or trigger tools with delegated permissions. The core failure is boundary collapse between data and directive, which turns ordinary content ingestion into an execution risk.

Q: How should organisations decide when to allow autonomous coding agents?

A: Allow autonomy only when the task is bounded, reproducible, and reversible. If the work can change multiple files, call external tools, or run in cloud environments, it needs stronger scoping, monitoring, and rollback readiness. Higher autonomy should always require higher assurance, not just faster delivery.

Technical breakdown

Why AI-generated code creates new appsec attack surface

AI code generation changes the security problem from reviewing isolated changes to governing a high-volume, probabilistic output stream. Large language models generate code by pattern matching across training data, so the output can include insecure defaults, outdated libraries, or code that looks plausible but behaves unsafely in context. The risk increases when developers copy suggestions directly into production workflows without independent validation. This is different from traditional code review because the failure may be systemic, not a single bug. Practical implication: treat AI-generated code as untrusted input until it passes the same verification gates as externally sourced code.

Practical implication: require review, testing, and dependency validation for AI-generated code before it reaches shared environments.

How prompt injection and data poisoning alter AI tool trust

Prompt injection manipulates an AI system through crafted input so it follows attacker intent instead of user intent. Data poisoning changes the training or retrieval substrate so the model learns or retrieves malicious content as if it were legitimate. In coding workflows, either technique can cause an assistant to recommend unsafe code, leak sensitive content, or execute actions the operator did not intend. The important point is that the model's trust boundary is not the same as the application's trust boundary. Practical implication: isolate AI assistants from production secrets and enforce contextual controls on prompts, retrieval sources, and tool permissions.

Practical implication: separate model access from production credentials and restrict what the AI system can read or call.

Why autonomous coding agents raise identity governance questions

LLM agents differ from simple copilots because they can chain actions across tools, files, and services with limited human intervention. That turns the agent into an operational identity with its own permissions, lifecycle, and audit requirements. If an agent can download packages, run commands, or modify code, then the governance question is no longer only about code quality but about delegated authority. This is where NHI and agentic AI security intersect directly: agent identities need scoped access, revocation, logging, and approval boundaries. Practical implication: manage coding agents as privileged non-human identities, not as passive software features.

Practical implication: assign agent-specific permissions, logs, and revocation paths before allowing tool use in development pipelines.

Threat narrative

Attacker objective: The attacker aims to convert AI-assisted development into a path for code theft, secret exposure, or malicious code execution inside trusted software pipelines.

1. Entry begins when an attacker slips malicious instructions into a prompt, website, package, or dataset that an AI coding tool consumes.
2. Escalation occurs when the assistant follows those instructions, fetches untrusted dependencies, reveals secrets, or runs unsafe commands with delegated access.
3. Impact follows when insecure code, leaked credentials, or malicious updates reach repositories and downstream production systems.

NHI Mgmt Group analysis

AI-generated code is now an identity governance problem, not just an application security problem. When AI tools can create, suggest, or execute code, they rely on credentials, API tokens, repository permissions, and execution rights that must be scoped and revoked. That means the security failure is often over-delegated access rather than code quality alone. Practitioners should manage AI development systems as governed identities with explicit authority boundaries.

Prompt injection exposes a control gap that conventional review workflows do not cover. Human code review assumes the risky artifact is visible and static, but prompt-driven systems can change behavior at runtime based on external content. That makes source provenance, retrieval trust, and tool permissions central controls. The practical conclusion is that appsec teams need runtime trust controls, not just pre-merge inspection.

Shadow AI creates the same governance debt that shadow IT created, but with faster execution paths. Unapproved AI coding tools can import external dependencies, handle sensitive prompts, and interact with repositories outside central oversight. The result is fragmented policy enforcement across development teams and service accounts. Security leaders should treat ungoverned AI tooling as a lifecycle and access management issue before it becomes a breach issue.

AI assistant oversight cannot rely on the old assumption that more human review alone will absorb the risk. The article's core warning is that code volume is rising faster than review capacity, which means governance has to shift upstream into tooling, permissions, and automated controls. That is where application security and identity governance now overlap most clearly. Teams should redesign controls around delegated machine action, not just developer intent.

What this signals

AI-assisted development will keep expanding the number of places where credentials, tokens, and build permissions live, which means identity teams need to move from periodic review to continuous control of machine access paths. The organisation that can see where AI tools read, write, and execute will be the one that can govern them.

Delegated code authority gap: AI systems are being granted the practical equivalent of developer authority without matching lifecycle controls, so the next programme priority is to map every model, assistant, and agent to a revocable identity and a defined approval boundary.

For programmes that already struggle with secrets sprawl, the stronger signal is that AI adoption will widen the gap between where access exists and where governance can observe it. Linking development tooling to identity policy and to the NIST Cybersecurity Framework 2.0 gives teams a cleaner way to think about protection and recovery.

For practitioners

• Define AI coding tool trust boundaries Classify every code assistant, agent, and model integration by the data it can see, the commands it can run, and the repositories it can touch. Deny access to production secrets by default and require explicit approval for any tool that can execute code or install packages.
• Review delegated permissions for coding agents Inventory the service accounts, API keys, and OAuth grants used by AI development tools. Apply least privilege, short-lived credentials, and revocation paths so an assistant cannot continue to act after the task ends.
• Validate AI-generated dependencies before merge Require dependency provenance checks, software composition analysis, and human approval for any library or code block produced by an AI tool. Pay special attention to generated snippets that introduce outdated or unmaintained packages.
• Harden prompts and retrieval sources Restrict which websites, documents, and datasets an AI assistant can ingest, especially when the assistant has access to internal code or secrets. Use allowlists, retrieval filtering, and logging to reduce prompt injection exposure.

Key takeaways

• AI-generated code expands appsec risk because it increases code volume, dependency exposure, and secret leakage faster than humans can review it.
• The practical failure mode is delegated machine access without enough governance, especially when assistants can read prompts, call tools, or run commands.
• Security teams need to treat coding agents and assistants as governed non-human identities with scoped permissions, logging, and revocation paths.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• Autonomous Coding Agent: A software agent that can decide, sequence, and execute development tasks with minimal human intervention. In practice, it reads code, invokes tools, and changes files in runtime, so governance must focus on its permissions, inputs, and action boundaries rather than only on the resulting code.
• Development Trust Boundary: The set of systems, data, and permissions a development tool can touch during normal use. When an AI assistant is integrated into editors, repos, or external services, that boundary widens and must be managed explicitly. Security teams should define the boundary before broadening access.

What's in the full article

LEGIT Security's full analysis covers the operational detail this post intentionally leaves for the source:

• Specific examples of AI-generated code failure modes that teams can test for in review and pipeline controls
• Details on the GitLab Duo prompt injection finding and the kinds of data it exposed
• Examples of how autonomous AI agents can be constrained when they are allowed to use external tools
• Practical guidance on securing AI-generated code without blocking development workflows

👉 The full LEGIT Security article covers AI coding assistant risks, autonomous agent behaviour, and examples of insecure generated code.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle controls. It helps security practitioners connect application workflows to the identity controls that keep delegated access bounded.