LLM application security is shifting to code, supply chain, and runtime

At a glance

What this is: Aikido argues that LLM application security now has to cover code, dependencies, developer tooling and runtime usage because model-only guardrails miss the main attack surface.

Why it matters: For IAM, PAM and NHI teams, the key lesson is that AI systems now create their own access paths, secrets exposure points and third-party trust dependencies that need lifecycle governance.

By the numbers:

• 24% of production code is now AI-generated, according to Aikido.
• 69% of organizations have discovered vulnerabilities in AI-generated code, according to Aikido.
• 20% reported a serious incident as a result, according to Aikido.
• At least five Fortune 500 companies were affected by PromptPwnd, according to Aikido.

Context

LLM application security is the control problem that starts where model guardrails end. The application layer includes the code, dependencies, developer tools, and runtime integrations that let an AI feature touch data and execute actions, which means the real exposure often sits outside the model itself.

That matters because AI development is changing the identity and trust boundaries inside software delivery. AI-generated code, hallucinated packages, and AI-assisted workflows all create new places where secrets, permissions, and third-party components can be abused without the model ever being the direct target.

Key questions

Q: What fails when LLM security is limited to model guardrails?

A: Model guardrails can shape output, but they do not inspect the workflow files, package installs, extensions or secrets that surround the model. When the real weakness sits in code or runtime context, the attacker bypasses the guardrail and uses the application layer as the path to impact.

Q: Why do hallucinated package references create real supply chain risk in AI agent workflows?

A: Hallucinated package names become dangerous when agents and humans trust them without verification. A plausible but nonexistent package can spread through copied skill files, then trigger install prompts that users approve too quickly. That turns a naming mistake into an execution path for typosquatting, slopsquatting, and unintended dependency installation across many repositories.

Q: What are the signs that shadow AI is becoming a governance problem rather than a productivity aid?

A: The clearest signs are widespread use outside IT visibility, repeated sharing of corporate data with GenAI tools, and no consistent approval path for new applications. If employees are using AI because no sanctioned option exists, or if security teams cannot see which tools are active, the issue has moved from isolated behaviour to unmanaged risk.

Q: How should teams govern privileged tokens in agentic AI workflows?

A: Treat any workflow that lets an agent act on untrusted input as a high-risk execution path. Privileged tokens should not be available by default, and the agent’s authority should be narrowed so a prompt injection or malicious document cannot turn ordinary context into secret access.

Technical breakdown

Why model guardrails miss application-layer risk

Model guardrails operate at inference time and shape what the LLM says or refuses to say. They do not inspect workflow files, dependency installs, IDE plugins, secret exposure in source, or the permissions attached to an agent running inside a CI pipeline. That is why PromptPwnd-style attacks matter: the prompt was only the entry point, while the vulnerability lived in the surrounding application logic and its privileged execution context. LLM application security therefore sits closer to SAST, secret scanning, SCA, and runtime inventory than to content moderation. It is a control plane problem, not just a model behaviour problem.

Practical implication: Security teams need to govern prompts, code, dependencies and execution context as one attack surface, not as separate workstreams.

How hallucinated packages become supply chain attacks

LLMs sometimes invent package names that sound plausible enough for developers or agents to install. Attackers can pre-register those names, a pattern often called slopsquatting, and turn a model quirk into a supply chain injection point. The technical issue is not only detection after install, but preventing untrusted packages from ever entering the dependency tree. That means scanning against known malicious names, checking provenance, and blocking install-time execution paths where hallucinated packages or compromised extensions could gain a foothold. In AI-assisted development, the package registry has become part of the threat model.

Practical implication: Treat package install as an enforcement point and block untrusted dependencies before they reach build or runtime.

Why developer environments are now part of the security boundary

AI coding assistants connect to MCP servers, extensions, local tooling, and sometimes production systems. Those connections create a trust chain that can be abused if a malicious server or compromised extension is installed on the developer machine. In practical terms, the developer workstation is no longer just a user endpoint; it is an active control surface for AI-enabled software creation. If an attacker can influence the tools the assistant trusts, they can redirect outputs, exfiltrate context, or widen access to secrets and repos. That makes device posture, extension control, and AI tool inventory core parts of LLM application security.

Practical implication: Include AI tooling, extensions and MCP servers in endpoint and software inventory controls.

Threat narrative

Attacker objective: The attacker wants to use the AI application supply chain or runtime context to steal secrets, inject malware, or expose sensitive data at scale.

1. Entry occurs through AI-generated code, untrusted prompt input, or a hallucinated package name that developers or agents accept into the workflow.
2. Credential access or abuse follows when the application or agent uses privileged tokens, exposed secrets, or trusted integrations inside GitHub Actions or similar execution environments.
3. Escalation happens as the attacker reaches code execution, supply chain insertion, or broader application context through malicious packages, compromised extensions, or misconfigured AI-generated logic.
4. Impact is secret leakage, unauthorized model usage, exposed customer data, or a broader application compromise that model guardrails alone would not stop.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Application-layer AI risk is now the decisive control problem. LLM guardrails can constrain model behaviour, but they do not govern the code, dependencies, developer tools, or runtime integrations that actually move data and privileges. That shifts the practical centre of gravity to application security, where SAST, SCA, secrets detection and runtime inventory provide the real enforcement points. For practitioners, the lesson is that model safety without application governance is incomplete.

AI-generated code has created an oversight gap that traditional review workflows cannot absorb. When 24% of production code is AI-generated, the volume and speed of change outpace manual inspection and ad hoc pull request review. The result is not just more code, but more unreviewed attack surface with inherited flaws, exposed inputs and untrusted dependencies. Practitioners should treat AI-assisted development as a governance problem, not only a productivity issue.

Slopsquatting is a named supply chain failure mode, not a niche edge case. Hallucinated package names become exploitable when attackers register them before developers or agents do. That means the control gap is package trust at install time, not just vulnerability detection after the fact. The practical implication is that dependency governance must cover provenance, install blocking and package inventory, especially where AI tools are naming libraries on behalf of developers.

Shadow AI extends identity governance into model usage and data egress. Unapproved models and undisclosed integrations create the same governance blind spots that unmanaged service accounts do in IAM: nobody owns the access, nobody reviews the data paths, and nobody can prove compliance. That makes model inventory, usage tracking and data-region awareness part of the identity boundary for AI systems. Practitioners should treat unauthorized model calls as an access problem, not merely a policy exception.

PromptPwnd exposed a workflow trust gap that many teams still underestimate. The issue was not that the model became malicious, but that privileged tokens were available to an agent acting on untrusted input. This is the same governance mistake seen in other identity failures: assuming a trusted workflow remains trustworthy after context changes. For practitioners, the control implication is to narrow privilege around agentic workflows and to verify every trust hop, not just the model output.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

AI application security will increasingly be judged on enforcement points, not on model promises. Teams that can scan at creation time, block unsafe dependencies, and inventory model usage will be better positioned than those trying to compensate with policy alone. The practical shift is to make AI tooling visible inside the same governance model you already use for secrets and privileged access.

Shadow AI creates an identity problem as much as a data problem. When applications call models nobody approved, the missing control is ownership of the access path. That means programme leaders should extend access governance to AI services, then prove that every model call has an accountable owner and a defined data boundary.

For practitioners

• Scan AI-generated code at the point of creation Run SAST and secret detection inside the IDE so untrusted prompt output is checked before it reaches a pull request or build pipeline.
• Block hallucinated and malicious packages at install time Enforce dependency controls that stop slopsquatted packages, suspicious registries and known-malicious artifacts before they enter the dependency tree.
• Inventory MCP servers and AI extensions on developer machines Treat AI tooling as part of endpoint governance and monitor which servers, plugins and extensions can influence coding assistants or access project context.
• Track model usage in production applications Record which models an application calls, where data flows, and whether those integrations are approved so shadow AI can be detected before audit findings or exposure incidents.
• Separate privileged tokens from untrusted AI inputs Keep high-value credentials out of agent contexts that process external content, and restrict what those agents can do when prompted by untrusted data.

Key takeaways

• LLM application security is the control layer that governs code, dependencies, developer tools and runtime integrations around the model.
• AI-generated code, hallucinated packages and shadow AI all expand the attack surface faster than model-only guardrails can reduce it.
• Practitioners should shift enforcement to creation time, install time and usage time, where the most practical controls can still stop abuse.

Key terms

• LLM Application Security: LLM application security is the practice of protecting systems that embed large language models into production workflows. It focuses on preventing prompt abuse, data leakage, unsafe outputs, and misuse of connected tools so the model does not become a control bypass for the broader application.
• Slopsquatting: Slopsquatting is a supply-chain attack that exploits hallucinated package names suggested by AI systems. An attacker registers the invented name in a public registry and waits for a developer or build pipeline to install it. The risk sits at the intersection of model error, dependency trust, and software delivery speed.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity and secrets management in the context of modern identity programmes. It helps security and identity teams connect access control, lifecycle governance and emerging AI-driven identity risk.