LLM application security is shifting to code, supply chain, and runtime

At a glance

What this is: The article argues that LLM security belongs in the application layer, where AI-generated code, dependencies, developer tools, and runtime model usage create risks model guardrails do not address.

Why it matters: For IAM and security teams, this matters because AI-assisted development expands the identity and access surface around code, tools, and models, creating new governance gaps across secrets, permissions, and shadow AI.

By the numbers:

• 24% of production code is now AI-generated, according to Aikido's State of AI in Security and Development 2026 report.
• 69% of organizations have discovered vulnerabilities in AI-generated code, and 20% reported a serious incident as a result.
• At least five Fortune 500 companies were affected when untrusted input reached agents with privileged tokens in GitHub Actions workflows, according to Aikido's PromptPwnd research.

👉 Read Aikido's comparison of top LLM security tools for AI applications

Context

LLM application security is the discipline of protecting the code, dependencies, developer workflows, and runtime integrations around AI applications. The article's core point is that model guardrails alone do not address the places where many real incidents begin, especially in AI-generated code and agentic tooling.

That matters because AI has expanded the security boundary into places identity teams already care about: privileged tokens in workflows, secret exposure, model access in production, and unauthorised tool use. The article is representative of a broader market shift, not an isolated case, because organisations are now discovering that LLM risk is an access-control and software-supply-chain problem as much as a model-safety problem.

Key questions

Q: How should security teams govern AI-generated code in production environments?

A: Security teams should treat AI-generated code as normal production code with extra provenance risk. Require architectural review, test coverage, static analysis, and approval before merge. Then bind the agent and the build pipeline to least privilege, short-lived credentials, and complete audit logging so implementation speed does not outrun control.

Q: Why do AI coding agents increase supply-chain risk?

A: AI coding agents can choose tooling, install packages, and edit lock files in ways that may bypass the controls humans expect in a managed environment. That widens the blind spot around untracked software and makes package provenance harder to enforce. The practical response is to restrict agent permissions, inventory their workspaces, and review their install behaviour like any other privileged automation.

Q: What do teams get wrong about LLM-as-judge guardrails?

A: They often assume the judgment layer is a replacement for other controls. In practice, it is another control with its own failure modes, latency, cost, and bypass surface. It should be used for targeted high-risk decisions, backed by logging and adversarial testing, not as a universal fix for every AI workflow.

Q: How should organisations govern shadow AI without blocking legitimate use?

A: Start with approved-use policy, tool inventory, and data classification. Then require that any AI system handling internal information has named owners, logged access, and defined credential paths. The goal is not prohibition, but visibility and control. If a tool cannot be inventoried or monitored, it should not process sensitive data.

Technical breakdown

LLM application security vs model security

LLM application security focuses on the software layer around the model: source code, prompts, dependencies, CI/CD, developer tools, and runtime model calls. Model security, by contrast, covers the behaviour of the model itself, including prompt injection resilience, guardrails, and red-teaming. The article is right to separate the two because many failures happen before a model even answers a prompt. If untrusted input reaches a privileged workflow, or an AI assistant installs a malicious package, the control failure sits in application design, not in the model response path.

Practical implication: security teams need separate controls for application-layer risk and model-layer risk, not a single LLM guardrail strategy.

AI-generated code, SAST, and secrets detection

When AI writes code, it increases output volume faster than human review capacity. That does not automatically mean the code is unsafe, but it does mean security findings, hardcoded secrets, and unsafe prompt handling can enter production at greater speed. SAST and secrets detection remain relevant because they catch insecure patterns in the code actually deployed, including untrusted input flowing into prompts and credentials exposed to agentic tools. The issue is governance, not novelty: AI code still needs the same control objectives as human code, only at a much higher cadence.

Practical implication: run SAST and secret scanning at creation time in the IDE, not only later in CI or release gates.

Hallucinated packages, MCP servers, and shadow AI

The article connects three related risks: slopsquatted packages, malicious MCP servers, and shadow AI. Hallucinated package names become supply-chain entry points when attackers publish those names in advance. MCP servers extend trust into tools and context sources that agents can invoke, which means a compromised extension or server can become an access path. Shadow AI adds the governance problem of uncatalogued model usage, where teams cannot prove which providers, regions, or data flows are active. Together, these are identity and trust problems disguised as developer productivity features.

Practical implication: inventory model calls, restrict tool installation, and treat MCP-enabled development as part of the privileged software supply chain.

Threat narrative

Attacker objective: The attacker aims to hijack AI-assisted development and runtime workflows to steal secrets, manipulate code paths, and expand access through trusted automation.

1. Entry occurs when attackers exploit untrusted input, hallucinated package names, or compromised developer tooling to reach AI-assisted workflows.
2. Escalation follows when those workflows hold privileged tokens, trusted prompts, or installation rights that let the attacker influence code, packages, or model context.
3. Impact occurs when secrets leak, unauthorized models are called, or malicious code and data flow into production systems and prompt logs.

NHI Mgmt Group analysis

Application-layer LLM security is now a governance requirement, not a niche AppSec concern. The article correctly shifts the focus from model behaviour to the code, tools, and data paths that surround LLMs. That is where most real-world exposure appears, including secrets in workflows, unsafe prompt construction, and dependency risk. For identity teams, the lesson is direct: if an AI system can call tools or access credentials, it is operating inside an access-control boundary, not outside it.

AI-generated code creates an LLM security debt problem. The more code AI produces, the more organisations inherit unreviewed patterns at speed, and the harder it becomes to keep review, scanning, and remediation in sync. This does not change the control objectives, but it changes the operating model. Security teams need continuous enforcement in the IDE, in dependency selection, and in release pipelines, or the backlog becomes structurally unmanageable.

MCP-enabled development expands the trusted computing base into developer tools. The article highlights a category that many AppSec programmes still under-model: agents, plug-ins, and context servers that can touch secrets or execute actions on behalf of a user. That is an identity problem as much as a tooling problem because it depends on who or what is allowed to act, under which permissions, and for how long. Practitioners should treat MCP access like privileged integration, not ordinary extensibility.

Shadow AI makes governance evidence impossible if model usage is not inventoried. The strongest control in the article is visibility into which models are actually called in production. Without that, compliance, data residency, and incident response all lose precision. The broader market signal is clear: AI governance is moving from policy language to telemetry, and teams that cannot enumerate model access will struggle to enforce any control consistently.

OWASP-style application controls now define the practical baseline for LLM risk management. The article maps naturally to OWASP LLM application risk categories because the failures are concrete: supply chain, improper output handling, excessive agency, and misuse of external dependencies. That is the right lens for current programmes. Teams should use those categories to define scope, then back them with SAST, SCA, secrets handling, and runtime inventory.

What this signals

LLM security programmes are converging with identity governance. Once applications can call models, install packages, and interact with MCP servers, the question is no longer only whether code is safe. It becomes who or what is authorised to act, what secrets are exposed, and how quickly the organisation can prove that shadow AI is not active outside policy. The practical control shift is toward telemetry-rich inventory and approval boundaries rather than isolated point tools.

Application security teams should expect AI governance to become measurable, not aspirational. The article points toward a future where model usage tracking, dependency control, and developer environment protection are as auditable as access reviews. That lines up with the broader evidence that agent behaviour is already exceeding intended scope in many environments, and it reinforces why identity visibility matters across AI pipelines and production systems.

LLM security debt: the faster AI increases code throughput, the more organisations need automated enforcement to prevent unreviewed patterns from accumulating across code, packages, and runtime model calls. Teams that still rely on periodic review will see the gap widen between policy and actual execution.

For practitioners

• Implement IDE-time scanning for AI-generated code Move SAST and secrets detection into the IDE so untrusted input, hardcoded credentials, and unsafe prompt patterns are caught when code is written, not after it enters a pull request.
• Block hallucinated and malicious packages before install Use package-firewall controls and dependency allowlists to stop slopsquatted npm or Python packages at install time, and tie exceptions to explicit approval workflows.
• Treat MCP servers as privileged integrations Inventory every MCP server, extension, and connected tool that can reach prompts, files, or secrets, then require approval and monitoring before those tools are trusted in developer environments.
• Inventory production model usage continuously Track which models each application calls, where data is sent, and whether any shadow AI integrations are active so compliance and incident response can rely on evidence rather than assumptions.
• Separate application controls from model guardrails Define one control set for code, dependencies, and runtime integrations, and a second for prompt safety and model behaviour so teams do not mistake one layer of coverage for the other.

Key takeaways

• LLM risk is increasingly an application-layer problem, because code, dependencies, and developer tooling create exposure that model guardrails cannot stop.
• AI-generated code and agentic workflows expand the attack surface faster than manual review can absorb, so continuous scanning and inventory are now operational requirements.
• The practical control gap is visibility into who or what can call models, install packages, and use privileged tokens inside AI development pipelines.

Key terms

• LLM Application Security: LLM application security is the practice of protecting systems that embed large language models into production workflows. It focuses on preventing prompt abuse, data leakage, unsafe outputs, and misuse of connected tools so the model does not become a control bypass for the broader application.
• Slopsquatting: Slopsquatting is a supply-chain attack that exploits hallucinated package names suggested by AI systems. An attacker registers the invented name in a public registry and waits for a developer or build pipeline to install it. The risk sits at the intersection of model error, dependency trust, and software delivery speed.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.
• MCP Server: An MCP server is a tool endpoint that connects an AI agent to external systems and data sources through Model Context Protocol. Because it extends what the agent can reach, it becomes part of the identity and access surface and must be reviewed like any other privileged connector.

What's in the full article

Aikido's full blog covers the operational detail this post intentionally leaves for the source:

• Side-by-side feature comparison across Aikido, Snyk, Semgrep, Endor Labs, and Wiz for AI application security workflows.
• Hands-on explanation of how Aikido's IDE scanning, package blocking, and runtime tracking are positioned in the development lifecycle.
• Detailed treatment of PromptPwnd, slopsquatting, and malicious MCP server threats that underpin the product evaluation.
• Tool-by-tool coverage notes for teams deciding whether they need code-side, supply-chain, or environment-level controls.

👉 Aikido's full post covers the tool-by-tool coverage, workflow tradeoffs, and implementation details behind LLM application security.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle fundamentals. It helps security practitioners connect access control, credential risk, and operational governance across modern identity programmes.