LLM framework security risks expose classic injection failures

At a glance

What this is: This is an analysis of common LLM framework vulnerabilities and the recurring implementation mistakes that turn flexible orchestration features into injection and remote code execution risk.

Why it matters: It matters because teams building AI applications must treat LLM frameworks as security boundaries, especially where prompts, tools, files, databases, and external requests intersect with identity, access, and execution control.

By the numbers:

• 79% of organisations have experienced secrets leaks, with 77% of these incidents resulting in tangible damage.
• 97% of NHIs carry excessive privileges, increasing unauthorised access and broadening the attack surface.
• Only 5.7% of organisations have full visibility into their service accounts.
• 30.9% of organisations store long-term credentials directly in code.

👉 Read Flatt Security's analysis of LLM framework security risks and CVEs

Context

LLM frameworks such as LangChain, LangChainjs, LlamaIndex, and Haystack reduce the effort needed to connect models to tools, files, databases, and web sources. That same abstraction also expands the attack surface, because untrusted input can flow into URL loaders, path builders, SQL generators, templates, and code execution features without enough boundary checks.

For identity and access teams, the important issue is not only classic application security. When an LLM framework is allowed to call tools, retrieve data, or execute code, it becomes a control point for credentials, service access, and workload permissions. That makes the boundary between application logic and identity governance much more important than many AI build teams assume.

Key questions

Q: What breaks when LLM frameworks expose code execution or dangerous request features?

A: They turn prompt or input handling into an execution path. If code execution, arbitrary requests, or similar features are left enabled, a crafted prompt can become server-side command execution, internal network access, or data exfiltration. The key failure is not the model itself but the framework’s willingness to treat untrusted text as trusted instruction.

Q: Why do LLM frameworks create injection risk even when the model is not compromised?

A: Because the application can still be compromised through the framework boundary. A model can generate SQL, template content, file paths, or requests that the wrapper executes without enough validation. That means the attacker only needs control of input flowing into the workflow, not the model weights or training set.

Q: How should security teams handle URLs, paths, and queries in AI applications?

A: They should validate each class of input separately and never let free-form text shape privileged operations. URLs need allowlists, file paths need traversal checks, and generated SQL needs post-generation inspection before execution. The safest approach is to keep data, templates, and commands in different trust zones.

Q: Should organisations sandbox every risky LLM workflow or redesign it first?

A: Redesign first when the dangerous function is not essential, because removing the feature is stronger than containing it. Use sandboxing when external execution, file access, or other high-risk operations are genuinely required. In either case, pair the design choice with minimal permissions and explicit limits on runtime behaviour.

Technical breakdown

Why LLM framework abstractions create injection paths

LLM frameworks wrap several risky operations behind developer-friendly interfaces. A single chain may validate user prompts, fetch external content, query a database, render templates, and invoke tools. If each step accepts user-controlled strings without tight validation, the framework becomes a routing layer for SSRF, path traversal, SQL injection, or template injection. The problem is not the model alone. It is the combination of model output, orchestration logic, and unsafe trust in external parameters.

Practical implication: treat every framework input as untrusted and apply validation at each boundary, not only at the application edge.

How deprecated options and experimental tools become execution risks

The article shows a common pattern in which features intended for development, testing, or high-trust use are left enabled in production. Options such as dangerous request handling or Python execution wrappers can let attacker-controlled text become executable code or network activity. In LLM systems, that is especially dangerous because the model may generate apparently valid actions that the framework then executes. Security teams should recognise these options as policy decisions, not convenience settings.

Practical implication: inventory every experimental or dangerous feature and disable anything that allows code execution, arbitrary requests, or broad tool access.

Why template, SQL, and path handling need strict separation

Several of the cited CVEs arise from a simple mistake: treating data as if it were safe program logic. Path traversal appears when strings like ../ are concatenated into file paths. SQL injection appears when natural language is allowed to shape database queries without post-generation validation. Server-side template injection appears when template syntax is not separated from user data. These are old web security failures expressed through LLM orchestration patterns, which means mature secure coding practices still apply.

Practical implication: separate templates, code, and data, then enforce allowlists, sandboxing, and output validation before execution.

Threat narrative

Attacker objective: The attacker wants to convert a benign LLM workflow into arbitrary execution, data exposure, or control over the host application.

1. Entry begins when an attacker supplies crafted prompt, URL, path, SQL, or template input to an exposed LLM workflow.
2. Escalation follows when the framework turns that input into a database query, file access, network request, or code execution action.
3. Impact occurs when the attacker reads internal data, executes arbitrary commands, or manipulates the underlying application state.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• AI LLM hijack breach — attackers used stolen AWS access keys to hijack Anthropic LLM models on Bedrock.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Classic application security failures are reappearing inside LLM frameworks. The article’s strongest lesson is that AI orchestration does not replace secure coding discipline. SSRF, path traversal, SQL injection, and template injection all re-emerge when framework developers trust model output or user input too early. The practical conclusion is that AI application teams need the same boundary discipline they would apply to any internet-facing execution layer.

LLM toolchains widen the identity and access problem, not just the attack surface. When frameworks can call APIs, databases, files, or shell-like tools, they become actors that need scoped permissions and lifecycle controls. That is where NHI governance becomes relevant, because framework credentials, service identities, and execution tokens can create the same overreach problems seen in other machine-to-machine systems. Least privilege must extend to the framework’s own runtime access, not only to the model.

Experimental and deprecated options create governance debt in production AI systems. Features such as unsafe requests or dynamic code execution are often enabled because they reduce early development friction. Once shipped, they become hard to retire because applications depend on them. That creates a persistent policy gap between what the team built and what the environment can safely allow. Practitioners should treat feature selection as part of AI governance, not only engineering convenience.

Prompt injection is only one part of the control failure pattern. The article shows that secure design must also block unintended database, file, template, and command paths that the model can trigger indirectly. This broadens the control conversation from content safety to execution safety, which is where many AI governance programmes are still immature. Teams should map these risks to OWASP Agentic AI Top 10 and NIST AI Risk Management Framework controls.

Named concept: execution boundary collapse. This article repeatedly shows what happens when data, prompts, templates, and commands are allowed to cross the same trust boundary. Once that boundary collapses, the framework can no longer distinguish user intent from executable behaviour. The practitioner takeaway is to redesign the application so that generated content never becomes privileged instruction without validation.

From our research:

• 79% of organisations have experienced secrets leaks, with 77% of these incidents resulting in tangible damage, according to our Ultimate Guide to NHIs.
• Only 5.7% of organisations have full visibility into their service accounts, which is why AI toolchains should never inherit broad runtime access by default.
• For deeper context: 97% of NHIs carry excessive privileges, according to the Ultimate Guide to NHIs, and that same over-privilege pattern can appear inside LLM frameworks.

What this signals

LLM framework security is converging with identity governance because the framework itself increasingly acts like a machine identity with access to data, tools, and infrastructure. That makes permission scope, credential lifecycle, and auditability part of AI application security, not a separate NHI programme. Teams should align these controls with the NIST AI Risk Management Framework and map framework access to the Ultimate Guide to NHIs when service credentials are involved.

Execution boundary collapse: when prompts, templates, SQL, and command hooks share one trust boundary, the control failure becomes systemic rather than local. That means AI security reviews should test where generated content can become privileged action, then redesign the workflow so the model can suggest but not directly execute high-risk operations.

The next phase of AI application governance will likely be measured by how well organisations prevent untrusted model output from reaching sensitive operations. In practice, that means stronger input validation, strict tool scoping, and runtime isolation for anything that can read files, query data, or call external systems.

For practitioners

• Disable high-risk execution features by default Remove or tightly gate components that allow arbitrary code execution, dangerous request handling, or untrusted command invocation in production LLM workflows.
• Enforce allowlists for URLs and file paths Validate external URLs against an allowlist and block path strings that contain traversal patterns such as ../ before they reach loader or file APIs.
• Separate data from templates and SQL generation Keep template syntax out of user input, validate model-generated SQL before execution, and reject requests that can modify resources or escape expected structure.
• Constrain tool permissions to the minimum required Scope framework service accounts, API access, and database permissions so an abused workflow cannot pivot into broader system access or internal data.
• Add sandboxing and timeout controls for risky operations Run external command execution, file processing, and long-running inference-adjacent tasks in a sandbox with strict CPU, memory, and execution-time limits.

Key takeaways

• LLM frameworks can reintroduce familiar web application flaws when untrusted input reaches loaders, paths, SQL, templates, or execution hooks.
• The practical risk is not abstract AI misuse, but direct server-side impact through SSRF, path traversal, SQL injection, template injection, and RCE.
• The strongest control pattern is still classic security discipline: validate inputs, separate data from code, sandbox risky functions, and minimise permissions.

Key terms

• LLM Application Framework: An LLM application framework is a software layer that reduces the custom code needed to build model-powered products. It can handle retrieval, orchestration, prompt optimization, structured output, or streaming. The value is less about the model itself and more about organizing the surrounding application logic reliably.
• Execution Boundary Collapse: The point where software distribution, installation, and runtime all become one trust zone. In practice, package managers can execute code during install or import, which means a registry event can immediately become a host compromise when secrets or privileges are present.
• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Tool Permission Scoping: The practice of limiting what a non-human identity can do with each tool, including which data sources it can reach and which actions it can trigger. Strong scoping is a core control for agent governance because it reduces overreach, improves auditability, and supports revocation.

What's in the full article

Flatt Security's full article covers the implementation detail this post intentionally leaves for the source:

• Code examples showing how each vulnerable LLM framework pattern is triggered in practice
• CVE-by-CVE breakdown of LangChain, LangChainjs, Haystack, and LlamaIndex failure modes
• Specific patch and mitigation examples, including validation logic and sandboxing patterns
• Developer guidance for deciding when dangerous framework features should be removed entirely

👉 Flatt Security's full article covers the vulnerability examples, countermeasures, and implementation lessons in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and the access control patterns that matter when AI systems need scoped runtime permissions. It is designed for practitioners who need to govern machine access with the same discipline they apply to human identity.