Why do LLMs still produce vulnerable code even when prompts are well written?

Well written prompts are not enough because prompts rarely convey the environment a function will run in. LLMs need context about whether code handles PII, is externally exposed, or must satisfy specific security controls. Without those signals, they often optimize for functional correctness and miss security requirements that humans would normally infer from system context.

Why This Matters for Security Teams

Well written prompts can still produce vulnerable code because the prompt is not the same thing as the execution environment. An LLM may generate code that is syntactically correct, but it often lacks the context needed to infer whether the function handles PII, processes untrusted input, or sits behind an external API. That gap matters because the model will usually optimize for the stated task, not the hidden security expectations.

Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 is consistent on one point: model output quality is only one part of the control problem. Secure code requires constraints, policy, and review boundaries that are explicit enough for both humans and machines to act on. NHIMG research on the Analysis of Claude Code Security shows that code-generation tools become materially safer when security requirements are surfaced as first-class inputs rather than assumed from prompt wording alone.

In practice, many security teams discover prompt weaknesses only after generated code has already reached review, test, or production workflows, rather than through intentional security design.

How It Works in Practice

Prompt quality helps, but it does not replace structured context. A prompt like “write a login handler” tells the model the function goal, not the risk profile. The model does not reliably know whether the code must resist injection, protect secrets, enforce tenant boundaries, or meet a compliance standard. That is why teams get better results when prompts are paired with explicit security metadata, policy checks, and environment-specific constraints.

Practitioners increasingly use the same pattern recommended in CSA MAESTRO agentic AI threat modeling framework: define the task, define the data classification, define the trust boundary, and define the required controls before generation. In code-assist workflows, that can mean telling the model that the function is internet-facing, that inputs are attacker-controlled, or that secrets must never be printed or logged. It also means giving reviewers a clear checklist tied to policy, not just style.

• Specify whether the code handles PII, credentials, regulated data, or public input.
• Require the model to use approved libraries, safe defaults, and explicit validation.
• Attach policy-as-code checks so insecure patterns are caught after generation.
• Use secure coding rules from the start, rather than hoping the model infers them.

NHIMG’s 12,000 Secrets Found in Public LLM Training Dataset reinforces the larger point: models can reproduce insecure or sensitive patterns if the task framing does not actively constrain them. This is also why NIST’s NIST AI 600-1 Generative AI Profile emphasises governance, measurement, and monitoring, not just prompt design. These controls tend to break down when teams let code assistants operate against live schemas, production secrets, or broad repository access because the model cannot infer the full blast radius from text alone.

Common Variations and Edge Cases

Tighter prompting often increases workflow overhead, requiring organisations to balance developer speed against review depth and policy maintenance. That tradeoff is real, especially in fast-moving engineering teams.

There is no universal standard yet for how much context should be embedded in prompts versus enforced by downstream tooling. Best practice is evolving, but current guidance suggests treating prompts as one control layer, not the control layer. For low-risk utility scripts, a concise prompt and lightweight review may be enough. For code that touches authentication, payments, tenant isolation, or production data, security context must be explicit and machine-checkable.

Edge cases also matter. A prompt that is safe in a sandbox can become unsafe when the same template is reused for a production service. Similarly, a model may generate acceptable code for one language or framework and unsafe code for another if the secure-by-default libraries differ. That is where the combination of Analysis of Claude Code Security and the OWASP Top 10 for Agentic Applications 2026 is useful: they remind teams to test for context loss, prompt injection, and unreviewed assumptions. The practical rule is simple. If a human reviewer would need environment knowledge to judge the code, the model needs that context too.

Related resources from NHI Mgmt Group

• Why do LLMs still produce unsafe outputs after passing basic safety tests?
• Why do AI-generated systems still need human review even when the code looks correct?
• Why do even well-trained employees still fall for spear phishing in organisations with strong awareness programmes?
• Why do compiled applications still expose sensitive data even after code is built and shipped?