What happens when a code model can find vulnerabilities but cannot reliably avoid introducing them?

Why this creates the wrong kind of confidence

A code model that can spot vulnerabilities but still writes them creates asymmetric value: it improves detection and review assistance, but it does not reduce the probability that unsafe code will be produced in the first place. That matters because teams may treat stronger analysis as a sign that generation is becoming safer, when the real control gap is prevention.

In practice, this often shifts effort rather than removing it. Engineers still need to inspect outputs, and the model may increase the volume of security-relevant findings without lowering the number of defects that reach review. The outcome is better offensive insight, but not a corresponding drop in engineering exposure.

Where the risk shows up in the delivery pipeline

The failure is not limited to obvious security bugs. A model that can identify insecure patterns but not reliably avoid them can still emit weak authorization logic, unsafe input handling, fragile secrets handling, or broken trust assumptions. If the output is accepted with too much confidence, the organisation can normalize insecure code as an acceptable baseline and miss repeated drift across many changes.

This also changes how review work scales. The more often the model produces risky constructs, the more reviewers are forced into manual correction mode instead of higher-value validation. That can slow delivery, increase change fatigue, and create a backlog of “known but unprevented” issues that are easy to underestimate during planning.

What good operating practice looks like

Use the model as a reviewer aid, not as evidence that the generated code is safe by default. The useful question is whether it reduces defect introduction, not whether it can explain defects after the fact. In a secure engineering workflow, outputs still need policy checks, test coverage, and human judgment before they are trusted in production.

When a model repeatedly finds the same vulnerability classes it also tends to produce, that is a sign to tighten the development control plane rather than expand usage. Teams should tune prompts, constrain allowed patterns, and add automated checks that fail closed on known-risk constructs instead of relying on post-generation review alone. Guidance on secure-by-design product expectations is aligned with the EU Cyber Resilience Act, which pushes security into the product lifecycle rather than leaving it to end-stage review.

Risk and Threat Considerations

This pattern can create a false sense of control because the model appears security-aware while still introducing exploitable code paths. If insecure output is accepted at scale, the main risk is not one bad snippet, but repeated exposure across many commits, services, and teams, with human reviewers becoming the last line of defence.

Failure mechanism: The model detects vulnerability patterns but does not reliably suppress them during generation, so insecure constructs still enter the codebase and must be caught later by people or downstream tooling.

Impact: Review burden rises, insecure patterns can reach production, and organisations may overestimate the protective value of the model while actual engineering risk remains largely unchanged.

Practitioner Guidance

What to verify: Treat the model as effective only if it measurably reduces defect introduction rates, not just finding rates. A useful test is whether the same issue still appears in generated code after policy constraints, secure templates, and automated checks are applied.

Decision rule: If the model is better at explaining vulnerabilities than preventing them, keep humans and automated controls in the enforcement path. If it cannot consistently avoid a class of defect, do not delegate that class to unconstrained generation.

What practitioners underestimate: Security review is not free when the model “knows” the vulnerability. The real cost is that the organisation still pays for detection, correction, and rework, while the trust risk increases if teams assume the model has already made the code safe.

Practitioner takeaway: The value of a code model is judged by how much risky code it prevents, not by how well it can criticise risky code after producing it.

Related resources from NHI Mgmt Group

• What breaks when AI tools find code vulnerabilities but cannot prove exploitability?
• What happens when software manufacturers cannot prove they handled vulnerabilities and AI-generated code with sufficient controls?
• What happens when organisations rely on a generative model that cannot reliably distinguish safe from unsafe prompts?
• What happens when FinTech teams cannot reliably separate real vulnerabilities from noise?