TL;DR: Among more than 4,000 Java tasks, GPT-5.2 High delivers the strongest security posture at 16 blocker vulnerabilities per MLOC, but also produces the highest code volume at 974,379 LOC, while Claude Sonnet 4.5 reaches 198 blocker vulnerabilities per MLOC and Opus 4.5 Thinking drops that to 44, according to Sonar. Functional correctness is not enough when AI-generated code also creates maintainability and review burden at scale, especially for teams shipping into production.
At a glance
What this is: Sonar's benchmark of more than 4,000 Java tasks shows that AI code generation creates a three-way trade-off between functional success, security defects, and maintainability debt.
Why it matters: Engineering and security teams need this distinction because code that passes tests can still create review burden, concurrency faults, and blocker vulnerabilities that increase production risk.
By the numbers:
- Sonar's analysis across 4,000+ Java tasks found GPT-5.2 High produced the highest code volume at 974,379 LOC.
- Claude Sonnet 4.5 reached 198 blocker vulnerabilities per MLOC, including path traversal and injection flaws.
Context
AI code generation changes the security problem from isolated coding mistakes to repeated, scalable quality drift. When a model produces working code quickly, teams can still inherit hidden debt in review effort, concurrency handling, input validation, and long-term maintainability.
The identity angle here is indirect but real: engineering teams are effectively trusting a non-human system to generate application logic that may later govern authentication, authorization, and data handling paths. That makes code quality part of broader software assurance, not just developer productivity. For teams shipping production software, the typical failure mode is assuming passing tests equals safe code.
Key questions
Q: What fails when teams choose AI coding models only by pass rate?
A: They can end up selecting models that solve benchmark tasks while generating code with hidden security defects, excessive verbosity, and large amounts of technical debt. Pass rate alone does not tell you whether the output is easy to review, safe to operate, or cheap to maintain. Governance needs quality, security, and maintainability signals together.
Q: Why do verbose AI-generated codebases increase security risk?
A: Verbose code usually adds more branches, more state, and more opportunities for inconsistent validation or error handling. That expands the review surface and makes it easier for injection, traversal, or concurrency defects to survive into production. The risk is not just size, but the extra complexity that hides flaws from both humans and automated checks.
Q: How can organisations tell whether an AI coding model is safe enough to use?
A: Use model-specific measures rather than generic vendor claims. Track pass rates by vulnerability class, compare reasoning and non-reasoning variants, and test the model on the languages and application patterns your teams actually ship. A model is only fit for use if it improves security outcomes in your environment without reducing review discipline.
Q: Should organisations treat AI code generators as developer tools or software supply chain risk?
A: They should treat them as both, but govern them as supply chain risk first. Generated code can enter the build with latent defects that are harder to detect than ordinary programming mistakes, so model evaluation, scanning, and review controls belong upstream of release. Productivity gains do not remove the need for assurance controls.
Technical breakdown
Why pass rates do not capture code safety
Pass rate measures whether generated code solves the benchmark task, but it does not capture how safely or cleanly it does so. Large language models often reach the right output through verbose, brittle, or overly complex logic. That matters because security defects, maintainability smells, and concurrency faults can remain invisible to functional tests. In practice, a model can appear strong on correctness while still creating code that is difficult to review, hard to refactor, and easier to exploit once integrated into real systems.
Practical implication: Use functional testing as a baseline only and require static analysis, code review, and security checks before accepting AI-generated code.
Why blocker vulnerabilities and smells rise together
Blocker vulnerabilities are severe issues such as injection or path traversal, while code smells are maintainability signals that increase technical debt. The article's results show these categories are not separate problems in AI-generated code. Verbose generations tend to introduce more decision points, more branching, and more opportunities for weak input handling or inconsistent patterns. That creates a compound risk: the code becomes both harder to maintain and harder to secure, especially when output is copied directly into production pathways.
Practical implication: Treat code smell density as an early warning signal for downstream security review burden and refactoring cost.
How reasoning mode changes security verification
Reasoning modes can improve constraint checking because the model spends more effort on internal consistency before emitting code. In the article, Opus 4.5 Thinking performs better on blocker vulnerabilities than a non-thinking counterpart, which suggests more deliberate verification of input handling and control flow. That does not guarantee safe code, but it can reduce obvious failures when the task requires stateful logic or trust boundary handling. The point is not that reasoning always wins, but that model behavior shifts materially across security-relevant dimensions.
Practical implication: Benchmark candidate models on security-sensitive tasks, not just completion quality, before allowing them into developer workflows.
NHI Mgmt Group analysis
AI-generated code introduces governance debt, not just development acceleration. When teams optimise for pass rate alone, they underweight the code review, remediation, and refactoring burden that arrives after generation. This is especially relevant where generated code touches authentication, session handling, or data processing logic. Practitioners should treat model selection as an assurance decision, not a productivity purchase.
High functional performance can mask security fragility. A model that solves more tasks may still produce more concurrency bugs, more verbose control flow, or more blocker-level flaws. That means the most capable model is not automatically the safest or cheapest to operationalise. Engineering leaders should compare models on defect shape, not only pass/fail outcomes.
Code smell density is the clearest signal of long-tail maintenance cost. When 92% to 96% of issues are maintainability-related, the operational impact extends beyond immediate vulnerabilities into slower change cycles and larger review queues. That is a software assurance problem as much as a coding problem. Teams should measure AI output against maintainability as a first-order control objective.
Security verification has to move left into model governance. If a model can generate working code that still contains path traversal, injection, or concurrency defects, then review cannot be limited to final output. Model evaluation, prompt constraints, and automated scanning must become part of the software supply chain. Practitioners should govern AI coding tools as upstream control points, not downstream convenience layers.
From our research library:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
What this signals
AI code generation is becoming a software assurance problem before it is a developer productivity problem. When model outputs vary so sharply in defect shape, organisations need model-level controls for review, scanning, and approval rather than assuming one benchmark score is enough.
Maintainability debt is the more universal cost centre than security defects. The article's issue profile shows that teams will spend as much effort compensating for code complexity and smell density as they will triaging outright vulnerabilities, which changes how AI coding tools should be governed.
Security teams should expect more control debt to move into the build pipeline. The practical response is to treat generated code as untrusted until it has passed the same assurance checks applied to externally sourced software components.
For practitioners
- Define acceptance thresholds for AI-generated code Set separate gates for functional correctness, blocker vulnerabilities, concurrency defects, and maintainability smells before code enters merge review.
- Run static analysis on every AI-generated change Scan generated code for injection patterns, path traversal, unsafe resource handling, and excessive complexity before approval.
- Benchmark models on security-sensitive tasks Compare candidate models using the same prompt set and measure not only pass rate but also blocker density and smell volume.
- Limit direct production use of verbose outputs Require refactoring for high-volume generations so review teams can reduce the attack surface and maintenance burden before deployment.
Key takeaways
- AI coding tools can produce code that passes tests while still creating serious review, security, and maintainability debt.
- The mix of defects matters as much as the pass rate, because verbosity and complexity expand the attack surface and the remediation workload.
- Teams should evaluate models on security-sensitive output quality and automate scanning before generated code reaches production.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while OWASP ASVS, NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V1 — Encoding and Sanitization | Injection and traversal flaws arise when generated code mishandles input encoding and validation. |
| V8 — Authorization | Generated code often embeds access-control logic that must not be trusted without review. | |
| Recommendation — Apply V1 checks to validate AI-generated code that processes user input or constructs paths. Review AI-generated authorization paths for missing checks, privilege escalation, and logic bypasses. | ||
| NIST CSF 2.0 | PR.DS-10 — Data-in-Transit is Protected | AI-generated application code can mishandle secure transport and trust boundaries in production systems. |
| Recommendation — Enforce secure communication checks on generated code that exchanges data across services. | ||
| CIS Controls v8 | CIS-16 — Application Software Security | This article is about securing code before it enters production, which maps directly to software security controls. |
| Recommendation — Gate AI-generated code through application security testing and secure coding review before release. | ||
| MITRE ATT&CK | TA0004;TA0006 — Privilege Escalation; Credential Access | The flaws discussed can enable attackers to escalate privileges or reach sensitive data paths. |
| Recommendation — Map risky generated code to privilege escalation and credential-access scenarios during threat modelling. | ||
Key terms
- Blocker Vulnerability: A blocker vulnerability is a severe software flaw that can directly enable exploitation, data exposure, or control failure if released. In AI-generated code, blocker findings often point to unsafe input handling, path traversal, injection risk, or other defects that demand rejection or remediation before production use.
- Code Smell: A code smell is a maintainability problem that makes software harder to understand, test, or change safely. It is not always an immediate exploit, but high smell density increases review burden, slows patching, and raises the chance that security defects survive future changes.
- Cognitive Complexity: A metric that estimates how hard a function or code path is to understand, based on nesting, branching, and flow changes. Higher scores usually mean more mental effort for reviewers and a greater chance of maintenance errors.
- Static analysis: Static analysis is the inspection of source code or configuration without executing it. It helps identify insecure patterns early, but its value depends on accuracy, timing, and whether the output is usable enough for developers to fix issues while they are still working on the code.
Deepen your knowledge
NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore a course that helps security and engineering teams govern machine identities, secrets, and agentic AI risk with confidence.
Published by the NHIMG editorial team on August 20, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org