GPT-5 code output raises the cost of verifying AI-generated software

At a glance

What this is: Sonar’s updated LLM coding analysis finds GPT-5 improves functional performance, but still produces complex, issue-prone code that needs automated verification.

Why it matters: For IAM, NHI, and broader security programmes, this matters because AI-generated code can introduce security, access, and maintainability risk faster than teams can review it manually.

By the numbers:

• GPT-5-minimal introduced 3.90 issues for every correct solution, nearly double the rate of Claude Sonnet 4.
• GPT-5-minimal produced a vulnerability density of 0.12 per KLOC, the lowest in the study.
• GPT-5-minimal produced a code smell density of 25.28 per KLOC, showing a heavy maintainability burden.

👉 Read Sonar's analysis of GPT-5 coding quality, complexity, and security

Context

AI-generated code only creates value when teams can verify it at the same pace it is produced. The core governance gap is not whether a model can solve a benchmark task, but whether the resulting code is secure, maintainable, and reviewable in a real engineering workflow. That is especially important for identity-heavy systems, where flaws in authentication, authorisation, or secret handling can turn a functional snippet into an operational risk.

Sonar’s updated analysis uses static analysis to compare several leading models on Java coding tasks, making the finding more useful than a simple benchmark leaderboard. The message for practitioners is that model quality is multi-dimensional: higher pass rates do not automatically mean better production readiness. In AI-assisted development, verification controls matter as much as generation quality, and that applies equally to human-written and machine-generated code.

Key questions

Q: How should teams validate AI-generated mobile code before release?

A: Teams should validate AI-generated mobile code against mobile-specific security standards, then test authenticated flows that exercise token handling, local storage, and backend API access. A clean compile or successful QA pass is not enough. The release decision should depend on whether the generated code has been checked under realistic runtime and identity conditions.

Q: Why do high benchmark scores not guarantee safe AI-generated software?

A: High benchmark scores only show that a model can solve a task, not that its output is clean, maintainable, or secure. A model may generate verbose code, tangled control flow, or repeated smell patterns that increase review burden and future defect risk. Production safety depends on verification controls, not on benchmark reputation alone.

Q: What do organisations get wrong about AI coding tools?

A: They often treat prompting skill as the main issue when the real problem is product fit, workflow design, and control placement. If users need deep tribal knowledge just to get acceptable output, the programme has an adoption and governance problem, not only a training problem.

Q: Should organisations use the same controls for human-written and AI-generated code?

A: Yes. From a security and governance perspective, the code’s source matters less than the risk it introduces. Identity logic, secrets handling, and authorization checks should all pass the same review and testing standards regardless of who or what wrote them. That keeps release criteria consistent and prevents model output from becoming a special case.

Technical breakdown

Why higher pass rates do not equal production-ready code

Benchmark pass rates measure whether a model can satisfy a task prompt, not whether its output is fit for long-term software operation. A model can complete more tests while still generating code that is verbose, structurally awkward, or difficult to review. That matters because production risk often lives in complexity, maintainability, and control flow, not just obvious compile failures. When static analysis finds dense smells and repetitive logic, the real issue is downstream cost: more time for remediation, more review burden, and more chance that insecure patterns survive into release. Functional success and engineering quality are related, but they are not interchangeable.

Practical implication: Treat benchmark scores as a capability signal, not a release gate, and run automated static analysis on every AI-generated change set.

Code smells, complexity, and the hidden governance cost

Code smells are maintainability indicators that point to deeper design problems, such as overly long methods, duplicated logic, or tangled dependencies. Cyclomatic and cognitive complexity measure how hard code is to reason about and modify. When these metrics rise, review quality usually falls because humans have to spend more time understanding structure before they can assess security or correctness. In AI-assisted development, this creates governance debt: the code may work today, but it becomes expensive to maintain, refactor, and secure tomorrow. That debt is especially risky in systems that handle credentials, permissions, or policy enforcement.

Practical implication: Set complexity thresholds for AI-generated code and require refactoring before acceptance when those thresholds are exceeded.

Why security-focused output still needs policy enforcement

A lower vulnerability density does not mean a model is safe to use without controls. Security issues can be rare while still being high impact, and models may reintroduce specific flaw classes such as injection or path traversal even when overall vulnerability counts look favourable. That is the governance lesson in this report: quality assurance must be model-agnostic and policy-driven, because output behaviour changes by model and by task. For identity and access systems, even one flawed authorization path or secret-handling mistake can create disproportionate blast radius across the environment.

Practical implication: Apply the same security policy checks, review rules, and approval workflow to all model outputs, regardless of the model’s apparent quality profile.

NHI Mgmt Group analysis

AI-generated code now needs governance as a control plane, not a convenience layer. The report shows that better benchmark performance does not remove the need for structural verification, because complexity and maintainability risks remain high even when functional results improve. In practice, AI code generation expands the attack surface for software delivery unless static analysis, policy checks, and review gates operate as mandatory controls. For teams running identity-heavy applications, that means the code pipeline itself becomes part of IAM and security governance.

Model-agnostic verification is the right operating assumption. The central lesson is not that GPT-5 is uniquely risky, but that no model should be trusted to produce production-ready code without independent verification. That aligns with the broader security principle that control effectiveness cannot depend on the source of the change. Security teams should therefore standardise checks across human and AI contributors, because the control objective is code quality and safety, not model reputation.

Complexity debt is the named risk this analysis makes visible. GPT-5-minimal’s output shows that even when vulnerability counts stay relatively low, code smell density and cognitive complexity can rise sharply. Complexity debt: the accumulation of hard-to-read, hard-to-maintain, and hard-to-review code that increases future defect and security risk. Teams that ignore this will see review bottlenecks, slower remediation, and a larger chance that a small logic mistake becomes an access-control failure.

Security-focused AI output still needs defensive depth. The report’s lower vulnerability density is useful, but it can create false comfort if organisations stop at aggregate security metrics. The more relevant question is whether the generated code changes the review burden, introduces recurring flaw patterns, or complicates remediation. That is why code generation governance should be tied to secure SDLC controls, not treated as a separate AI issue.

Identity and access systems deserve the strictest scrutiny because their failure modes compound quickly. Any AI-generated code that touches session management, entitlement checks, secrets handling, or policy decisions can magnify downstream risk. For practitioners, the lesson is straightforward: do not allow model output to bypass the same control expectations applied to sensitive human-authored code.

What this signals

Complexity debt will become a stronger governance signal as AI code generation scales. Teams that already struggle with review capacity will feel that pressure first, because verbose output forces more human time into understanding structure before security can be judged. The operational answer is to combine static analysis with policy thresholds and, where relevant, identity-centric review for code that touches credentials or access paths.

The next maturity step is to stop asking whether a model is better and start asking whether the delivery pipeline can prove that generated code meets the same safety criteria as human-written code. That is the control shift from experimentation to governance. Where code handles secrets, sessions, or privilege logic, AI output should be treated as sensitive change and reviewed accordingly.

For practitioners

• Enforce static analysis on every AI-generated commit Run automated static analysis and security scanning before merge, and block release if the code fails maintainability or security thresholds.
• Set complexity thresholds for AI-assisted code Define maximum acceptable cyclomatic and cognitive complexity for generated code, then require refactoring when outputs exceed those limits.
• Apply the same review rules to all model outputs Use one approval workflow for human and AI contributors so that benchmark reputation never replaces independent code verification.
• Prioritise identity-sensitive paths for manual review Require extra scrutiny for authentication, authorization, session handling, and secret management code because small defects in these paths can create disproportionate blast radius.

Key takeaways

• GPT-5 improves functional coding output, but the report shows that higher pass rates do not eliminate the need for independent verification.
• The bigger operational risk is complexity and review burden, with more defects per passing task and much higher code smell density.
• Security teams should standardise static analysis, complexity thresholds, and policy-based review for every AI-generated change set.

Key terms

• Code Smell: A code smell is a sign that software may be harder to understand, test, or maintain than it should be. It does not automatically mean the code is broken, but it often points to structural problems that raise future defect and security risk, especially in fast-changing AI-generated code.
• Cyclomatic Complexity: Cyclomatic complexity measures how many distinct paths exist through a piece of code. Higher values usually mean more branching, more review effort, and more room for logic mistakes, which is why this metric matters when evaluating AI-generated software for production readiness.
• Static analysis: Static analysis is the inspection of source code or configuration without executing it. It helps identify insecure patterns early, but its value depends on accuracy, timing, and whether the output is usable enough for developers to fix issues while they are still working on the code.
• AI-Generated Code Risk: AI-generated code risk is the possibility that model-produced output introduces insecure credential handling, hidden secrets, or repeated patterns that weaken control design. The risk is operational, because the output can be merged quickly and at scale before humans catch the problem.

What's in the full report

Sonar's full analysis covers the detailed benchmark methodology and issue breakdown this post intentionally leaves for the source:

• Side-by-side performance tables for GPT-5-minimal, Claude Sonnet 4, Claude 3.7 Sonnet, GPT-4o, Llama 3.2 Vision 90B, and OpenCoder-8B
• Issue-type breakdowns showing where code smells, bugs, and vulnerabilities concentrate across model outputs
• Methodology notes on the Java benchmark set and the static analysis approach used to score generated code
• Model-specific observations on verbosity, complexity, and recurring flaw patterns that help teams interpret the results in practice

👉 Sonar's full post includes the benchmark tables, defect patterns, and methodology details behind the update.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle control. It helps security and identity practitioners build the governance discipline needed for modern automation and AI-driven delivery.