AI-generated code needs hybrid AppSec controls, not model-only security

At a glance

What this is: This is an application security analysis of why AI-generated code can be functional without being secure, and why hybrid controls are needed to close that gap.

Why it matters: It matters to IAM and security teams because AI-assisted development expands the attack surface, introduces new governance points in the software lifecycle, and requires better control over AI components, secrets, and code provenance.

By the numbers:

• Frontier models now produce working code 83% to 95% of the time, but secure and functional code reaches only 24% to 36%.
• Adding inference-time security steps raises the secure and functional rate from 24% to 36% to 47% to 56%.
• Checkmarx says its hybrid engine achieved an F1 score of 0.64 across seven production codebases.

👉 Read Checkmarx's analysis of AI-generated code, hybrid AppSec, and AI Inventory

Context

AI-generated code changes the security problem from finding defects after the fact to governing how code is produced, checked, and remediated. The central issue is not whether models can write code, but whether the resulting software is secure, auditable, and tied to a defensible control process. In practice, that means AppSec now has to manage both deterministic enforcement and probabilistic generation, while tracking the AI components embedded in the development stack.

That creates a governance problem for IAM-adjacent programmes as well. AI tools, agents, MCP servers, SDKs, and automated remediation systems all become part of the delivery chain, with their own identity, access, and accountability questions. Checkmarx frames the answer as a hybrid model, but the broader lesson is that AI-assisted development needs independent verification, lifecycle visibility, and control points that do not depend on the same model producing the code.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: When does AI-assisted development create more risk than it reduces?

A: It becomes net risk when code volume grows faster than ownership, review, and fix capacity. That is especially true when secrets, dependencies, and business logic are handled in separate tools. If teams cannot prioritise by reachability and impact, speed turns into hidden debt.

Q: What do security teams get wrong about AI-generated code risk?

A: They often focus on catching insecure output after code is written, which is too late for AI-native workflows. The more important control point is the moment the agent is allowed to initiate the action. If that step is not governed, testing becomes a detection layer rather than a prevention layer.

Q: How can organisations reduce AI-driven vulnerability backlog faster?

A: Use context-aware remediation that prioritises reachable, exploitable issues and generates fixes developers can apply quickly. Combine that with clear ownership and policy gates so the backlog shrinks without weakening controls. The goal is faster verified removal of risk, not just more alerts.

Technical breakdown

Why AI-generated code is often functional but still insecure

Large language models can generate syntactically valid code that passes basic tests, but security is a different property. Functional correctness measures whether code works. Secure code also needs safe data handling, correct access boundaries, defensive error handling, and absence of known vulnerability patterns. When a model is optimising for speed or plausibility, it can produce code that appears right while still embedding injection flaws, unsafe deserialisation, weak authentication flows, or risky dependencies. That gap widens when developers trust generated output too early in the lifecycle.

Practical implication: Treat model output as untrusted code until it passes independent security validation.

Deterministic and probabilistic controls in hybrid AppSec

Deterministic controls apply fixed rules and repeatable checks, such as policy enforcement, static analysis, and reachability assessment. Probabilistic controls use AI reasoning to prioritise findings, infer context, and recommend fixes. The value of a hybrid model is not that AI replaces rules, but that it improves triage and remediation while deterministic tools preserve auditability and consistent enforcement. That matters because relying on one model to generate and judge code creates a circular control problem with no independent check on its own output.

Practical implication: Pair AI-assisted review with rule-based enforcement so the same system is not judging its own work.

AI inventory and the identity of AI components

AI inventory is about discovering and governing the components that participate in code production, including models, agents, MCP servers, and SDKs. In identity terms, those components behave like managed non-human identities because they make requests, call tools, and influence runtime behaviour. Without inventory, teams cannot assign ownership, monitor access, or determine which AI components can reach sensitive systems or generate deployable artifacts. This is where application security intersects directly with NHI governance.

Practical implication: Inventory AI components as governed identities, not just as software libraries.

Threat narrative

Attacker objective: The attacker objective is to get insecure or manipulated code accepted into the software supply chain and into production.

1. Entry occurs when developers accept AI-generated code, or AI-assisted fixes, into the delivery pipeline without independent verification.
2. Escalation follows when insecure code, risky dependencies, or overbroad AI component access are promoted into build, test, or runtime environments.
3. Impact appears as vulnerabilities moving into production faster than teams can detect, triage, or remediate them, increasing exposure across the application estate.

NHI Mgmt Group analysis

AI-generated code creates an assurance gap, not just a productivity gain. The real problem is that output quality and security quality are not the same thing, and organisations often optimise for throughput first. That creates a control gap in which functional code is accepted before it is independently verified. For AppSec teams, the practitioner conclusion is simple: generation speed cannot become the security standard.

Deterministic verification remains the control plane for trust. AI can assist with triage and remediation, but it cannot be the only layer checking the output it helped create. The article reinforces a broader AppSec principle that auditability and reproducibility matter more when code creation is accelerated by models. For security architects, the conclusion is that deterministic enforcement must stay in place even as AI expands.

AI Inventory is really an identity and governance problem. The moment models, agents, MCP servers, and SDKs can influence code or runtime behaviour, they need lifecycle control, ownership, and visibility similar to other non-human identities. AI governance debt: this is the growing gap between how quickly AI components are adopted and how slowly they are inventoried, authorised, and reviewed. The practitioner conclusion is to govern AI components as assets with identity-like accountability.

Mass remediation will matter more than mass detection in AI-assisted development. When code production accelerates, backlog reduction becomes a control objective, not just an efficiency metric. The article points toward a shift in AppSec from finding more issues to fixing them faster with context-aware remediation. For programme leaders, the conclusion is to judge tooling by how much verified risk it removes from the pipeline.

What this signals

AI governance debt: organisations that scale AI-assisted coding faster than they scale verification will accumulate a backlog of untrusted outputs, undocumented components, and inconsistent remediation decisions. That debt shows up later as audit failures, repeated findings, and weak assurance over what actually reached production.

For identity and platform teams, the practical signal is that models, agents, MCP servers, and remediation services now need lifecycle ownership, not just DevOps convenience. Aligning those components to the NHI Lifecycle Management Guide helps turn them into governable assets instead of unmanaged contributors to the delivery chain. The same discipline also reinforces policy alignment with NIST SP 800-53 Rev 5 Security and Privacy Controls.

Security leaders should expect AI-assisted development to shift budget conversations from detection volume to verified remediation throughput. The organisations that gain control fastest will be the ones that can show how many findings were removed, how quickly fixes were validated, and which AI components were actually under governance.

For practitioners

• Embed independent verification before merge Require every AI-generated change to pass deterministic checks, security review, and policy enforcement before it can merge into the main branch. Do not let the model that produced the code be the only system evaluating it.
• Inventory AI components as governed assets Track models, agents, MCP servers, and SDKs in an AI-BOM so ownership, access, and review obligations are explicit. Use the NHI Lifecycle Management Guide to align onboarding, review, and offboarding of these components.
• Prioritise risk-based remediation over raw finding volume Use reachability, exploitability, and business context to sort findings so teams fix what can actually be abused first. This reduces noise and keeps remediation aligned to real exposure rather than scanner output.
• Limit AI-assisted fixes to controlled remediation paths Allow AI-generated remediation suggestions only where policy, test coverage, and code review can validate the change. Treat any automated fix as a recommendation until it has been checked by a separate control layer.

Key takeaways

• AI-generated code can be highly functional while still failing basic security expectations, so correctness and assurance must be measured separately.
• Hybrid AppSec works because deterministic controls preserve trust while AI improves triage and remediation, not because the model becomes self-verifying.
• AI components now behave like governed non-human identities, which means inventory, ownership, and lifecycle control are part of application security.

Key terms

• AI-BOM: An AI bill of materials is a structured inventory of the components that define an AI agent, including the model, prompt, tools, retrieval sources, and dependencies. In practice, it is the evidence base for review, change control, and risk assessment when the agent evolves after deployment.
• Deterministic Capability Control: Deterministic capability control means limiting what an agent can do through code or system policy, not through model persuasion. In agentic AI, this is the difference between asking for safe behaviour and making unsafe behaviour impossible within the runtime boundary.
• Probabilistic Control: A probabilistic control uses model inference or AI reasoning to infer context, rank findings, or recommend actions. It can speed analysis and improve relevance, but it cannot replace independent verification because its conclusions vary with inputs, training, and prompt context.
• Secure And Functional Code: Secure and functional code is output that both performs the intended task and resists common security failure modes. It must satisfy correctness and security at the same time, which is why it is a stricter standard than either working code or vulnerability-free snippets alone.

What's in the full article

Checkmarx's full article covers the operational detail this post intentionally leaves for the source:

• The step-by-step breakdown of its hybrid AppSec workflow for deterministic checks and AI-assisted reasoning
• The specific benchmark setup behind the reported F1 score and secure-code rates
• The product-level detail behind Triage Assist, Remediation Assist, Developer Assist, and AI Inventory
• The full discussion of the AI development lifecycle control points and remediation economics

👉 The full Checkmarx article covers the benchmark details, remediation workflow, and AI Inventory functions in more depth.

Deepen your knowledge

NHI Mgmt Group’s NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security programme that AI-assisted development increasingly depends on.