AI-generated code is widening the AppSec confidence gap

At a glance

What this is: AI-assisted development is increasing code throughput and making insecure code easier to trust because it often appears functional before it is security-reviewed.

Why it matters: For IAM and security practitioners, the key issue is that faster code delivery expands the attack surface, increases hidden dependencies, and creates governance gaps around who or what is introducing risk into the software supply chain.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read Checkmarx's analysis of AI-generated code risk in AppSec

Context

AI-generated code is changing application security because software delivery is moving faster than many validation processes were designed to handle. The problem is not that generated code always fails, but that functional code can still be insecure, especially when developers trust output that looks polished and compiles cleanly. In practice, AI-generated code creates a governance challenge for AppSec, IAM, and software supply chain teams because it can introduce new dependencies, new integrations, and new trust assumptions without a matching security review.

This matters to identity and access programmes because modern development environments are now full of machine-driven actors and machine-created artefacts. Model integrations, MCP connections, automation agents, and AI-assisted workflows all create new paths for secret exposure, authorisation drift, and unmanaged access to code and pipelines. When teams cannot see which tools are in use or how data flows through them, they cannot govern the resulting identity surface effectively.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: Why do AI-assisted workflows create hidden application security risk?

A: AI-assisted workflows create hidden risk because they expand the number of systems, dependencies, and trust relationships involved in producing code. Even when the output works, the surrounding tooling can introduce secret exposure, data flow ambiguity, and permission drift. Security teams need visibility into the full AI-assisted delivery chain, not just the final commit.

Q: What do AppSec teams get wrong about code that compiles successfully?

A: Teams often confuse functional correctness with security assurance. Code can compile, pass tests, and still contain insecure logic, weak authorisation checks, or unsafe secret handling. The practical mistake is reviewing AI-generated output for speed and polish while assuming the model has already accounted for threat modelling, which it has not.

Q: How should organisations respond when backlog growth outpaces AppSec review capacity?

A: They should automate triage, prioritise findings by exposure and exploitability, and push security controls earlier into the delivery workflow. If backlog growth is sustained, it is a sign that governance is lagging behind the development model. Leaders should measure whether findings are being resolved before code reaches production, not just whether they are being logged.

Technical breakdown

Why AI-generated code creates a false sense of security

AI coding systems are optimised for plausibility, not exploit resistance. They predict patterns that are likely to work based on training data, which means they can produce code that runs correctly while still violating secure design principles, secret-handling rules, or environment-specific threat assumptions. The security problem is compounded when developers review output primarily for syntax, speed, and feature fit. In that situation, functional correctness becomes a proxy for trust, even though no model understands the application threat model or runtime abuse paths.

Practical implication: AppSec teams need security validation that tests exploitability, not just compilation or test pass rates.

How AI-assisted delivery expands the software supply chain

AI-assisted development is not just faster coding. It adds new components to the software supply chain, including model integrations, prompt-driven workflows, and external context connections such as MCP-linked tools and data sources. Each of those elements can change data exposure, dependency trust, and approval boundaries. That means the attack surface is no longer limited to source code and package dependencies. It also includes the systems that help generate code, enrich it, and move it through the pipeline.

Practical implication: security inventories must include AI tools, model connections, and workflow integrations alongside traditional application dependencies.

Why backlog growth becomes a governance failure, not just an operations problem

When AI increases code volume faster than review capacity, the issue shifts from operational strain to control failure. Manual identify, triage, remediate, and verify cycles do not scale when pull requests multiply and more contributors rely on generated output. Traditional scanners were built for a slower development model, so they often produce noise rather than prioritised security decisions. The result is not simply more alerts. It is delayed assurance, missed exposure windows, and weaker decision-making about what should be merged, blocked, or remediated first.

Practical implication: teams should automate triage and remediation prioritisation before backlog growth turns into permanent control debt.

Threat narrative

Attacker objective: The attacker aims to exploit AI-amplified development trust to introduce or activate weaknesses that later enable compromise of applications, data, or connected systems.

1. Entry begins when AI-generated code introduces insecure logic, hidden dependencies, or new model-linked integrations into the development pipeline.
2. Escalation occurs when developers trust functional output, merge it quickly, and propagate the flaw into production services or automation paths.
3. Impact follows when the insecure code or exposed integration enables broader compromise, data exposure, or unauthorised access through the application layer.

NHI Mgmt Group analysis

AI-generated code has created a confidence gap, not just a speed problem. The core risk is that output quality makes code feel trustworthy before security validation has happened. That shifts the burden onto AppSec teams to detect exploitability at machine speed, not merely react after merge. In identity terms, this is also a governance issue because AI-assisted workflows are now part of the trust chain that determines what enters production.

Model integrations and MCP connections are becoming part of the application trust boundary. Once code generation depends on linked tools, external context, and AI-assisted workflows, security teams must govern not only the code but the identity and permission model around the systems producing it. That makes AI governance, workload identity, and secrets management inseparable from application security. The practical conclusion is that AI supply chain inventory is now a control requirement, not an optional nice-to-have.

Agentic scanning reflects a broader shift from manual inspection to control automation. The industry is moving toward hybrid analysis, automated triage, and context-aware remediation because human review alone cannot keep pace with modern delivery velocity. This does not remove the need for engineering judgement, but it does change where that judgement is applied. Security teams should treat automation as the mechanism that preserves assurance when volume outstrips review capacity.

AppSec backlog is now a measurable indicator of governance drift. When remediation slows while code volume accelerates, the organisation is effectively accepting more unreviewed risk into production. That is especially relevant where AI-generated code, secrets exposure, and machine-driven workflow integration overlap. Practitioners should read backlog growth as a signal that controls are lagging the delivery model, not simply that teams need to work harder.

What this signals

AI-assisted development is turning application security into a higher-velocity governance problem, not just a tooling problem. The more code is generated or scaffolded by AI, the more programme owners need reliable visibility into model usage, external integrations, and the permissions granted to the systems that produce code. That is where identity governance starts to overlap directly with AppSec, because machine-driven workflows now need ownership, scope, and review just like any other privileged system.

AI trust debt: when developers merge generated code because it looks correct, organisations accumulate unreviewed risk that only becomes visible after deployment. The practical response is to move from reactive backlog management to policy-driven assurance in the pipeline, using automation to flag secrets, risky integrations, and authorisation weaknesses before they reach production.

For practitioners

• Inventory AI-assisted development paths Map where AI code generation, agentic workflows, model integrations, and MCP connections are already in use across delivery teams. Include shadow usage, because unmanaged tool adoption creates blind spots in the application trust boundary.
• Shift validation from syntax to exploitability Add security checks that evaluate whether generated code can be abused in the target environment, not only whether it compiles or passes tests. Prioritise auth flows, secret handling, and external integration points where AI output tends to look correct but still fail securely.
• Automate triage for high-volume findings Use deterministic analysis and context-aware prioritisation to reduce noise in large AppSec queues. The goal is to route clear fixes quickly, escalate genuine exposure patterns, and stop backlog growth from becoming accepted risk debt.
• Govern the identity of AI systems in the pipeline Assign ownership for models, assistants, and workflow agents that can generate or modify code. Tie their permissions to least privilege, track their access to repositories and secrets, and review them as part of the software supply chain rather than treating them as invisible tooling.

Key takeaways

• AI-generated code is widening the gap between what appears to work and what is actually secure.
• Code velocity is now a governance issue because review capacity, not developer intent, determines how much risk reaches production.
• Teams need identity-aware visibility into AI tools, model integrations, and pipeline permissions before generated output becomes trusted by default.

Key terms

• AI-Generated Code Risk: AI-generated code risk is the possibility that model-produced output introduces insecure credential handling, hidden secrets, or repeated patterns that weaken control design. The risk is operational, because the output can be merged quickly and at scale before humans catch the problem.
• AI-BOM: An AI bill of materials is a structured inventory of the components that define an AI agent, including the model, prompt, tools, retrieval sources, and dependencies. In practice, it is the evidence base for review, change control, and risk assessment when the agent evolves after deployment.
• Agentic scanning: A security testing approach that blends deterministic analysis with AI-assisted reasoning and automated triage. It is designed for modern development environments where code volume, complexity, and delivery speed outpace purely manual review.
• Software Supply Chain: A software supply chain is the set of tools, identities, dependencies, and processes that turn source code into deployed software. Because it relies on automation and privileged machine identities, it becomes a governance problem when access, signing, and deployment controls are too broad.

What's in the full article

Checkmarx's full article covers the operational detail this post intentionally leaves for the source:

• How the vendor frames agentic scanning for high-volume AppSec triage and remediation.
• The specific role of AI-BOMs in tracking models, connections, and data flows across development environments.
• How Checkmarx positions deterministic analysis alongside AI-driven detection inside the delivery pipeline.
• The vendor's description of Developer Assist and how it fits into workflow automation.

👉 Checkmarx's full post covers the AI-BOM angle, agentic scanning model, and development workflow implications.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It helps practitioners connect identity controls to the broader security programmes that now depend on machine-driven workflows.