AI-led code development is exposing new AppSec governance gaps

At a glance

What this is: NHI Mgmt Group's analysis, drawing on LEGIT Security's blog post: AI-led software development is expanding AppSec scope from code scanning to discovery, AI-specific testing, threat modeling, and governance of toxic combinations.

Why it matters: AppSec and platform teams now have to control AI use as a change-management and governance problem, not just a coding-pattern problem, because AI tools and generated code can create new risk paths in delivery pipelines.

By the numbers:

• On average, 17% of repos per organization have developers using GenAI tools plus lack of branch protection or code review.

Context

AI-generated code changes the AppSec problem from scanning finished code to governing how code is produced, reviewed, and deployed. That matters because AI tools can introduce unfamiliar weaknesses, while the surrounding delivery process may still assume human-authored code and stable review patterns.

The identity angle is indirect but real: development pipelines increasingly rely on service accounts, tokens, and non-human tools that can introduce change at machine speed. When AI use is undiscovered or ungoverned, AppSec teams lose visibility into who or what is influencing software risk before it reaches production.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: Why do AI-generated code pipelines become risky when branch protection is missing?

A: Branch protection and code review act as the brake on rapid change. When AI speeds up code creation but those controls are absent, insecure or unreviewed changes can reach production with less scrutiny. The risk is not AI alone, but AI combined with weakened release governance.

Q: What are the signs that AppSec governance is lagging behind AI development?

A: The clearest signs are unclear code ownership, delayed vulnerability routing, inconsistent framework approval, and release decisions that rely on manual review after the change is already too large to inspect well. When teams start asking who owns the code only after a finding appears, governance is already behind the delivery model.

Q: Should organisations treat AI-specific testing separately from standard AppSec testing?

A: Yes. Standard scanners still matter, but they do not reliably detect model poisoning, excessive agency, or malicious model behaviour. AI-specific testing should sit alongside existing AppSec checks so teams assess both the code and the AI behaviour embedded in the application and pipeline.

Technical breakdown

AI discovery and governance across the delivery chain

AI discovery is the control that tells you where AI is participating in software creation. That includes generated code, developer copilots, AI services, MCP-connected tools, and AI models embedded in build or development workflows. Without discovery, security teams cannot know which repositories, pipelines, or teams need additional guardrails, so governance becomes reactive instead of policy-driven. The operational issue is not only the presence of AI, but the absence of inventory and ownership for AI-enabled change.

Practical implication: build an inventory of AI tools, models, and AI-generated code paths before you try to enforce policy.

Why traditional AppSec testing misses AI-specific weaknesses

Conventional static and dynamic testing was built for familiar software defects, not model behaviour. AI-specific testing has to account for risks such as training model poisoning, excessive agency, and malicious or low-reputation models that can shape output or execution paths in ways traditional scanners will miss. When an application exposes AI interfaces or uses an agent, the security boundary now includes model interactions, not just source code and endpoints.

Practical implication: add AI-specific test cases and model trust checks to the AppSec workflow rather than relying only on standard scanners.

Toxic combinations are a governance failure, not a tooling quirk

A toxic combination appears when AI-generated code is used in a pipeline that also lacks a core control such as branch protection or code review. The danger is multiplicative: AI accelerates change, while weak review controls remove the brake. That creates a condition where insecure code can move faster and with less scrutiny, especially when development teams treat GenAI adoption as a productivity issue instead of a security control change. This is a governance gap, not just a developer behaviour issue.

Practical implication: tie AI-use discovery to branch protection and review enforcement so risky pipelines are identified as control failures, not exceptions.

Threat narrative

Attacker objective: The practical objective is to get insecure or poorly governed AI-influenced code into production before the organisation notices the added risk.

1. Entry begins when developers introduce AI-generated code, AI services, or MCP-connected tools into the software delivery environment without central discovery.
2. Credentialed delivery paths and weak review controls then allow AI-assisted changes to move through pipelines with insufficient scrutiny, especially where branch protection is missing.
3. Escalation occurs when the AI component or generated code introduces new attack surface, such as model exposure, excessive agency, or unvetted dependencies, into the application.
4. Impact is the release of software that carries latent AppSec weaknesses, governance blind spots, and compounded risk from toxic control combinations.

NHI Mgmt Group analysis

AI discovery is now an AppSec control, not an optional inventory exercise. If teams cannot identify where generated code, AI services, and developer copilots are used, they cannot govern the risk introduced by those tools. The discipline shifts from code inspection after the fact to policy enforcement at the point AI enters the delivery chain. That is the new baseline for AppSec programme design.

Toxic combinations are the most useful way to think about AI-led development risk. The article’s 17% figure shows that the issue is not AI in isolation but AI plus missing controls, especially branch protection and code review. That pattern should be treated as a governance defect in the software supply chain, not merely a development preference. Practitioners should prioritise control pairing, because one weak safeguard can magnify the risk of AI-assisted change.

AI-specific testing needs to sit alongside, not underneath, traditional AppSec. Traditional scanners remain necessary, but they do not see model poisoning, excessive agency, or untrusted model provenance with enough fidelity. Security teams therefore need separate test logic for AI behaviour and AI trustworthiness. For practitioners, the message is clear: modern AppSec now includes model-aware assurance as well as source-code assurance.

AI governance debt: the accumulated gap between rapid AI adoption and the controls needed to inventory, test, and approve AI-driven change. This article shows how quickly that debt becomes operational when development teams can add AI into pipelines faster than governance can track it. The longer the gap remains open, the more likely it is that AI usage will spread through repositories and builds without adequate review. Practitioners should measure and reduce that debt as a formal security objective.

AppSec is moving toward change-governance of the software factory itself. The important question is no longer only whether code is secure, but whether the process that produced it was visible, reviewed, and constrained. That is where AI-generated code changes the programme model. Teams that keep treating AI as a coding feature will miss the governance layer that now determines real delivery risk.

From our research library:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: Top 10 NHI Issues

What this signals

AI governance debt: the gap between rapid AI adoption in development and the controls needed to inventory, approve, and test that change. AppSec teams should treat this as a programme metric, because the longer it persists the more AI use will spread through the SDLC without consistent oversight.

The practical signal for practitioners is that discovery must lead governance. Once AI-generated code, copilots, and model-backed services are visible, teams can align branch protection, code review, and AI-specific testing to the same risk register instead of managing each control in isolation.

For practitioners

• Discover AI use across repositories and pipelines Inventory AI-generated code, AI services, MCP-connected tools, and developer copilots so you can map where AI influences software change.
• Add AI-specific security tests to AppSec gates Extend testing beyond traditional scanning to look for model poisoning, excessive agency, and untrusted or low-reputation models in the delivery chain.
• Treat branch protection as a control paired with AI use Flag any pipeline where GenAI is used but branch protection or code review is missing, because that combination creates an outsized governance gap.
• Trigger threat modeling when AI enters an application path Require review before deployment whenever a team introduces an AI interface, an agent, or a user-facing model workflow into an existing application.
• Track toxic combinations as a measurable risk class Report on development pipelines where AI-assisted code creation overlaps with missing review, missing analysis, or other security control gaps.

Key takeaways

• AI-led development is turning AppSec into a governance discipline that must discover AI use, not just inspect code.
• The most concerning pattern is AI-assisted change combined with missing review controls, because that pairing accelerates risk into production.
• Security teams need AI-specific testing and threat modelling for applications that now expose AI interfaces or generated behaviour.

Key terms

• API Governance Debt: API governance debt is the accumulation of weak, inconsistent, or missing controls around how APIs are designed, approved, documented, secured, and retired. It creates hidden risk over time. Technically, it includes gaps in authentication, authorization, versioning, inventory, ownership, logging, and policy enforcement that make APIs harder to trust and govern.
• Toxic Risk Combinations: Toxic risk combinations are unsafe interactions between datasets, access permissions, and AI workflows that only become problematic when combined. Individually they may appear harmless, but together they can expose sensitive information, enable re-identification, or create unintended inferences that traditional controls may miss.
• AI-Specific Security Testing: AI-specific security testing evaluates how models, agents, and AI-enabled workflows behave under malicious input, weak provenance, or unsafe delegation. It goes beyond traditional scanners by checking model source, behavioural controls, and failure modes unique to AI systems.
• AI Discovery: AI discovery is the process of automatically finding AI tools, embedded features, agents, and integrations operating in an environment. It provides the first visibility layer for governance, but it does not by itself explain ownership, permissions, or risk.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. For practitioners dealing with machine identities, AI-driven change, and governance gaps, the course gives a useful baseline for stronger control design.