AI-led code development is exposing new AppSec governance gaps

At a glance

What this is: This is an AppSec analysis of four controls needed for AI-led software development, with a strong emphasis on discovery, testing, threat modelling, and the risk created when GenAI use combines with weak code governance.

Why it matters: It matters because identity, access, and governance teams now have to understand where AI tools, MCP use, and AI-generated code alter control boundaries in development pipelines and application risk.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read LEGIT Security's whitepaper on AppSec requirements for the AI-led software development era

Context

AI-generated code is changing application security because the control surface now includes models, prompts, AI services, and developer tools that can introduce risk before code ever reaches runtime. The first governance failure is usually visibility: teams cannot secure what they cannot identify, especially when AI use is embedded inside normal developer workflows and mixed with standard build and review processes.

The identity angle is real even in an AppSec post. MCP, AI services, and code assistants create new trust relationships, while secrets, tokens, and automation credentials can be exposed through both human misuse and machine-generated output. That makes AppSec, IAM, and NHI governance converge around lifecycle control, approval boundaries, and change detection rather than isolated point controls.

This is a typical pattern for AI-era software teams: innovation arrives faster than governance, and risk accumulates in the gaps between developer productivity tools and security enforcement.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: Why do AI coding tools create a security risk even when code looks correct?

A: They optimise for syntax and pattern completion, not contextual security reasoning. Code can compile, satisfy tests, and still mishandle input, logging, or trust boundaries in ways that create exploitable flaws. That is why functional correctness and security correctness must be measured separately in governed pipelines.

Q: What breaks when branch protection and code review are missing in GenAI workflows?

A: The release process stops acting as a control boundary. GenAI-assisted changes can move faster than human review, allowing insecure logic, unsafe dependencies, or exposed secrets to enter the codebase without the checks that normally catch them. That is why the combination is more dangerous than either issue alone.

Q: Who is accountable when AI components are added without threat modelling?

A: Accountability should sit with the product owner, engineering lead, and security review function that approved the change. If AI components are introduced without a formal threat model, the organisation has lost the point at which trust boundaries should have been re-evaluated. Governance must assign ownership before deployment, not after incidents.

Technical breakdown

AI discovery and visibility in software pipelines

AI discovery is the process of finding where AI is present in the development environment, including model creation, AI services, MCP connections, and code assistants such as Copilot or Cursor. The important point is that AI use is not always visible from the application layer alone. It can appear in IDE plugins, pipeline steps, service integrations, or model calls embedded in code. Without inventory and classification, teams cannot distinguish sanctioned AI use from shadow AI or unmanaged experimentation.

Practical implication: build an inventory that maps AI tools, MCP usage, and generated-code pathways to the teams and repositories that depend on them.

Why AI-specific security testing differs from classic scanning

Traditional application scanners look for known code and configuration flaws, but AI introduces weakness classes such as training data poisoning, excessive agency, malicious models, and unsafe model behaviour. AI-specific security testing therefore has to evaluate model provenance, output control, and how the application behaves when the model is influenced or misused. The risk is not only malformed code. It is also the possibility that a model or agent behaves in ways the application owner did not anticipate.

Practical implication: add tests that validate model source, prompt handling, and behavioural guardrails before relying on conventional AppSec tooling alone.

Threat modelling when an app gains AI interfaces or agents

Threat modelling must change when an application starts exposing AI interfaces, running agents, or processing user input through a model. Those changes alter trust boundaries, decision paths, and abuse cases. A chatbot, for example, may create data exposure or privilege issues long before anyone notices a classic vulnerability. The key control challenge is timing: if security teams only review after deployment, the model has already shaped behaviour and data flow inside the application.

Practical implication: require threat modelling at the moment AI components are introduced, not after they are already operating in production.

Threat narrative

Attacker objective: The attacker objective is to exploit weak AI-era development governance to push unsafe code or extract sensitive secrets through the software supply chain.

1. Entry occurs when developers introduce GenAI tools, AI services, or MCP-integrated components into a repository or pipeline without consistent governance.
2. Escalation happens when those AI-enabled workflows combine with missing branch protection, missing code review, or weak static analysis, creating a toxic combination.
3. Impact is the release of insecure code, unreviewed AI behaviour, or exposed secrets into production paths that are harder to detect and remediate.

NHI Mgmt Group analysis

AI discovery is now an identity and governance problem, not just an AppSec inventory problem. When teams cannot see where AI services, MCP connections, and code assistants are used, they cannot assign accountability or apply lifecycle controls. The governance failure is not the presence of AI, but the absence of a reliable map of who or what is allowed to invoke it. Practitioners should treat AI discovery as a control boundary issue across IAM, NHI, and application change management.

Toxic combinations are the right concept for AI-led software risk because the danger is multiplicative. GenAI use becomes materially more dangerous when paired with missing branch protection, weak review, or absent static analysis. That means the control question is not whether AI is used, but whether AI use is operating inside a governed release path. Security teams should measure the combined exposure of tooling, repository controls, and deployment discipline.

Threat modelling must move from periodic review to change-triggered review. The article’s core operational insight is that the introduction of an AI component changes the attack surface immediately. If teams wait until the application is mature, they are already governing a system whose trust assumptions have shifted. Practitioners should make AI introduction a mandatory trigger for architecture, AppSec, and identity review.

Shadow AI and unmanaged MCP usage are the emerging blind spot in software governance. AI components can enter a development environment through plugins, services, or embedded workflows without the visibility that traditional software supply chain controls expect. That makes the boundary between sanctioned automation and unmanaged capability harder to enforce. Security leaders should align AppSec telemetry with identity governance so they can detect, approve, and revoke AI-related access paths.

AppSec teams need a shared control model with IAM and NHI teams before AI code generation scales further. AI-generated code often depends on secrets, service accounts, and machine-to-machine trust that sit outside classic developer workflow controls. The result is a governance gap between code provenance and runtime identity. Practitioners should unify approval, secret handling, and privileged access policy around AI-enabled development.

What this signals

AI-led software development will increasingly force security teams to align AppSec telemetry with identity and access governance. The practical issue is not only code quality, but also who or what can invoke AI services, manage secrets, and push changes through release pipelines. Teams that cannot connect those control layers will struggle to separate sanctioned automation from unmanaged experimentation.

The most useful operational concept here is toxic combination management: security leaders should track combinations of AI use, weak repository controls, and missing review gates as a single risk signal. That approach is more actionable than treating AI adoption as a generic productivity trend. It also maps cleanly to NIST Cybersecurity Framework 2.0 and NIST SP 800-53 Rev 5 Security and Privacy Controls when organisations need a common control language.

For practitioners

• Map AI use across development environments Inventory every repository, pipeline, IDE plugin, model integration, and MCP connection that touches code generation or analysis. Classify each by owner, approved use case, and access path so shadow AI does not remain hidden inside normal developer workflows.
• Add AI-specific security checks before release Extend AppSec testing to cover model provenance, prompt handling, malicious model detection, and behaviour under manipulated inputs. Use these checks alongside conventional scanners so AI-specific weaknesses do not pass through as ordinary code quality issues.
• Trigger threat modelling on AI introduction Require a formal threat model whenever a team introduces an AI component, an agent, or a user-facing model interface. Tie that review to architecture approval and release gating so risk is assessed before the capability reaches production users.
• Treat toxic combinations as a priority control category Flag repositories where GenAI-assisted development coincides with missing branch protection, absent code review, or weak static analysis. Prioritise those pathways first because the combined control failure creates a much larger exposure than any single issue alone.

Key takeaways

• AI in development becomes an AppSec issue when discovery, testing, and governance do not keep pace with how teams actually build software.
• The real risk signal is the toxic combination of GenAI use with weak code controls, not AI use in isolation.
• Security teams need change-triggered review, AI-specific testing, and identity-aware governance before AI-generated code scales further.

Key terms

• AI Discovery: AI discovery is the process of automatically finding AI tools, embedded features, agents, and integrations operating in an environment. It provides the first visibility layer for governance, but it does not by itself explain ownership, permissions, or risk.
• Toxic Access Combination: A toxic access combination is a set of permissions that becomes dangerous when granted together, even if each entitlement looks acceptable on its own. In identity governance, these combinations matter because they can enable misuse, separation-of-duties failures, or broader compromise.
• AI-Specific Security Testing: AI-specific security testing evaluates how models, agents, and AI-enabled workflows behave under malicious input, weak provenance, or unsafe delegation. It goes beyond traditional scanners by checking model source, behavioural controls, and failure modes unique to AI systems.

What's in the full article

LEGIT Security's full whitepaper covers the operational detail this post intentionally leaves for the source:

• Practical guidance on identifying AI-generated code and AI services across development environments
• The article’s breakdown of AI-specific security testing gaps, including model poisoning and excessive agency
• Examples of toxic combinations such as GenAI use paired with missing branch protection or code review

👉 The full LEGIT Security whitepaper covers discovery guidance, testing priorities, and toxic-combination examples in more operational detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, IAM, secrets management, and agentic AI identity. It is designed for practitioners who need to connect identity controls to modern software and automation risks.