TL;DR: AI-generated code can ship with vulnerabilities at high rates, and user trust in model-produced code increases the chance that insecure output bypasses review, according to Arxan Technologies. The real issue is governance drift: code generation, dependency selection, and deployment controls now need to be treated as one security chain, not separate tasks.
At a glance
What this is: This article argues that AI-generated code creates a security blind spot because insecure output, risky dependencies, and automation bias can all bypass normal development controls.
Why it matters: For IAM, PAM, and NHI practitioners, the lesson is that code pipelines increasingly depend on governed non-human access, policy enforcement, and traceable change control across the software lifecycle.
By the numbers:
- Out of 130 code samples generated using InCoder and Github Copilot, 68% and 73% of the code samples contained vulnerabilities when checked manually.
- ChatGPT was used to generate 21 programs in five different programming languages, and only five out of 21 were initially secure.
- A 2023 industry survey of 537 technology and IT workers and managers found that 76% responded that AI code is more secure than human produced code.
- An average of 48% of the code produced by five different LLMs contains at least one bug that could potentially lead to malicious exploitation.
👉 Read Arxan Technologies' analysis of securing AI-generated code in release pipelines
Context
AI-generated code is not automatically unsafe, but it is now common enough that organisations must assume code quality and security can drift faster than manual review keeps up. The problem is not only the model output itself, but also the surrounding pipeline where dependency suggestions, policy checks, and deployment approvals can fail to catch insecure patterns. In environments where AI systems can write or propose code, the security boundary shifts to the controls that govern what gets accepted, tested, and released.
That creates a genuine identity and access issue as well. The pipeline is operated by humans and non-human identities, including build services, scanners, policy engines, and deployment automation, each of which needs scoped permissions and auditability. Where those privileges are too broad, the organisation does not just inherit code risk, it inherits governance risk. In this respect, the article describes a typical enterprise failure mode rather than an edge case.
Key questions
Q: How should security teams govern AI-generated code in production pipelines?
A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.
Q: What do security teams get wrong about AI-generated code risk?
A: They often focus on catching insecure output after code is written, which is too late for AI-native workflows. The more important control point is the moment the agent is allowed to initiate the action. If that step is not governed, testing becomes a detection layer rather than a prevention layer.
Q: What do security and engineering teams get wrong about AI-assisted development?
A: They often confuse faster output with better control. AI assistants can generate code quickly, but they do not automatically preserve architecture, reuse patterns, or the source of truth. If teams only measure throughput, they may miss the fact that they are borrowing speed from the future and creating systems that are harder to govern.
Q: How can organisations prove where insecure AI-generated code came from?
A: By maintaining chain of custody for the full software lifecycle. Teams need records of prompts, outputs, dependency selections, policy decisions, and deployment actions so they can reconstruct how a flaw entered the pipeline. Without provenance, remediation becomes guesswork and accountability weakens.
Technical breakdown
Why AI-generated code bypasses normal review assumptions
AI-generated code creates risk because reviewers often treat machine-produced output as inherently disciplined, even when it is syntactically valid but semantically unsafe. The article shows that models can produce insecure patterns, vulnerable logic, and risky dependencies at scale. This is amplified by automation bias, where users assume the output is more trustworthy than human-written code and reduce scrutiny. The problem is not only bad code generation, but weak challenge processes around acceptance, review, and test coverage.
Practical implication: treat AI-generated code as untrusted input until it passes the same review and test gates as any external contribution.
How prompt injection and data poisoning affect code generation
Code generation systems can be manipulated through data poisoning, backdoors, and indirect prompt injection. Data poisoning changes what the model learns from compromised repositories. Backdoors insert trigger-based behaviour that can persist even after attempts to clean the model. Indirect prompt injection is more subtle: the model retrieves compromised content from an external source and follows hidden instructions embedded there. In code workflows, that can translate into insecure package suggestions, hidden malicious dependencies, or unsafe implementation patterns.
Practical implication: constrain retrieval sources, validate imported context, and test for malicious influence in both training and inference paths.
Why RBAC and chain of custody matter in AI code pipelines
The article ties secure code generation to governance controls rather than model performance alone. RBAC limits who can generate, review, approve, and deploy AI-produced code, while software chain of custody preserves the history of prompts, modifications, policy checks, and release events. That combination matters because post-incident investigation depends on knowing which non-human identity, policy, or dependency introduced the weakness. Without traceability, remediation becomes guesswork instead of evidence-led response.
Practical implication: map build and deployment permissions to least privilege, and require provenance records for every AI-assisted change.
NHI Mgmt Group analysis
AI-generated code is becoming a governance problem before it is a code-quality problem. The article is about insecure output, but the deeper issue is that organisations are now accepting machine-generated changes through pipelines built for human authorship. That weakens review discipline, provenance, and approval accountability. Security teams should treat AI-assisted development as a controlled identity and policy domain, not just a productivity feature.
Automation bias creates a trust gap that conventional SDLC controls do not close. When teams assume model-generated code is safer than human-written code, they lower their scrutiny exactly where the risk is highest. That makes testing, policy enforcement, and release gating more important, not less. The result is a predictable control failure: security assurance is deferred to the toolchain, while the toolchain itself becomes the attack surface.
Non-human identities in CI/CD are now part of the AI code security boundary. Build systems, scanners, policy engines, and deployment automation all act on behalf of the organisation, which means their permissions determine whether insecure code is blocked or released. That intersection matters for IAM and PAM because overprivileged automation can silently bypass guardrails. The right question is not whether the model is safe, but whether the identities around it are constrained well enough to enforce safe outcomes.
Software chain of custody is the named concept this category needs. The article shows why provenance across prompt, dependency, validation, and deployment events is essential when code is machine-assisted. Without it, security teams cannot prove where a flaw entered or which automated control failed. Practitioners should treat provenance as a release control, not a documentation afterthought.
Policy-as-code only works when policy is coupled to release authority. The article’s OPA example is useful because it shows policy enforcement embedded in the delivery path, not bolted on after deployment. That aligns with NIST CSF and NIST SP 800-53 thinking on controlled change, access control, and auditability. The practical implication is simple: if policy cannot stop release, it is only advisory.
What this signals
Software supply chains now depend on identity controls as much as code controls. As AI-assisted development grows, the permissions attached to build agents, scanners, and release automation become part of the security perimeter. That means teams should align pipeline access with NIST Cybersecurity Framework 2.0 and the access-control expectations in NIST SP 800-53 Rev 5, rather than treating DevOps permissions as a separate concern.
Software chain of custody is becoming the practical control for AI-assisted change. Where human review used to provide context, organisations now need durable provenance across prompts, dependencies, policy checks, and release events. If that provenance is weak, the security team loses the ability to explain why a vulnerable change passed control gates. That is a governance failure, not just a development defect.
Secret exposure and code provenance are converging into one operating problem. Once AI-generated code is accepted, the same pipeline often handles credentials, API keys, and deployment tokens that can be reused outside intended scope. The consequence is broader than insecure syntax: it is persistent trust in identities and secrets that have already outlived their control window.
For practitioners
- Enforce review gates for AI-generated code Require every AI-assisted change to pass the same peer review, testing, and approval steps as human-authored code, with no exception for speed or confidence in the model output.
- Restrict pipeline identities with least privilege Limit build, scanning, policy, and release automation to the narrowest permissions needed, and separate generate, approve, and deploy functions across distinct non-human identities.
- Validate model-suggested dependencies before build promotion Block packages and libraries that fail security policy, and run dependency scanning early enough that risky components never reach staged or production environments.
- Track AI-assisted changes through chain of custody Record prompts, outputs, dependency selections, policy decisions, and deployment actions so investigators can trace how an insecure pattern entered the pipeline.
- Use policy-as-code to stop unsafe releases Connect policy checks to release authority so deployments fail closed when code violates access control, encryption, or approved dependency rules.
Key takeaways
- AI-generated code introduces governance risk because insecure output can move through trusted pipelines faster than manual review can contain it.
- The evidence in the article shows that vulnerability rates remain high, which makes policy enforcement and provenance controls operational necessities rather than optional extras.
- Teams should connect code review, release authority, and non-human identity governance so machine-assisted development cannot bypass security decisions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0, NIST SP 800-53 Rev 5, CIS Controls v8 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.AC-4 | Pipeline RBAC and release gating map to controlled access for AI-assisted delivery. |
| NIST SP 800-53 Rev 5 | AC-6 | Least privilege is central to restricting build and release automation. |
| CIS Controls v8 | CIS-5 , Account Management | Account management matters where build and deployment identities must be segregated. |
| MITRE ATT&CK | TA0002 , Execution; TA0006 , Credential Access; TA0011 , Command and Control | The article's attack patterns include malicious code execution and dependency abuse. |
| NIST AI RMF | GOVERN | AI RMF GOVERN fits the policy and accountability model described for AI-assisted code. |
Map AI-assisted code threats to ATT&CK tactics and monitor for execution and credential abuse.
Key terms
- AI-Generated Code Risk: AI-generated code risk is the possibility that model-produced output introduces insecure credential handling, hidden secrets, or repeated patterns that weaken control design. The risk is operational, because the output can be merged quickly and at scale before humans catch the problem.
- Automation Bias: Automation bias is the tendency to trust machine output as objective simply because it is machine-generated. In identity and governance programmes, this becomes a control problem when plausible agent decisions are accepted without questioning the embedded tradeoffs, making drift and misuse harder to detect.
- Software chain of custody: A record of how code moves from creation to modification, validation, approval, and deployment. For AI-assisted development, it also captures prompts, dependency choices, policy checks, and release decisions so investigators can reconstruct where a flaw entered and which control failed.
- Policy as Code: Policy as code stores authorization logic in version control and evaluates it through testable, reviewable rules. For agent governance, it makes runtime decisions reproducible and measurable, which is critical when actions can be triggered by untrusted content and executed at machine speed.
What's in the full article
Arxan Technologies' full article covers the operational detail this post intentionally leaves for the source:
- Integration specifics for SAST, SCA, OPA, ArgoCD, and Argo Rollouts in a secure release pipeline
- The article's step-by-step compliance framework for testing, policy enforcement, traceability, and rollback
- How RBAC is applied across developers, security teams, and administrators in the AI code workflow
- Examples of how chain of custody records prompts, dependencies, and deployment actions for investigation
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and identity lifecycle controls. It helps security practitioners connect non-human access decisions to the broader governance model their programmes depend on.
Published by the NHIMG editorial team on September 3, 2026.
NHI Mgmt Group — the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
