Human judgment is becoming the missing control in AI code

At a glance

What this is: This is a CyberArk analysis of how AI-generated code weakens the transparency and accountability markers that once supported safe review, with human judgment and machine identity emerging as the missing controls.

Why it matters: IAM, PAM, and NHI teams need to treat AI-assisted development as an identity and provenance problem, because code generation now depends on machine identities, review discipline, and traceable authority.

By the numbers:

• 92% of leaders have concerns about AI-generated code, according to CyberArk research.
• 78% believe AI-developed code will lead to a security reckoning, according to CyberArk research.

Context

AI-generated code creates a governance gap because the usual trust markers in software development, commit history, authorship, review trail, and dependency lineage, become less reliable when code is assembled by models and assistants rather than written line by line by people. That matters to identity security because provenance is itself a control surface, not just a process detail.

CyberArk's argument is that the review model has to change from informal trust in output quality to explicit judgment, traceability, and identity-bound accountability. In practical terms, AI code development starts to look less like traditional software craftsmanship and more like governed non-human activity.

The core issue is not speed alone. It is that teams can no longer assume the actor behind a commit is a person with stable intent and a durable audit trail, which means identity governance has to follow the code path, not just the developer log-in.

Key questions

Q: What is the failure mode when AI-generated code is reviewed only for syntax and not for provenance?

A: Teams lose the ability to explain who or what introduced a risky dependency, secret, or logic change. Syntax checks can pass while the real failure is missing lineage, so the code reaches production without a trustworthy decision trail. The control gap is not code correctness alone, but traceable authorisation for the change.

Q: Why do AI coding tools create a security risk even when code looks correct?

A: They optimise for syntax and pattern completion, not contextual security reasoning. Code can compile, satisfy tests, and still mishandle input, logging, or trust boundaries in ways that create exploitable flaws. That is why functional correctness and security correctness must be measured separately in governed pipelines.

Q: What are the signs that AI-generated code is becoming too complex to trust?

A: The main warning signs are rising cognitive complexity, excessive branching, and code that expands far beyond the task’s scope. In practice, those patterns increase the likelihood of hard to spot defects, especially in concurrency and state handling. Security and engineering teams should flag outputs that become architecturally elaborate when a simpler implementation would satisfy the requirement.

Q: What should organisations do when AI tools increase code volume faster than review capacity?

A: They should assume manual review alone will miss defects and move enforcement into automated controls. That means scanning AI-generated and human-written code with the same standards, using quality gates, and requiring traceability for every change. The goal is to make review scalable enough that speed does not erase control.

Technical breakdown

Why AI-generated code breaks provenance assumptions

Traditional software review assumes the source of a change can be traced to a named contributor, a known dependency, and a review path that leaves durable evidence. AI-generated code disrupts that model because the model boundary obscures authorship, prompt intent, and the interaction flow that produced the output. The result is not simply faster code creation but weaker lineage, which makes it harder to distinguish deliberate design from model output, hidden dependency introduction, or accidental logic drift. In identity terms, provenance is no longer guaranteed by human authorship alone. Practical implication: treat code lineage as an enforceable control, not an informal by-product of development.

Practical implication: require traceable authorship and machine-verifiable provenance for AI-assisted commits.

Why judgment-in-the-loop is more than a code review slogan

Judgment-in-the-loop means a skilled human applies deliberate scrutiny before AI-generated code reaches production, but the real control is not the presence of any reviewer. It is the quality of the reviewer, the scope of their authority, and their ability to question whether the code should exist at all, not just whether it compiles. That matters because AI can produce code that is syntactically valid, operationally plausible, and still unsafe. The article's examples show that hallucinated dependencies, literal execution, and silent deletion of safeguards can all pass superficial checks. Practical implication: align review authority with decision quality, not with approval volume.

Practical implication: require a meaningful human decision, not a token approval, before AI code is merged.

Why machine identities now sit inside the software supply chain

The article links trustworthy AI-assisted development to verified machine identities for workloads such as microservices, containers, and AI agents. That is because trust in the shipped code increasingly depends on cryptographic proof of what executed, what signed, and what authored the build path. Once prompts, models, and interaction flows influence the output, the supply chain extends beyond packages and versions into non-human execution contexts that need their own identity controls. This is where code provenance becomes an NHI problem, not just a secure coding problem. Practical implication: bind build and deployment steps to workload identity and signing rather than human assumption.

Practical implication: bind build and deploy steps to workload identity and code signing.

Threat narrative

Attacker objective: The objective is to push untrusted or poorly understood code into production with enough plausibility to evade review and create lasting integrity and accountability failures.

1. Entry occurs when AI-generated code is introduced into the development workflow through prompts, assistants, or low-code platforms that produce code faster than humans can scrutinise it.
2. Credential or control abuse appears when the generated code reuses embedded secrets, over-trusts an API key, or imports dependencies without a human understanding of the trust boundary.
3. Escalation follows when the code is deployed without sufficient judgment, allowing unsafe logic, hidden dependencies, or removed safeguards to operate in production.
4. Impact is the loss of traceability, integrity, or recovery confidence when teams can no longer explain who or what caused the change or how the failure propagated.

Breaches seen in the wild

• EmeraldWhale Git config credential theft: Tokens in exposed .git/config files let EMERALDWHALE clone private repositories and steal more than 15,000 cloud credentials.
• Twitter source code leak 2023: Twitter source code and internal tools were posted to GitHub by "FreeSpeechEnthusiast" and stayed public for months before a 2023 takedown.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI-generated code turns provenance into an identity control problem: The article is right to treat authorship, intent, and traceability as part of the control plane, not the documentation layer. Once code is assembled by tools that do not leave human-readable decision trails, the governance question changes from who wrote it to what identity and signing evidence can prove it belongs in production. The practical conclusion is that provenance must be enforced as a control, not assumed as a by-product of development.

Judgment-in-the-loop is a governance pattern, not a sentiment: The missing control is not simply human review, but review by the right human with authority to challenge whether the code should exist. That matters because AI-generated code can look reasonable while still embedding insecure dependencies, unsafe logic, or deleted safeguards. The practitioner takeaway is that review gates need decision quality, not just throughput.

Verified machine identities belong in the software supply chain: Workloads, containers, and AI agents increasingly participate in code generation, build, and deployment as non-human actors. The trust model therefore has to extend beyond developer log-ins to signed, attributable machine identities that can prove which runtime created or moved the code. Teams that do not govern those identities are treating non-human execution as if it were still a human-only process.

Identity-first development is the new provenance boundary: The article points to a broader shift where code security, workload identity, and software supply chain assurance converge. This is not just about preventing bad code from landing in production. It is about ensuring every non-human actor in the delivery path is attributable, bounded, and reviewable before its output becomes system state. Practitioners should treat AI-assisted delivery as an identity lifecycle problem as much as a coding problem.

From our research library:

• Internal repositories are 6x more likely to contain secrets than public ones (32.2% vs 5.6%), contradicting the assumption that private repos are safe, according to the State of Secrets Sprawl 2026.
• Read next: Guide to the Secret Sprawl Challenge

What this signals

Provenance debt: AI-assisted delivery creates a gap between the code that ships and the evidence needed to explain why it exists. Teams should expect this gap to surface first in dependency risk, secret exposure, and incident reconstruction, not in abstract policy debates.

Identity governance now has to extend into the delivery pipeline. If a build can be produced, modified, or deployed by non-human actors, then workload identity, signing, and review authority become part of the same control surface rather than separate concerns.

For practitioners

• Enforce judgment-in-the-loop review Require a named human reviewer with authority to reject AI-generated changes that lack clear rationale, traceability, or safe dependency selection.
• Bind AI-assisted builds to machine identity Use workload identity and code signing so AI-assisted build, test, and deployment steps are attributable and verifiable end to end.
• Track prompts and interaction flows Record prompts, model outputs, and handoff points so teams can reconstruct how a change was produced when debugging or investigating.
• Treat embedded secrets as deployment blockers Scan generated code and pipelines for hardcoded credentials, plain-text API keys, and secret reuse before any merge or release.
• Preserve evidence during cleanup Prevent automated code cleanup from removing error-handling, logging, or diagnostic paths that are needed for later investigation.

Key takeaways

• AI-generated code weakens the provenance signals that made open-source review and enterprise change control trustworthy.
• The practical risk is not only bad code, but also the loss of evidence needed to explain who or what approved it.
• Judgment-in-the-loop and verified machine identities are the controls that turn AI-assisted delivery back into governed change.

Key terms

• Judgment-in-the-loop: A human review control that requires someone with the right context to decide whether AI-generated output should move forward. It goes beyond checking correctness and asks whether the change makes sense, is safe to ship, and can be attributed to a verified decision-maker.
• Code provenance: Code provenance is the verifiable history of where code came from and who or what created it. In security practice, it combines authorship, timestamps, signatures, and build lineage so teams can prove a change was trusted before it reached production.
• Machine Identity: The digital identity of a machine, device, or workload, such as a server, container, or VM, used to authenticate it within a network. Sometimes used interchangeably with NHI, though NHI is the broader category.
• Software Bill of Materials: A software bill of materials is an inventory of the components and dependencies used in an application. It helps teams identify what they shipped, but it becomes most useful when paired with source verification, signature checks, and policy enforcement for third-party code.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.