Human judgment is becoming the missing control in AI code

At a glance

What this is: CyberArk’s article argues that AI-generated code is creating an accountability gap, and that human judgment plus identity-first controls are needed to preserve provenance and traceability.

Why it matters: IAM and NHI practitioners should treat AI-assisted development as an identity and audit problem because code provenance, workload identity, and approval authority now intersect.

👉 Read CyberArk's analysis of human judgment in AI-driven development

Context

AI-assisted development changes the trust model for software delivery because code can now be generated, modified, and deployed with less visible human authorship. That creates an NHI governance problem as much as a development problem, since agents, build systems, and service accounts can all influence what reaches production without a clear accountability trail.

The practical issue is not just code quality. It is whether organisations can prove who, or what, authorised a change, which dependencies were introduced, and whether the surrounding workload identities were constrained enough to make the change safe. That is a typical starting point for teams that already understand the risk of opaque automation, but have not yet mapped it to software supply-chain identity controls.

For readers building policy around AI coding and agentic workflows, the relevant lens is lifecycle governance. The NHI Lifecycle Management Guide is useful here because the same questions that apply to service accounts, rotation, and offboarding also apply to AI-assisted build pipelines and the identities that act within them.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: Why do AI coding tools increase governance risk for IAM and NHI teams?

A: AI coding tools increase governance risk because they obscure who created the logic, which identities executed it, and whether the resulting automation has the right access scope. That creates blind spots in auditability, approval authority, and secret handling. IAM and NHI teams need controls that can prove both the actor and the action.

Q: What is the difference between code review and judgment-in-the-loop?

A: Code review checks whether software looks acceptable. Judgment-in-the-loop checks whether the software should exist, whether the AI-generated logic makes sense, and whether the change can be safely attributed and governed. In practice, judgment-in-the-loop is a security control for intent and accountability, not just a quality gate.

Q: When do AI-generated changes become a workload identity problem?

A: They become a workload identity problem when build systems, agents, or deployment pipelines can act with broad or shared credentials. At that point, the main question is no longer only what the code does, but which principal created, approved, or executed it. Verified workload identity becomes essential for traceability and containment.

Technical breakdown

Why AI-generated code breaks provenance and traceability

Traditional open-source and human-authored code leaves a trail: commits, contributors, timestamps, review comments, and dependency changes. AI-generated code often compresses or obscures that trail, especially when low-code and prompt-driven tools assemble logic from opaque model behaviour. The result is a weaker provenance chain, meaning teams cannot easily answer who introduced a dependency, why a control was bypassed, or whether a change was reviewed by a competent human. In identity terms, the authoring principal becomes ambiguous, and that ambiguity weakens auditability.

Practical implication: Treat AI-assisted commits as identity-bearing events and require traceable approval before code reaches production.

What judgment-in-the-loop changes in AI development

Judgment-in-the-loop is not just another review step. It is a deliberate control that forces a human to evaluate whether generated logic makes sense, whether it should exist, and whether the system can explain who or what approved it. This matters because AI can produce code that passes tests while still embedding flawed assumptions, unsafe dependencies, or destructive automation paths. The risk is highest when human review becomes ceremonial. Effective judgment means reviewing intent, failure modes, and access scope, not just syntax.

Practical implication: Build mandatory human approval into any pipeline that can create, modify, or deploy code with production access.

Why workload identity belongs in the software supply chain

The article’s strongest security point is that trust in AI-generated code cannot rest on appearance alone. It has to be anchored in cryptographic proof and workload identity for the systems that build, test, and deploy software. That includes microservices, containers, and AI agents that can execute actions or call tools. When those identities are unmanaged, code provenance becomes fragile because the pipeline itself can no longer prove which actor performed a change or whether an action came from the intended automation path.

Practical implication: Bind build and deployment actions to verified workload identities and deny anonymous or shared execution paths.

Threat narrative

Attacker objective: The objective is to weaponise trusted development automation so that unsafe code or agent actions reach production with minimal attribution.

1. Entry occurs when AI-generated logic or an AI agent introduces code through a trusted development workflow with insufficient human scrutiny.
2. Escalation happens when the generated code reuses exposed secrets, weak dependencies, or overly broad service account access to extend its effect.
3. Impact follows when the code deploys into production, erases safeguards, or automates destructive actions faster than teams can trace the source.

Breaches seen in the wild

• Moltbook AI agent keys breach — Moltbook breach exposed 1.5M AI agent keys.
• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI-assisted development is now an identity governance problem, not just a code-quality problem. Once prompts, models, and workflow agents participate in software creation, the security question shifts from whether the code compiles to whether the actor behind it can be trusted and audited. That moves the control plane into IAM, workload identity, and approval governance. Practitioners should treat AI code paths as governed identities, not just developer convenience.

Judgment-in-the-loop is the right control concept because speed without discernment creates operational debt. Human review has to evaluate intent, scope, and blast radius, not merely look for syntax errors. The article is correct that a passing test suite does not prove safe behaviour. The practical conclusion is that teams need explicit review authority for AI-generated changes before production promotion.

Code provenance must expand beyond software components to include the identities that created and deployed them. SBOM thinking is necessary but incomplete if it stops at libraries and versions. A modern control model has to cover prompts, models, execution paths, and the workload identities used by build systems and agents. That is where accountability lives, and without it, incident response loses its first-order evidence.

Identity-first development pipelines are becoming a baseline requirement for autonomous software delivery. If software can be created and modified by AI, then cryptographic proof of actor identity becomes the only durable way to sustain trust. That does not eliminate the need for review, but it makes review meaningful because teams can trace the change back to a verified principal. Practitioners should align software supply-chain controls with NHI governance, not treat them as separate programmes.

From our research:

• Only 1.5 out of 10 organisations are highly confident in their ability to secure NHIs, compared to nearly 1 in 4 for securing human identities, according to The State of Non-Human Identity Security.
• Lack of credential rotation is cited as the top cause of NHI-related attacks by 45% of organisations, with inadequate monitoring and logging and over-privileged accounts both at 37%, according to The State of Non-Human Identity Security.
• For the lifecycle controls behind this gap, see NHI Lifecycle Management Guide for how provisioning, rotation, and offboarding should work in practice.

What this signals

AI-assisted development pushes the governance problem upstream, into the systems that create code rather than only the systems that run it. That means security programmes need to connect software delivery, workload identity, and approval policy in a single control story. Identity provenance debt: the longer teams allow generated changes to move without clear attribution, the harder it becomes to prove who authorized a production-impacting action. Practitioners should prepare for audits that ask for evidence of both human review and machine execution paths.

With 1 in 4 organisations already investing in dedicated NHI security capabilities, the control gap around AI-assisted workflows is likely to become a board-level issue rather than a niche engineering concern. That shift will favour programmes that can show scoped access, traceable approvals, and deterministic deployment identities. Teams should expect stronger pressure to connect AI coding governance to Top 10 NHI Issues and to baseline trust with NIST Cybersecurity Framework 2.0 functions.

For practitioners

• Require human approval for AI-generated commits Gate every AI-assisted change behind a review step that checks intent, blast radius, and whether the code should exist at all.
• Bind build actions to verified workload identities Use cryptographic identity for CI/CD systems, containers, and agents so deployment activity can be traced to a known principal.
• Extend SBOM governance to prompts and model flows Track the prompts, model boundaries, and interaction paths that shape generated output, not just package names and versions.
• Remove shared secrets from automation paths Replace plain-text API keys in agents and build workflows with scoped credentials, short-lived access, and isolated execution roles.
• Document who can approve production promotion Assign explicit authority for AI-generated changes so audits can show who validated the change and under what policy.

Key takeaways

• AI-generated code creates an accountability gap when provenance, authorship, and execution identities are no longer obvious.
• Human review remains necessary, but it must evaluate intent and blast radius, not just code correctness.
• Identity-first pipelines are the practical path to traceable AI-assisted development because they tie changes to verified principals.

Key terms

• Judgment-in-the-loop: A human review control that requires someone with the right context to decide whether AI-generated output should move forward. It goes beyond checking correctness and asks whether the change makes sense, is safe to ship, and can be attributed to a verified decision-maker.
• Identity-first development pipeline: A software delivery pipeline that treats every build, test, deploy, and automation step as an identity event. Each action is tied to a verified workload identity so organisations can trace authorship, authority, and execution across human and machine actors.
• Code provenance: The evidence trail that shows where code came from, who changed it, and why it was accepted. In AI-assisted development, provenance must include prompts, model boundaries, approvals, and the identities that executed the change, not just the final commit.
• Identity provenance debt: The accumulated security and audit risk that appears when organisations cannot reliably trace who or what created a change. As AI-generated output moves faster than review and attribution, the debt grows and makes incident response, compliance, and accountability harder.

Deepen your knowledge

AI-assisted development and workload identity governance are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building controls for AI-generated code and agentic workflows, it is worth exploring.

This post draws on content published by CyberArk: Vibe check your vibe code: Adding human judgment to AI-driven development. Read the original.