AI agents are changing software delivery and AppSec governance

At a glance

What this is: This analysis explains why autonomous AI agents are becoming part of software delivery and how their speed creates new application security and governance gaps.

Why it matters: It matters because AppSec, IAM, and security teams need controls that address machine-led action, tool use, and oversight in development pipelines, not just human developer behaviour.

By the numbers:

• In Gartner’s 2024 Software Engineering Leader survey, over 50% of respondents said they are currently using or actively planning to adopt generative or autonomous AI tools.
• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read Checkmarx's analysis of AI agents and application security governance

Context

Autonomous AI agents are software entities that can plan and act with limited human intervention, which means they are no longer just coding assistants. In software engineering, that creates a governance problem for AI agent governance, because the system can now change code, trigger workflows, and influence release decisions faster than conventional review cycles can reliably intervene.

The primary issue is not whether AI can accelerate developer productivity. It is that the control plane around software delivery was built for human-paced decision-making, while agentic workflows introduce machine-paced action, delegation, and tool use. That creates a genuine intersection with identity and access governance, because agent permissions, secrets, and workload trust now need to be managed as carefully as human access.

This starting position is typical of fast-moving engineering organisations, which often adopt AI capabilities before they fully map the associated access and accountability model.

Key questions

Q: How should security teams govern AI-generated code in production pipelines?

A: Security teams should treat AI-generated code as a controlled identity event, not just a development artifact. Require human approval, traceable authorship, scoped workload identities, and evidence of intent before production promotion. The goal is to preserve provenance and limit blast radius when generated logic behaves unexpectedly.

Q: Why do AI agents create new AppSec risk even when code quality improves?

A: AI agents reduce manual work, but they also compress decision-making into machine-paced actions that can bypass human review windows. The risk increases when agents can consume untrusted input, select tools, and execute changes in the same workflow. Better code generation does not eliminate delegation risk, access misuse, or supply chain drift.

Q: What breaks when policy controls are only applied after code is generated?

A: When policy checks happen too late, the agent has already influenced code, dependencies, or pipeline behaviour. That leaves teams trying to catch problems after the risky action has occurred, which is slower and less reliable than preventing the action in the first place. Runtime enforcement and pre-execution guardrails are the only controls that scale with agent speed.

Q: Why do AI agents make non-human identity governance harder?

A: AI agents make governance harder because they can request tools, act autonomously, and change behaviour across sessions while still relying on machine credentials. That increases the number of access paths security teams must supervise. The result is a stronger need for task-scoped access, explicit ownership, and continuous monitoring of what the agent can reach.

Technical breakdown

How autonomous AI agents alter the software delivery trust model

Autonomous AI agents differ from simple automation because they can decide what to do next, invoke tools, and adapt their actions based on runtime context. In software delivery, that means they may create code, open pull requests, run tests, or query internal systems without a human author stepping through each action. The risk is not just erroneous output. It is that the trust boundary moves from a person making a deliberate change to a system chaining actions across multiple tools. That complicates provenance, accountability, and review because the meaningful security question becomes who or what was authorised to act at each step.

Practical implication: map every agent action to a specific identity, permission, and approval boundary before letting it touch production-adjacent systems.

Why prompt injection and supply chain drift matter in agentic AI

Prompt injection is a manipulation technique where malicious instructions are embedded in inputs that the agent treats as operational guidance. Supply chain drift happens when the agent starts pulling in packages, snippets, or dependencies that were never vetted through the normal software assurance process. In practice, these two failure modes combine when an agent both consumes untrusted content and then acts on it through tools with broad permissions. The result is not just a bad answer. It can become code execution, package compromise, or policy bypass in the development lifecycle.

Practical implication: restrict the tools, registries, and prompts that agents can reach, and verify every external dependency before it enters a build path.

Why policy-as-code and runtime monitoring must move closer to the agent

Traditional AppSec often assumes insecure code will be caught in review, staging, or scanning after creation. Agentic workflows compress that timeline, so controls need to operate while the agent is still deciding and acting. Policy-as-code gives teams a way to enforce boundaries during pull requests, build steps, and pipeline execution. Runtime monitoring then checks for abnormal behaviour such as unusual resource consumption, repeated policy violations, or unexplained access to internal systems. Together, these controls shift security from retrospective detection to active containment.

Practical implication: enforce policy at the point of action and monitor agent behaviour continuously rather than relying on post-build discovery.

NHI Mgmt Group analysis

AI agent governance has become an access-control problem, not just a development efficiency issue. Once agents can independently create, modify, and route software artefacts, they behave like privileged operational actors inside the delivery chain. That means the core control question shifts from code quality alone to authorisation, delegation, and auditability. Teams that treat AI agents as enhanced tools rather than governed runtime entities will leave gaps in accountability and access control.

AI governance debt is the hidden risk behind rapid agent adoption. The faster organisations layer agents into IDEs, CI/CD systems, and infrastructure workflows, the more difficult it becomes to retroactively define ownership, scope, and rollback boundaries. This is not a tooling problem in isolation. It is a governance debt problem that grows each time an agent is allowed to act before the control model is finalised. Practitioners should interpret early adoption as a signal to document decision rights now.

Prompt injection and supply chain drift expose a shared failure mode: untrusted inputs gaining execution influence. These risks are different in mechanism but similar in consequence, because both let external content shape internal action. That makes software engineering controls and identity controls converge around the same principle: nothing should inherit trust simply because an agent can read it. The practical lesson is to constrain what can be consumed, what can be executed, and what can be delegated.

Identity governance must extend to the agent identity itself. When AI systems can call APIs, access repositories, and interact with internal services, they need scoped identities, monitored entitlements, and revocation logic just like any other high-risk non-human identity. This is where NHI governance and agentic AI governance meet. The discipline now has to account for software that both consumes and acts on privileges, which makes least privilege and lifecycle control non-negotiable.

From our research: 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

What this signals

AI governance debt will compound fastest in engineering teams that treat agents as productivity add-ons rather than governed actors. The practical signal is that access reviews, pipeline monitoring, and rollback planning must move earlier in the lifecycle. Teams that wait for a formal AI programme will already have created untracked permissions and undocumented decision paths.

Agent control design should now sit alongside secrets management and workload identity in platform security reviews. If an agent can act on behalf of a team, its permissions and evidence trail need to be reviewed with the same discipline used for other privileged workloads. That makes lifecycle control and revocation speed more important than raw model capability.

Practical programmes should align their controls to NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 while defining where each agent can act, what it can touch, and who can terminate it. The named concept here is agent governance debt, which captures the cost of deferring control design until after deployment.

For practitioners

• Define agent identity boundaries Assign every AI agent a distinct identity, bounded permissions, and a named owner so access can be revoked, reviewed, and audited like any other privileged workload. Start with the agents that can reach source control, CI/CD, or internal APIs.
• Constrain tool and data access Limit which repositories, registries, prompts, and internal systems an agent can reach. Use allowlists for tools and data sources, and deny access by default when the agent does not need the capability for its current task.
• Enforce policy at execution time Apply policy-as-code in pull requests, build steps, and deployment workflows so an agent cannot bypass controls simply because it operates faster than a human reviewer. Couple this with approval gates for high-risk changes.
• Monitor for anomalous agent behaviour Log agent actions, access patterns, failed policy checks, and resource spikes, then correlate them with code changes and pipeline events. Look for repeated tool calls, unexplained package pulls, and access outside the expected task scope.
• Build an agent rollback and revocation runbook Prepare a fast way to disable agent credentials, remove pipeline permissions, and revert risky code when an agent misbehaves. The runbook should cover both the agent identity and any secrets or tokens it may have exposed.

Key takeaways

• AI agents are becoming operational actors inside software delivery, which turns AppSec into a governance and access-control problem as much as a code-quality problem.
• Gartner’s survey signal shows adoption is already well under way, while NHIMG research shows most organisations are still not governing agent behaviour effectively.
• Teams should move controls into the workflow, assign identities to agents, and monitor runtime behaviour before agentic adoption outpaces the control model.

Key terms

• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• AI Governance: AI governance is the set of controls used to discover, classify, approve, restrict, monitor, and revoke AI-enabled access. It connects identity, data, and policy so organisations can manage what AI can reach, what it can share, and when it should be stopped.
• Policy as Code: Policy as code stores authorization logic in version control and evaluates it through testable, reviewable rules. For agent governance, it makes runtime decisions reproducible and measurable, which is critical when actions can be triggered by untrusted content and executed at machine speed.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full article

Checkmarx's full article covers the operational detail this post intentionally leaves for the source:

• Examples of how the vendor suggests embedding controls into IDEs, build pipelines, and merge workflows
• Specific use cases for AI-augmented vulnerability remediation and code-fix practices
• The vendor's practical guidance for secure prompt design and monitoring agent behaviour
• Implementation-oriented discussion of multidisciplinary governance teams for DevSecOps adoption

👉 The full Checkmarx article covers the deployment examples, security practices, and team operating model in more detail

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners design control models for non-human systems that can act, delegate, and be audited.