What do teams get wrong about using AI agents in software delivery

Teams often assume an AI agent will behave like a constrained assistant, when in practice it may behave like an independent actor unless guardrails are enforced. Common mistakes include giving it broad permissions, skipping environment isolation, and trusting generated test results or status claims without verification. The right model is controlled automation, not delegated authority.

Why software delivery changes the risk profile of AI agents

AI agents are different from normal developer tools because they can act, call services, change state, and continue working with limited supervision. That changes software delivery from a prompt-and-review workflow into a control problem: teams must decide what an agent may touch, how much authority it gets, and how its actions are verified. OWASP’s OWASP Top 10 for Agentic Applications 2026 is useful here because the main failure mode is not the model being “wrong” in the abstract, but the system granting it too much operational trust.

The most common mistake is to treat an agent like a helpful junior engineer rather than an autonomous software actor with execution paths. That leads to overbroad permissions, weak approval boundaries, and a false sense of safety when the output looks polished. The concern is not limited to code generation; it also includes test execution, deployment actions, issue updates, and infrastructure changes. In practice, many teams encounter the real impact only after an agent has already been allowed to perform actions that should have remained human-approved.

How teams should think about agent behaviour in delivery pipelines

An effective delivery model starts by separating suggestion from action. An AI agent can draft code, propose test cases, summarise pipeline status, or prepare change descriptions, but it should not be assumed to have standing authority to merge, deploy, or modify production-adjacent resources unless that authority has been explicitly designed and monitored. The more the agent can do across tools, the more its errors resemble process failures rather than simple content mistakes.

That is why isolation and scoped execution matter. Teams need to constrain the agent to the smallest workable environment, then verify the outputs before those outputs influence downstream decisions. In software delivery, a persuasive but unverified test report is a control failure if it changes release confidence. Likewise, a generated status update that claims a build passed is only useful when the underlying logs, test artefacts, and pipeline state are independently checked. The AI risk framing in the NIST AI Risk Management Framework fits this pattern because delivery teams need governance around reliability, accountability, and traceability rather than blind automation.

Keep write access, deployment rights, and secret usage separate from prompt generation where possible.
Use short-lived, task-specific access instead of persistent authority for routine delivery work.
Verify agent-produced claims against source systems, not against the natural-language output alone.
Require human approval for actions that change release state, environment state, or security posture.

Where teams also use the agent to call external tools or orchestrate chained tasks, the boundary problem becomes more visible: every additional integration increases the chance that one mistaken instruction becomes a real operational change. This guidance breaks down when organisations cannot separate the agent’s helpful drafting role from the systems that actually execute delivery.

Common mistakes when organisations scale agents from pilot to production

Tighter control often slows delivery at first, so organisations have to balance speed against the cost of unchecked autonomy.

The first scaling mistake is granting broad permissions because the pilot “worked.” A demo environment is not representative of production risk, and a successful proof of concept often hides the fact that the agent was operating on narrow, synthetic, or low-impact targets. A second mistake is assuming that if the agent can complete a task end to end once, it can do so safely at scale. At scale, the failure pattern changes: small prompt drift, hidden tool dependencies, or inconsistent test coverage can turn one bad decision into repeated bad decisions.

There is also a governance gap teams often miss. If no one owns the agent’s allowed actions, escalation path, and review criteria, then the agent quietly becomes a shared dependency with no clear accountability. That is especially problematic in delivery pipelines, where automation already moves quickly and where status outputs are often trusted by default. The practical question is not whether the agent can help, but whether the team can prove which parts of the pipeline remain inspectable, reversible, and human-governed.

Some practitioners argue that once monitoring is added, broader autonomy is acceptable. That view is not consensus. Monitoring helps, but it does not replace permission design, isolated execution, and verification of side effects. The safest pattern is to expand capability only after teams can show that the agent’s actions are bounded, auditable, and recoverable.

Risk and Threat Considerations

The material risk is delegated authority without adequate containment. In software delivery, an AI agent that can read, write, call tools, and continue across sessions can create exposure to accidental change, misuse of privileges, and hidden supply-chain impact. The primary issue is not simply bad code generation; it is that an overtrusted agent can amplify a small mistake into an operational or security incident.

Failure mechanism: Teams give the agent broad tool access, weak environment separation, or excessive trust in generated status and test results. That allows an incorrect prompt, poisoned input, or misleading output to influence merges, deployments, secrets handling, or infrastructure changes without sufficient human review.

Impact: The result can be unsafe releases, incorrect environment changes, exposure of sensitive build or deployment credentials, and loss of confidence in pipeline evidence. In a mature delivery process, the deepest damage is often governance failure: the organisation can no longer tell which actions were truly verified and which were merely generated.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF, CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
OWASP Agentic AI Top 10	A1 — Agentic Access Control	Covers overbroad agent permissions and delegated execution authority in delivery.
Recommendation — Restrict agent permissions to the minimum actions needed for the task.
NIST AI RMF	GOV — Govern	Applies to accountability, oversight, and control boundaries for AI use in delivery.
Recommendation — Define ownership, approval points, and accountability before expanding agent autonomy.
MITRE ATLAS	AML.T0010 — Prompt Injection	Relevant where manipulated inputs can steer agent actions in delivery tooling.
Recommendation — Hunt for instruction abuse and validate agent inputs before they reach tools.
CIS Controls v8	6 — Access Control Management	Supports least-privilege and access review for delivery tools and automation.
Recommendation — Review and revoke unnecessary access paths for build and deployment automation.
NIST CSF 2.0	PR.AC-4 — Access Permissions and Authorizations	Fits the need to limit what an AI agent can do in delivery environments.
Recommendation — Apply least privilege to the agent’s execution and tool access rights.

Practitioner Guidance

What to prioritise: Treat the agent’s authority as the main design variable, not its writing quality. Decide first what it may observe, suggest, and execute, then keep those boundaries explicit in the delivery process.

What to verify: Verify the agent’s outputs against source-of-truth systems before any release decision depends on them. A test summary, code review note, or pipeline status claim should be treated as untrusted until the underlying artefacts confirm it.

Decision rule: If an agent can change state, reach secrets, or influence release approval, require stronger isolation and a human approval step. If it is only drafting or classifying, the control burden is lower but still needs auditability.

Practitioner takeaway: The most important judgement is to design AI agents in delivery as constrained automation with explicit recovery paths, because convenience without containment usually turns into delegated risk.