Who is accountable when an agent turns feedback into a code change?

Why This Matters for Security Teams

When an agent converts feedback into a code change, the question is not whether automation helped, but whether the organisation preserved a clear approval chain. That distinction matters because code changes can affect availability, security, compliance, and customer data even when the original issue looked minor. The accountability model should stay tied to the workflow owner, while the agent is treated as an execution mechanism rather than a decision-maker. Guidance from the NIST AI Risk Management Framework is useful here because it emphasises governance, mapping responsibilities, and monitoring AI-enabled activity rather than assuming automation is self-governing.

Security teams often get this wrong by assuming that a generated pull request, ticket update, or remediation suggestion is low risk because a human can still intervene later. In practice, the real control issue is whether anyone can prove who authorised the change, who reviewed the evidence, and whether the agent was allowed to touch production-scoped code at all. That becomes especially important when the feedback came from a security finding, customer report, or automated detection pipeline. In practice, many security teams encounter accountability gaps only after an agent has already created a risky change request, rather than through intentional review design.

How It Works in Practice

The practical model is straightforward: the agent can draft, summarise, and prepare change artefacts, but a named human owner must remain accountable for approval and release. That owner should be able to explain why the change was accepted, what evidence was reviewed, and which fallback path existed if the agent misunderstood the feedback. For higher-risk environments, the evidence packet should include the triggering issue, the generated diff or patch, test results, policy checks, and the approval record.

Operationally, mature teams separate four functions:

• the person who owns the service or codebase,
• the person who validates the agent’s proposed change,
• the person who authorises deployment, and
• the system that records immutable audit evidence.

This is where agentic AI governance overlaps with secure software delivery. The OWASP Agentic AI Top 10 and the CSA MAESTRO agentic AI threat modeling framework both reinforce the need to define agent scope, action boundaries, and escalation points. For threat modelling, the MITRE ATLAS adversarial AI threat matrix is useful when the feedback loop itself could be manipulated, such as poisoned tickets, prompt injection in issue text, or adversarial change requests.

In a well-run process, the agent should not be able to bypass code owners, branch protections, security checks, or segregation of duties. The system should log whether the change was human-reviewed, machine-suggested, or machine-executed, because those distinctions matter during incident response and post-incident review. These controls tend to break down when feedback arrives through loosely governed chat channels or when agents are connected directly to deployment tooling without enforced approval gates.

Common Variations and Edge Cases

Tighter approval control often increases delivery friction, requiring organisations to balance speed against the risk of accidental or unauthorised change. That tradeoff becomes sharper in high-velocity engineering teams, where automated remediation is attractive but can conceal weak oversight if the review path is too thin.

There is no universal standard for this yet, but current guidance suggests using stronger controls as the change becomes more sensitive. A low-risk documentation fix may justify lightweight review, while a production code change tied to access control, secrets handling, or authentication logic should require explicit human approval and traceable evidence. The NIST AI Risk Management Framework and NIST SP 800-53 Rev 5 Security and Privacy Controls are especially relevant where auditability, change control, and access restriction need to be demonstrable.

Edge cases appear when the agent acts on behalf of a shared team, a temporary incident response squad, or a third-party managed service. In those environments, accountability should be attached to an identified service owner and a specific control plane, not to a generic “AI system” label. The lesson from recent reporting such as Anthropic's report on an AI-orchestrated cyber espionage campaign is that autonomy becomes risky fastest when systems can chain small decisions into real operational impact. The guidance breaks down most sharply in environments where agents can open, approve, and deploy changes through the same identity without independent oversight.

Related resources from NHI Mgmt Group

• Who is accountable when an agent reopens the same PR or repeats a side effect after recovery?
• Who is accountable when an AI coding agent sends traffic to the wrong system?
• Who is accountable when a self-modifying agent causes a bad outcome?
• Who is accountable when a vendor-hosted CX agent leaks data or exposes accounts?