What do organisations get wrong about custom agent accountability?

Where accountability actually starts for a custom agent

custom agent are often treated like black boxes once they are deployed, but that misses the real control surface. The meaningful question is not only what the agent produced, but who approved the instructions, what source material it was allowed to use, and who owns changes to either of those inputs. That is where accountability becomes traceable, reviewable and enforceable.

When organisations fixate on output logs alone, they are usually checking the last step in a chain of decisions that started much earlier. If the agent is tuned through prompts, policies, retrieval sources or tool permissions, then those upstream choices determine the agent’s behaviour more than any single response record. A solid accountability model therefore treats configuration ownership as a first-class security and governance control, not an administrative detail.

That is also why change tracking matters as much as approval. A safe agent can become unsafe through quiet prompt edits, source expansion, connector changes or revised approval rules, even when the runtime logs still look normal. The accountability record needs to show who changed what, when, why and under whose authority, so reviewers can reconstruct the decision path behind the agent’s recommendations.

Why logging outputs is too late to prove responsibility

Output logging is useful for forensics, but it rarely answers the governance question that most organisations are trying to solve. If a custom agent gives a bad recommendation, the real issue is often whether the underlying instructions were approved, whether the source set was vetted, and whether the person or team that changed them had the right mandate. Those are control questions, not transcript questions.

agent accountability breaks down when teams assume that “we can see what it said” is the same as “we know why it was allowed to say it.” The latter requires lineage for configuration and knowledge sources, plus ownership that survives handoffs between product, security, operations and the business function using the agent. Without that lineage, investigations tend to focus on symptoms instead of the decision point that created the behaviour.

In practice, this means the strongest evidence is usually upstream: configuration baselines, approval records, source allowlists, version history and named owners. Those artefacts make it possible to decide whether the agent acted within its intended scope or whether the scope itself was drifted, weakened or never properly sanctioned.

What mature agent governance should be able to answer

A mature model should let you answer three questions quickly: who owns the agent’s operating configuration, who approved the source material it can rely on, and who is accountable when either changes. If the answer is diffuse, the organisation has created a governance gap even if the agent appears well monitored.

That gap becomes more serious when agents are allowed to combine retrieval, tools and autonomous recommendations. Once an agent can infer from approved sources and then recommend or initiate action, the approval boundary has to cover the inputs as well as the outputs. NHIMG’s NHI Ownership and Accountability Guide is useful here because it reinforces the broader principle that ownership must be explicit, current and attached to the thing creating the risk, not inferred after the fact.

For agent programs, the practical standard is simple: every meaningful behaviour change should be attributable to a controlled change request, not to an unnoticed prompt edit or source expansion. If a team cannot point to that trail, then accountability is being asserted rather than demonstrated.

Risk and Threat Considerations

When accountability is anchored only in output logging, organisations create a blind spot for configuration drift, unauthorised instruction changes and source poisoning. That can lead to bad recommendations, unsupported inferences or privilege misalignment that is hard to spot after the fact.

Failure mechanism: A person with edit access changes prompts, retrieval sources or agent rules without a clear approval trail, so the agent’s behaviour changes even though its outputs still look routine.

Impact: Investigations cannot reliably determine whether the failure came from the model, the source set or the change process, which weakens trust, slows remediation and can spread the same defect across many deployed agents.

Practitioner Guidance

What to verify: Treat the agent configuration, not just the transcript, as the audit object. Verify that every production agent has a named owner, an approved source set and a versioned change record that can be reviewed independently of runtime logs.

Decision rule: If you cannot show who approved the instructions and knowledge sources, do not treat the agent as accountable enough for autonomous recommendations. Limit its scope until the configuration trail is explicit and reviewable.

What practitioners underestimate: Teams often assume monitoring will compensate for weak ownership, but logging cannot recreate missing governance. The harder problem is proving that the agent was authorised to know what it knew and to change only under controlled review.

Practitioner takeaway: Accountability is established before the agent speaks, because the real control point is the governed configuration that shapes its reasoning, not the output it later produces.

Related resources from NHI Mgmt Group

• What do organisations get wrong about AI agent inventory and visibility?
• What do organisations get wrong about agent identity attribution?
• What do organisations get wrong about AI agent safety at design time?
• What do organisations get wrong about AI agent authorisation?