Who is accountable when a model is coaxed into unsafe output over multiple turns?

Accountability sits with the programme that approved the model’s runtime governance, not with the individual user prompt. If the model can act, reveal data, or trigger downstream workflows, then the organisation must be able to prove that session-level controls, monitoring, and escalation handling were in place. That aligns AI oversight with established governance and access-control expectations.

Why This Matters for Security Teams

Multi-turn coaxing is not just a prompt quality issue. It is a governance problem because the unsafe outcome usually emerges only after the system has preserved context, accepted repeated nudges, and continued to execute with the same tool permissions. If a model can reveal sensitive data, call external services, or influence downstream workflows, accountability shifts to the team that approved those runtime conditions and the monitoring around them. NIST’s control baseline in NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it reinforces that access, logging, and oversight are organisational responsibilities, not user-side assumptions.

What many practitioners miss is that the risk is cumulative. A single safe response can become an unsafe one when the model is gradually led into policy exceptions, hidden dependencies, or overbroad tool use. That means the right question is not whether one prompt was malicious, but whether the session design made persistent manipulation possible without detection. In practice, many security teams encounter this only after a benign-looking conversational sequence has already triggered an unsafe action, rather than through intentional control testing.

How It Works in Practice

When a model is coaxed over multiple turns, the relevant control point is the session itself. The organisation needs to know whether the model retained context, whether that context could be poisoned, and whether any tool or workflow permissions were still active when the unsafe request arrived. Good practice is to treat conversational state as a governed asset with logging, escalation, and rollback paths, especially if the model can retrieve data or perform actions.

A practical response usually includes four layers:

• Session-level policy checks that reassess risk on every turn, not just at initial prompt acceptance.
• Output and action gating so the model cannot escalate from answer generation into execution without a second control.
• Telemetry that records prompt sequence, tool calls, policy hits, and human overrides for later review.
• Clear ownership for incident response when the model crosses a safety boundary or exfiltrates data.

That operational model fits current AI governance thinking in NIST AI Risk Management Framework and the adversarial behaviour patterns described by MITRE ATLAS. If the system is agentic, the same logic extends to tool invocation, workflow delegation, and inherited permissions: accountability is shared across the model owner, the platform owner, and the team that approved the runtime controls, but not displaced onto the end user who merely completed the conversation.

For teams operating under formal AI governance, the crucial test is whether they can reconstruct the session, explain the control decision, and show what stopped the unsafe turn from becoming an unsafe action. These controls tend to break down when the model is connected to live business systems with long-lived context and weak event logging because the conversation outlasts the visibility of the people approving it.

Common Variations and Edge Cases

Tighter runtime control often increases friction for legitimate users, requiring organisations to balance safety against usability and operational speed. That tradeoff becomes sharper when the model serves customer support, SOC triage, or internal automation, where repeated turns are normal and strict blocking can interrupt real work.

There is no universal standard for every deployment, but current guidance suggests a few exceptions need special handling. A passive chat assistant with no memory beyond the current turn creates different accountability pressure than an agent that can open tickets, query systems, or send messages. Likewise, a model used for internal drafting is not the same as one exposed to untrusted users with access to sensitive repositories. Where the system has tools, the organisation should treat every turn as potentially policy-relevant and every unsafe escalation as a control failure to be investigated, not as a user-only event.

This is also where agentic AI governance and NHI governance intersect. If the model is acting through a service identity, API token, or delegated permission set, then the session is not only an AI safety issue but also a non-human identity control problem. OWASP guidance for LLM applications and emerging agentic AI practices both point toward the same operational conclusion: prove who approved the runtime authority, what constraints were active, and how repeated-turn abuse would be detected before it reaches a downstream system.

Related resources from NHI Mgmt Group

• Who should be accountable for AI overspend when multiple teams share the same model?
• Who is accountable for model outputs that leak unsafe guidance through iterative probing?
• How can teams reduce the impact of unsafe model output in MCP workflows?
• Who is accountable when a jailbroken model causes an unsafe enterprise action?