Who is accountable when an AI chatbot makes a false customer promise?

Why This Matters for Security Teams

A false customer promise is not just a chatbot error; it is a service commitment expressed through software. If an AI says a refund is approved, a shipment is guaranteed, or a warranty is extended, the customer will treat that statement as binding unless the organisation clearly scoped the interaction. That is why accountability sits with the business that deployed the system, not the model itself. Current guidance suggests the right question is not “Can the model speak?” but “Who approved the policy, the data, and the action path?”

This is where governance, access control, and identity meet. Customer-facing AI is increasingly connected to CRM, order management, and support tooling, which means a bad statement can trigger a bad action. NIST SP 800-63 Digital Identity Guidelines is useful here because it reinforces that identity assurance and traceability matter when digital systems make assertions on behalf of an organisation. NHI teams also see the same pattern in real incidents such as the OmniGPT breach, where trust boundaries around AI output became operationally relevant fast. In practice, many security teams encounter accountability failures only after a customer disputes the promise, rather than through intentional control testing.

How It Works in Practice

Accountability becomes practical when the organisation can trace four things end to end: the user prompt, the model or agent response, the policy decision that allowed the response, and the downstream system action, if any. If the AI only answered in text, the issue is usually customer communications governance. If the AI also triggered a refund, discount, or case update, the issue becomes entitlement, approval logic, and workload identity.

For customer-facing assistants, the safest pattern is to separate “informational” responses from “committal” responses. The bot can explain policy, but it should not guarantee an exception unless a real authoriser or policy engine has approved it. That is where intent-based authorisation and just-in-time credentialing matter in adjacent agentic systems, because dynamic authority should be granted only for the specific task and revoked immediately after. NIST-AIRMF and NIST SP 800-63 Digital Identity Guidelines both support the broader principle that trustworthy AI requires traceability, identity context, and accountable human oversight.

• Log the exact prompt, retrieved context, policy decision, and response.
• Restrict the chatbot from making irreversible promises unless a downstream approval path exists.
• Use role-based access for operators, but do not assume RBAC alone can govern the content of AI responses.
• Bind the service to a verifiable workload identity so actions are attributable to the deployed system, not a generic API token.

NHI evidence from the DeepSeek breach shows how quickly sensitive AI surfaces can become operational risk when controls lag behind usage. These controls tend to break down when the chatbot is allowed to act across multiple backend systems without a policy gate, because the promise can become an automated change rather than a mere statement.

Common Variations and Edge Cases

Tighter promise controls often increase support friction, requiring organisations to balance customer convenience against legal and operational risk. That tradeoff is real, especially when business teams want the bot to feel decisive while legal teams want every exception to be conditional. Best practice is evolving, and there is no universal standard for this yet.

Edge cases usually appear where the chatbot mixes general guidance with account-specific decisions. A shipping estimate is usually acceptable if framed as an estimate; a guaranteed delivery date is different. A published policy summary is fine; a special exception to that policy is not, unless the workflow has explicit approval. The same caution applies when a bot is connected to agentic workflows, because autonomous systems can chain tools, escalate scope, and create commitments that were never intended. The security lesson from incidents such as the Schneider Electric credentials breach is that trust in the interface does not remove the need for identity, logging, and separation of duties.

Where teams usually go wrong is assuming the chatbot’s wording alone defines liability. In reality, the liability picture depends on policy, delegated authority, and whether the system had the ability to complete the promise. When customer support AI is integrated with account changes, the organisation should treat every committed answer as an operational action that needs governance, review, and revocation paths.

Related resources from NHI Mgmt Group

• What makes agentic AI an NHI governance issue?
• How should ecommerce teams govern customer-facing AI that can influence purchases?
• What makes idle secrets such a serious NHI risk?
• Why is NHI governance critical in the age of AI attacks?