Who is accountable when a customer-facing AI gives harmful or off-topic advice?

Why This Matters for Security Teams

When a customer-facing AI gives harmful or off-topic advice, the issue is not just model quality. It is a governance and service-delivery failure that can affect safety, trust, regulatory exposure, and customer outcomes. The deploying organisation owns the experience because it chose the use case, set the guardrails, and exposed the assistant to users. Current guidance from the NIST Cybersecurity Framework 2.0 reinforces that accountability sits with the operating organisation, not the technology supplier.

That distinction matters because models can be updated, fine-tuned, or replaced, but the customer still sees one branded service. If the assistant invents policy, escalates a complaint poorly, or drifts into unsafe advice, the business must answer for the outcome and the controls that failed to prevent it. NHIMG research on the DeepSeek breach shows how exposed data and weak operational controls can turn an AI system into a broader security and trust incident. In practice, many security teams encounter accountability gaps only after a customer escalation or incident review, rather than through intentional control design.

How It Works in Practice

Accountability becomes operational when ownership is assigned across product, security, legal, and customer support, with one party responsible for runtime behaviour. That means the organisation needs policy controls, escalation paths, and evidence that the assistant is constrained by intended use. The model provider may supply infrastructure or APIs, but the deploying organisation must decide what the assistant is allowed to say, when it should refuse, and how it should hand off to a human.

Practically, this includes logging prompts and outputs, defining red-flag topics, and applying policy-as-code to limit harmful advice at runtime. For customer-facing systems, best practice is evolving toward continuous risk management, not one-time sign-off. The NIST Cybersecurity Framework 2.0 is useful here because it frames governance, protection, detection, response, and recovery as organisational responsibilities. For AI-specific accountability, NHIMG recommends pairing that with explicit model-use governance and incident handling informed by the DeepSeek breach, where data exposure and trust failure became inseparable.

• Define one accountable owner for each customer-facing assistant, not just for the model contract.
• Set approval rules for permitted topics, required refusals, and mandatory human escalation.
• Monitor outputs for harmful advice, hallucinated policy, and off-topic responses.
• Document who can change prompts, tools, retrieval sources, and guardrails.

These controls tend to break down when the assistant is integrated into multiple channels with inconsistent policies and no single incident owner.

Common Variations and Edge Cases

Tighter control often increases operational overhead, requiring organisations to balance response speed against safety, review effort, and customer convenience. That tradeoff is real, especially for support bots, internal copilots exposed to customers, or AI features embedded in self-service portals. There is no universal standard for exact accountability allocation yet, but current guidance suggests the deploying organisation should always retain primary responsibility for the user outcome.

Edge cases usually arise when a vendor hosts the model, a systems integrator builds the workflow, and the customer environment supplies the data. In that setup, contractual boundaries matter, but they do not erase the deploying organisation’s duty to monitor outputs and manage harm. The strongest interpretation of NIST Cybersecurity Framework 2.0 is that the service operator must prove it knows where risk lives and who responds when the assistant crosses a line. For teams assessing exposure patterns and operational trust failures, the DeepSeek breach is a reminder that AI incidents often start with data or control failures, then end as customer-facing harm.

Where the assistant can take actions, not just generate text, the accountability bar rises further because the organisation is now responsible for both advice and execution.

Related resources from NHI Mgmt Group

• Who is accountable when an AI concierge gives guests incorrect or harmful information?
• What makes agentic AI an NHI governance issue?
• Why is NHI governance critical in the age of AI attacks?
• Why is single-provider AI agent governance not enough for enterprise security?