Organisations should define explicit identity and authorization steps before an AI assistant can act on behalf of a user. That includes verifying who is interacting, limiting what the assistant can do, and logging high-risk actions for review. The goal is to prevent prompt abuse, reduce fraud exposure, and keep the assistant inside a controlled business process.
Trust starts with a clear boundary between the customer and the assistant
An AI assistant can improve customer experience, but it should not be treated as a free-form proxy for the customer. Trust comes from making the assistant’s role explicit: what it may see, what it may say, what it may initiate, and when a human or stronger verification step must take over. That boundary should be visible in the product design and in the operating process.
For NIST AI Risk Management Framework, define trust and safety requirements as part of the AI system’s intended use and governance, not as a last-mile support issue. For ISO/IEC 42001:2023 AI Management System Standard, establish accountable AI operating controls so customer-facing behaviour is consistently reviewed, approved, and monitored.
A practical signal of good design is that the assistant can answer many questions, but cannot silently expand its authority when the conversation becomes sensitive, high-value, or ambiguous. In other words, the experience may feel seamless to the customer, but the control plane behind it should be deliberate and constrained.
Identity and authorization must be handled before the assistant acts
The core trust problem is not only what the assistant says, but what it is allowed to do on behalf of a user. Organisations should separate conversational access from action authority, then require explicit verification before the assistant can change account data, move money, disclose protected information, or trigger other business-critical workflows. When the action is higher risk than the conversation, the workflow needs a tighter gate.
NIST SP 800-207 Zero Trust Architecture supports this model by requiring continuous verification and least privilege rather than implicit trust in the session. NIST Cybersecurity Framework 2.0 is also useful for organising identity, access, logging, and response around the business process the assistant is operating inside.
In practice, the best designs separate low-risk assistance from high-risk execution. For example, the assistant may help a customer prepare a request, but a second control should confirm identity and intent before the request becomes an irreversible action.
Trust and safety depend on controlling abuse paths, not just normal use
Customer-facing assistants are attractive targets because they sit at the point where language, identity, and transaction flow meet. Attackers will try prompt abuse, impersonation, fraud, social engineering, and policy bypass to push the assistant into revealing data or performing an action it should not perform. The safety model therefore has to be tested against misuse, not only against expected customer journeys.
OWASP Agentic AI Top 10 is a strong fit where the assistant can take actions, because identity and privilege abuse, tool misuse, and human-agent trust exploitation are direct failure modes. For broader threat mapping, MITRE ATT&CK Enterprise helps teams think in attack paths such as credential access, privilege escalation, and abuse of trusted workflows.
The important practitioner point is that safety testing should include adversarial conversations, boundary probing, escalation attempts, and attempts to reuse one user’s context for another’s benefit. If those cases are not in the test plan, the assistant is probably overtrusted.
Risk and Threat Considerations
Customer-facing assistants create a concentrated trust boundary: if the assistant is overpermitted, tricked into accepting weak identity evidence, or allowed to act too far beyond the original intent, a single interaction can become fraud, data exposure, or unauthorised execution. The risk is highest where conversational convenience is allowed to outrun the controls that would normally protect a transaction.
Failure mechanism: The assistant accepts spoofed intent, weak verification, or poisoned context and then uses that trust to disclose information, bypass policy, or trigger a business action with inadequate authorization.
Impact: The result can be account takeover support, fraudulent transactions, privacy breaches, customer harm, and loss of confidence in the digital channel.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack surface, NIST AI RMF and NIST Zero Trust (SP 800-207) set the technical controls, and ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | AI customer interactions need governance, accountability, and risk controls. |
| Recommendation — Set governance and risk policies for customer-facing AI assistant behavior. | ||
| ISO/IEC 42001:2023 | AI Management System | An AI management system formalizes controls for trustworthy AI operations. |
| Recommendation — Establish accountable operating controls for customer-facing AI assistants. | ||
| NIST Zero Trust (SP 800-207) | ZT-207 — Zero Trust Architecture | Trust should be verified before an assistant acts on behalf of a user. |
| Recommendation — Require explicit verification and least privilege before sensitive assistant actions. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | AI assistants can be abused through overbroad identity and privilege. |
| Recommendation — Constrain assistant privileges and require step-up checks for high-risk actions. | ||
| MITRE ATT&CK | T1110 — Brute Force | Fraud and abuse attempts often include repeated authentication abuse. |
| T1078 — Valid Accounts | Trusted customer sessions and stolen accounts are common abuse paths. | |
| Recommendation — Monitor for repeated auth abuse and escalating abuse patterns in assistant workflows. Detect and investigate use of valid accounts in high-risk assistant actions. | ||
Practitioner Guidance
What to prioritise: Prioritise the highest-risk actions first, not the most visible chat flows. If an assistant can change money movement, account recovery, access rights, or personal data, that path needs stronger identity proof, tighter authorization, and better review than routine support questions.
What to verify: Verify that the assistant cannot convert a low-friction conversation into a high-impact action without an explicit control step. Good evidence includes clear authorization rules, audit logs for sensitive actions, and a documented break-glass or manual-review path for exceptions.
Practitioner takeaway: Trust and safety are earned when the assistant is useful by default but strictly bounded when it starts to act, especially at the point where convenience would otherwise outrun accountability.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 26, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
