Because IAM can authenticate users, but it cannot by itself control what those users paste into a model or how they interpret the response. AI risk often arises after valid access is granted, so the missing layer is policy, data classification, and workflow control around the interaction itself.
Why This Matters for Security Teams
Standard IAM is good at proving who signed in, but it does not solve the full chatbot risk surface. Once a user is authenticated, the real exposure comes from what they paste, what the model returns, and how that output is reused in downstream workflows. That is why chat controls must extend beyond login and into data handling, prompt governance, and output review.
This gap is visible in incident reporting around exposed credentials and AI misuse, including NHIMG coverage of the LLMjacking attack pattern and the broader NHI maturity findings in the 2024 Non-Human Identity Security Report. NIST also treats governance and risk controls as part of cybersecurity, not an afterthought to authentication, in the NIST Cybersecurity Framework 2.0.
The practical problem is that a valid session can still carry sensitive data into an unmanaged prompt, or allow a response to be copied into systems that should never receive it. In practice, many security teams encounter chatbot leakage only after sensitive data has already been pasted into the model, rather than through intentional data governance.
How It Works in Practice
Security teams need to separate identity control from interaction control. IAM authenticates the person, service, or agent, but chatbot risk is usually created by the content of the conversation and the context of the action. That means policy must evaluate the request at the moment of use, not just at sign-in.
Practical controls usually include data classification, prompt filtering, session logging, retention limits, and workflow approvals for high-risk actions. For example, a finance user may be allowed into the chatbot, but policy can still block pasting customer records, prevent the model from summarising regulated data, or require review before any output is exported. NHIMG’s guidance on Ultimate Guide to NHIs — Key Challenges and Risks is useful here because it frames the access problem as an operational control issue, not only an identity issue.
In more mature environments, the chatbot sits inside a policy layer that can inspect context such as user role, data sensitivity, destination system, and whether the output will trigger an automated action. That approach aligns with the direction of current guidance in OWASP and NIST, where the control point is the runtime decision, not only the account boundary. The OWASP NHI Top 10 is particularly relevant because it highlights how identity, prompt handling, and tool access combine into a single attack path.
- Use IAM for authentication and least privilege.
- Use DLP and classification to control what can be entered or returned.
- Use policy-as-code to evaluate prompt, context, and destination at runtime.
- Log prompts and outputs where compliance requires traceability.
These controls tend to break down when users can copy data between unmanaged SaaS tools and the chatbot, because the policy boundary no longer covers the full workflow.
Common Variations and Edge Cases
Tighter chatbot controls often increase friction, so organisations have to balance user productivity against the risk of data leakage and unsafe automation. There is no universal standard for this yet, especially where teams are still deciding how much prompt content should be logged or inspected.
One common edge case is employee use of approved chatbots for unapproved data classes. Another is agentic or tool-using chatbots, where a valid login can trigger email, ticketing, or code actions that exceed the original user intent. In those cases, static roles are too coarse, and current guidance suggests adding context-aware approval gates and short-lived access tokens instead of relying on a permanent entitlement model.
NHIMG’s Top 10 NHI Issues and Ultimate Guide to NHIs — Why NHI Security Matters Now both reinforce the same operational lesson: identity is necessary, but it is not sufficient when the workload can transform input into action. The control objective should be preventing sensitive content from becoming an unsafe instruction path, not just verifying that a user is allowed to open the chat window.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-03 | Covers weak secret and access handling around AI/chatbot workloads. |
| NIST CSF 2.0 | PR.AC-4 | Identity alone is insufficient; access must be limited by context. |
| NIST AI RMF | AI RMF governance addresses data, output, and workflow risk beyond IAM. |
Pair authenticated access with least-privilege and contextual authorization controls.
