Why do chat-based AI systems create new identity risk for organisations?

Why This Matters for Security Teams

Chat-based AI systems create a new identity problem because the user’s text can trigger action, delegation, and tool use in a single flow. That collapses the usual checks that separate intent from execution. A prompt can look harmless while still causing a connector to read data, create tickets, send messages, or call external APIs. In practice, the risk sits in the identity path around the model, not just in the model itself.

This is why conventional IAM patterns can miss the real exposure. Static roles assume predictable behaviour, but chat-driven systems are often goal-seeking and context sensitive. Guidance from NIST Cybersecurity Framework 2.0 still applies, but it has to be interpreted through the identity layer that surrounds agents, connectors, and approval workflows. NHI management becomes central when the system uses service accounts, API keys, or delegated tokens to move from conversation to action.

NHIMG research shows why this is so hard to ignore: in the Ultimate Guide to NHIs, 97% of NHIs carry excessive privileges, which broadens the blast radius when a chat workflow misfires. In practice, many security teams only discover this after an assistant has already touched systems it should never have reached.

How It Works in Practice

The operational risk begins when a chat interface is wired to connectors, retrieval, or agent tools. A user may ask for a summary, but the system may also retrieve records, open a document, or invoke an API as part of its chain of reasoning. If those actions reuse long-lived secrets or over-scoped tokens, the assistant inherits more power than the original request justifies.

Current guidance suggests treating the AI as a workload identity problem, not only an application problem. That means the system should prove what it is, what it is allowed to do, and for how long. In mature designs, identity is bound to the workload using cryptographic workload identity, while permissions are evaluated at runtime using intent-based or context-aware authorisation. This is where OWASP NHI Top 10 and Ultimate Guide to NHIs — Why NHI Security Matters Now are useful references for mapping the control gaps.

• Issue JIT credentials per task, then revoke them automatically when the task ends.
• Prefer short-lived secrets over static credentials, especially for connectors and tool chains.
• Evaluate authorisation at request time, not only at login time.
• Separate user intent from execution authority so a prompt cannot silently become a privileged action.
• Log the full chain of action ownership, including which identity approved and which identity executed.

For implementation, teams often align the control plane with policy-as-code, then use frameworks such as SPAFFE or OIDC-based workload identity to reduce token sprawl. The practical goal is zero standing privilege for the agent path, with every high-risk action requiring fresh context and explicit approval. These controls tend to break down when legacy SaaS connectors cannot issue short-lived tokens or when multiple agents share one service account because attribution and revocation become ambiguous.

Common Variations and Edge Cases

Tighter identity control often increases friction, so organisations must balance safety against throughput. That tradeoff becomes visible in customer support bots, coding assistants, and internal copilots where repeated low-risk actions can make per-task approval feel too slow. Best practice is evolving here: there is no universal standard for how much autonomy is acceptable, only a growing consensus that higher-risk actions need stronger runtime checks.

One common edge case is the human-in-the-loop design that appears safe but still fails because the approval is too coarse. If a person approves “send the report,” that approval may unintentionally cover data collection, file generation, and distribution. Another is multi-agent orchestration, where one agent delegates to another and the original owner loses sight of the effective permissions chain. 52 NHI Breaches Analysis shows how often these failures stem from hidden identity relationships rather than obvious authentication gaps.

Identity governance also changes when the system must act at machine speed. DeepSeek breach illustrates how exposed secrets can turn AI-related systems into a fast-moving target, while NIST AI risk guidance and frameworks such as CSA-MAESTRO and NIST-AIRMF push teams toward clear ownership, bounded autonomy, and continuous evaluation. For chat-based AI, the safest posture is to assume the prompt can become an action unless the identity layer proves otherwise.

Related resources from NHI Mgmt Group

• Why do AI agents create new risk in non-human identity management?
• Why do multi agent systems create more identity risk than single AI assistants?
• Why do AI native workflows create more identity risk than traditional engineering models?
• Why do AI agents increase non-human identity risk in existing IAM programmes?