AI chatbots in business expand value and risk for identity teams

At a glance

What this is: This is an analysis of how business AI chatbots create new operating value while widening security, data-handling, and governance risk across customer-facing and back-office workflows.

Why it matters: It matters because identity teams must decide whether existing IAM, NHI, and AI governance controls can safely cover chatbot access to data, actions, and user interactions.

By the numbers:

• Gartner has forecasted that approximately 30% of all Fortune 500 companies will use single AI-enabled channels for customer service by 2028.
• By November 2023, Erica had reached 40 million users, and by April 2024 it had handled 2 billion interactions.
• More than 27% of the global online population uses voice search on mobile.

👉 Read Lakera's analysis of AI chatbots in business

Context

AI chatbots in business are no longer confined to simple customer-service scripts. As models become more capable, organisations are wiring them into support, sales, hiring, finance, and service workflows that touch sensitive data and decisions.

That creates an identity governance problem as much as an AI problem. When a chatbot can see account data, retrieve internal knowledge, or trigger downstream actions, security teams must decide which controls govern the identity behind the interaction, what data it may reach, and how misuse is detected.

The primary keyword here is AI chatbots in business, but the real issue is governance at the boundary between conversational interfaces and enterprise access. Existing controls built for human users or static service accounts often assume a more predictable request pattern than modern AI systems actually create.

Key questions

Q: How should security teams govern AI chatbots that can access business systems?

A: Start by assigning each chatbot a clear identity, then limit its permissions to the smallest set of data and actions needed for the use case. Separate read access from write access, require approval for downstream actions, and log every connector, prompt, and response path. If the chatbot can change systems, not just answer questions, it needs governance like any other privileged access path.

Q: Why do AI chatbots create IAM risk even when they are not fully autonomous?

A: Because they often inherit access from service accounts, APIs, and vendor integrations that are broader than the conversation needs. The model may not be autonomous, but the surrounding identity and data paths can still expose sensitive systems. That means the security problem sits in delegation, permissions, and data reach, not only in model behaviour.

Q: What do organisations get wrong about chatbot hallucinations?

A: They often treat hallucination as a user-experience issue instead of a governance issue. Wrong answers can trigger bad customer decisions, unsafe internal actions, or reputational harm even when no attacker is present. Teams should validate source data, constrain where answers can be used, and avoid letting unverified outputs drive critical decisions.

Q: Who is accountable when a business chatbot leaks data or acts incorrectly?

A: Accountability should sit with the team that owns the identity, the data source, and the workflow the chatbot touches, not with the model alone. If the chatbot operates inside customer service, HR, or finance, those business owners share responsibility for access scope, logging, and approval paths. Governance must match the business function the chatbot serves.

Technical breakdown

How AI chatbots inherit enterprise access

Modern business chatbots sit on top of authentication, APIs, knowledge stores, and sometimes workflow engines. They do not create access by themselves, but they inherit the permissions of the account, service principal, or agent wrapper that connects them to enterprise systems. That matters because the chatbot becomes the front end to whatever identity is behind it. If the integration is over-scoped, the bot can expose more data than a human support agent would ever need. If the prompts or retrieval layer are weak, the system can surface sensitive records in ways the organisation did not intend.

Practical implication: map every chatbot to the identity and data sources it can reach, then reduce those permissions to the minimum viable scope.

Why jailbreaks and hallucinations are different control problems

Jailbreaking is an abuse problem, where an attacker tries to override the model’s intended behaviour and extract sensitive information or unsafe actions. Hallucination is a reliability problem, where the chatbot generates incorrect or invented outputs that may still sound authoritative to users. Security teams often treat these as one risk, but they require different controls. Abuse is about constraining prompts, tools, and outputs. Reliability is about validating retrieval sources, limiting autonomy, and making sure critical decisions are not taken from unverified model responses. A chatbot can be secure from a leakage perspective and still be operationally unsafe if users rely on bad answers.

Practical implication: separate abuse controls from answer-quality controls so that policy, content, and workflow safeguards are tested independently.

Why agentic AI changes the governance boundary

The article uses agentic AI in the broader business sense, but the governance issue only changes materially when the system can choose actions at runtime, select tools, and execute without a human approval gate. At that point, the chatbot is no longer just a conversational interface. It becomes an identity-bearing actor whose behaviour can expand or contract within a session. That shifts the problem from static access management to runtime control of tool use, session scope, and delegated authority. Even when the model is not fully autonomous, the control surface is larger than traditional IAM expects.

Practical implication: if the chatbot can act, not just answer, treat its runtime behaviour as an identity governance boundary and review the approval model accordingly.

Threat narrative

Attacker objective: The attacker aims to extract sensitive information, manipulate responses, or use the chatbot’s access path to reach business systems and data that should remain constrained.

1. Entry occurs when an attacker interacts with a business chatbot through public channels, a compromised integration, or crafted prompts that reach connected tools and data sources.
2. Escalation happens when the bot’s inherited permissions, retrieval layer, or workflow hooks expose more information or function than the user should be able to reach.
3. Impact follows when the system leaks sensitive data, misguides customers, or triggers unsafe business actions at scale across support, sales, or internal workflows.

Breaches seen in the wild

• DeepSeek breach — DeepSeek breach exposed 1M+ log lines and sensitive secret keys.
• Cisco DevHub NHI breach — IntelBroker exploited exposed Cisco credentials, API tokens and keys in DevHub.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI chatbots in business create an identity problem before they create an AI problem. The core issue is not whether the chatbot can hold a conversation, but which identity it inherits and which data paths that identity opens. When the bot sits in front of CRM, HR, or support systems, the governance question becomes who can act through it and under what bounds. Organisations should treat chatbot access as part of identity architecture, not as a UI feature.

Standing trust is the wrong assumption for modern chatbot deployments. Traditional IAM often assumes a user or service account is provisioned once and then governed through review cycles. Business chatbots break that assumption because their access is often spread across prompts, connectors, and vendor-managed services that are hard to see in one place. The result is access that looks temporary to the business but behaves persistently in the stack. Practitioners should recognise this as hidden identity sprawl.

Model errors and abuse paths must be governed separately. A chatbot that hallucinates and a chatbot that is jailbroken create different failure modes, even if the symptoms both look like bad answers. One is a trust problem in output quality, the other is an access-control problem in how the system is constrained. Security programmes that collapse those two risks into one control family tend to miss both. The practical lesson is to separate content safety, retrieval integrity, and permission governance.

What matters now is the control boundary around delegated action. If a chatbot only responds, standard IAM and data governance may be enough. Once it can retrieve, recommend, or act across business systems, the organisation must define where human approval ends and machine-influenced execution begins. That boundary should be explicit in policy, logging, and exception handling. Practitioners should not wait for a high-profile failure to redraw it.

From our research:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• For lifecycle and rotation context, review NHI Lifecycle Management Guide and align chatbot-adjacent credentials to the same operational discipline.

What this signals

AI chatbot programmes will keep expanding faster than most identity teams can model them. With 32.4% of security budgets already going to secrets management and code security in the reference research, the lesson is that access-path control is becoming a mainstream operational cost, not a niche control problem. Teams should expect chatbot governance to land in the same budget conversation as workload identity and secrets management.

Hidden identity sprawl is the most likely failure mode. A chatbot can be one interface while hiding several accounts, tokens, and retrieval systems underneath. That is why the governance conversation should shift from the visible bot to the invisible access chain it depends on, including how that chain is reviewed, revoked, and monitored.

Business teams need to decide now whether chatbots are only conversational or partly delegated. Once a system can retrieve records or trigger actions, the organisation is no longer managing a simple support channel. It is managing a controlled identity path that should be reviewed with the same seriousness as any other privileged integration.

For practitioners

• Inventory every chatbot identity and connector Map each AI chatbot to the account, service principal, API token, or vendor integration it uses, then list the exact systems and datasets that identity can touch. Remove unused connectors and prohibit shared credentials across customer-facing and internal workflows.
• Split abuse controls from answer-quality controls Test prompt-injection, jailbreak resistance, and tool restrictions separately from retrieval accuracy, hallucination rates, and user-output validation. A single control that claims to cover both usually leaves gaps in one of them.
• Limit delegated actions to explicit approval points Require human approval before the chatbot can create records, change customer data, open tickets, or trigger downstream workflows. If the system can act on behalf of staff or customers, put those actions behind logged, reviewable decision gates.
• Log data access at the identity layer Capture which identity accessed which record, which tool was used, and which response was returned so investigations can distinguish user behaviour from model behaviour. Without that separation, security teams cannot tell whether the failure came from access, retrieval, or generation.

Key takeaways

• AI chatbots in business introduce identity and data-access risk even when they appear to be simple support tools.
• The most important control question is not what the chatbot can say, but which identities, connectors, and data paths it can reach.
• Organisations should separate conversational safety, hallucination control, and delegated-action governance instead of treating them as one risk.

Key terms

• Chatbot identity: The identity used to connect a conversational AI system to enterprise data and services. It may be a service account, API token, or delegated integration. In practice, the chatbot inherits the permissions of that identity, so governance must focus on scope, logging, and revocation rather than the chat interface alone.
• Delegated action: A downstream operation that an AI system can trigger on behalf of a person or team, such as creating a ticket, updating a record, or retrieving restricted data. The risk is not only what the model says, but what it is allowed to do after it says it.
• Jailbreak: A prompt or interaction pattern that pushes a model to ignore its intended safety or policy boundaries. In business chatbots, jailbreaks matter because they can expose hidden data, bypass output restrictions, or induce the system to use connected tools in unsafe ways.
• Hallucination: An AI-generated response that is fluent and plausible but incorrect, unsupported, or fabricated. For identity and governance teams, hallucination is a control issue because users may act on it as if it were trusted system output, especially when the chatbot sits inside an operational workflow.

What's in the full article

Lakera's full article covers the operational detail this post intentionally leaves for the source:

• Detailed use-case examples across banking, healthcare, legal services, and retail.
• Specific product-style claims and implementation context around customer-service and business-process automation.
• The article's own list of chatbot benefits and risks, including productivity, multilingual support, and AI bias examples.
• The broader marketing context around Lakera's GenAI security platform positioning.

👉 Lakera's full article covers the chatbot use cases, risks, and business examples in more detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.