Why do AI chatbots and homegrown GenAI apps create new compliance risk for organisations?

Why This Matters for Security Teams

AI chatbots and homegrown GenAI apps expand the attack surface because they sit outside the control patterns most organisations use for files, email, and endpoint monitoring. Employees can paste regulated content into prompts, and models may echo that content back in responses, summaries, or logs. That creates compliance exposure under privacy, payment, and health rules even when the original system of record remains intact. Current guidance from the NIST AI 600-1 GenAI Profile treats prompt and response handling as a governance issue, not just a model-risk issue.

The problem is usually not malicious intent. It is uncontrolled adoption. Teams spin up copilots, internal chat portals, and workflow assistants faster than legal review, data classification, and retention controls can keep up. NHIMG research on the Top 10 NHI Issues shows how often identity and secret handling failures become operational incidents once automation is allowed to spread. In practice, many security teams encounter compliance leakage only after sensitive prompts have already been stored, replayed, or shared through an AI tool, rather than through intentional data governance.

How It Works in Practice

Compliance risk emerges when the chatbot becomes a new processing channel for regulated data. A user may paste card data into a support assistant, PHI into a drafting tool, or customer records into a summariser. The model does not need to “understand” the regulation for the organisation to be exposed. If prompts, embeddings, transcripts, plugins, or analytics logs retain that material, the business may have created a regulated copy outside approved systems. That is why NIST’s NIST Cybersecurity Framework 2.0 and NIST SP 800-53 Rev 5 Security and Privacy Controls both translate well to GenAI governance: identify data flows, restrict access, log appropriately, and retain only what is justified.

For homegrown apps, the risk grows because developers often add model calls without a full control plane. Common failure points include:

• no prompt redaction before transmission to external model endpoints;
• no role-based separation between ordinary users and privileged internal data sources;
• no policy check before the model retrieves records or tools invoke downstream actions;
• no retention rule for prompts, responses, and traces;
• no review of whether the AI vendor or platform becomes a processor or subprocessor under privacy law.

NHIMG’s Ultimate Guide to NHIs — Regulatory and Audit Perspectives is useful here because the same discipline used for non-human identities applies to GenAI services: know what is authenticating, what it can reach, and what evidence proves control. The challenge is that GenAI systems can generate regulated content at runtime even when no single user intended to create a record. These controls tend to break down when shadow AI tools are approved informally and then connected to production data sources without a data protection review.

Common Variations and Edge Cases

Tighter AI controls often increase friction for employees, requiring organisations to balance usability against compliance assurance. That tradeoff is especially visible in environments where rapid drafting, customer support, or software engineering depend on conversational tools. Current guidance suggests that there is no universal standard for every model deployment yet, so policy should reflect the data class, business purpose, and whether the tool is internal, hosted, or embedded in a customer-facing workflow.

Edge cases matter. A low-risk brainstorming bot may be acceptable with minimal logging, while a procurement assistant that sees supplier bank details needs much stronger filtering, audit trails, and approval gates. The same is true for fine-tuned internal models: if training or retrieval uses regulated content, then the compliance question is not only what the model outputs, but what it ingests and stores. NHIMG’s Ultimate Guide to NHIs — Lifecycle Processes for Managing NHIs is relevant because lifecycle governance must include onboarding, change control, monitoring, and retirement for AI-enabled services too.

For organisations dealing with highly sensitive data, the safest pattern is to treat every chatbot as a regulated processing boundary until proven otherwise. That is the practical lesson from both the standards side and NHIMG incident research, including the DeepSeek breach: visibility into what enters the model matters as much as what comes out.

Related resources from NHI Mgmt Group

• Why do AI agents create new compliance risk when organisations scale them across business functions?
• Why do sanctioned apps with new AI features create governance risk for organisations?
• Why do AI tools create new compliance risk for financial data access?
• Why do chat-based AI systems create new identity risk for organisations?