AI service privacy is weaker than enterprise teams assume

At a glance

What this is: This blog argues that enterprise AI service chats are not private vaults, and that compliance access, oversharing, and stored conversation history can expose secrets and sensitive data.

Why it matters: IAM, NHI, and governance teams need to treat AI workspaces as shared identity and data surfaces, because privileged access to chat systems can become a secret-exposure pathway across human, machine, and agentic programmes.

By the numbers:

• 44% of NHI tokens are exposed in the wild, being sent or stored over platforms like Teams, Jira tickets, Confluence pages, and code commits.
• 28% of secrets incidents now originate outside code repositories, in Slack, Jira, and Confluence, and are 13% more likely to be categorised as critical than code-based leaks.
• 64% of valid secrets leaked in 2022 are still valid and exploitable today, proving that detection alone is not enough without automated revocation.

👉 Read Token Security's analysis of privacy misconceptions in AI services

Context

AI service privacy becomes an identity and secrets problem the moment employees treat a chat interface as a safe place to paste credentials, internal code, or sensitive business context. The article argues that “not used for training” is being misread as confidentiality, when the real issue is that centrally stored conversations can still be accessed by privileged administrators or attackers.

That matters for NHI governance because the exposed material is not just personal content. It includes API keys, tokens, SSH keys, and other secrets that can be reused to move from a chat system into cloud services, repositories, or databases. In practice, the AI workspace becomes another discoverable store of non-human identity material.

Key questions

Q: How should security teams handle secrets pasted into enterprise AI chats?

A: Treat them as exposed credentials, not as private notes. Search exported conversations and uploaded files for API keys, tokens, SSH keys, and internal endpoints, then revoke or rotate anything active. Also restrict who can retrieve workspace history, because administrative access can become a parallel secrets-discovery channel.

Q: Why do enterprise AI services create extra risk for NHI governance?

A: They concentrate high-value secrets, sensitive files, and conversation history behind a small set of privileged admin controls. If those controls are over-broad or compromised, the attacker gets a large blast radius and can reuse exposed machine credentials across cloud, code, and identity systems.

Q: What do organisations get wrong about AI chat privacy?

A: They confuse “not used for training” with confidentiality. That label may address model training policy, but it does not remove storage, indexing, retention, or admin access. Practically, any sensitive content entered into the workspace should be assumed retrievable by someone with the right privileges.

Q: Who should be accountable for secrets exposed in AI workspaces?

A: Accountability should sit jointly with IAM, security operations, and the business owners of the workspace. IAM governs privileged access, security teams handle detection and revocation, and business leaders set acceptable-use rules. If the platform stores or reveals secrets, the governance gap is organisational, not just technical.

Technical breakdown

Why “not used for training” is not a confidentiality guarantee

The article draws a hard line between model training policy and data confidentiality. A platform can promise not to use conversations for training while still storing messages, files, metadata, and usage records for administrative and compliance purposes. That means the security question is not whether the model learns from the data, but who can retrieve it and under what permissions. In identity terms, this is a broad-visibility problem: once privileged access exists, the workspace becomes a searchable repository of sensitive content rather than a transient conversation layer.

Practical implication: security teams should classify AI chat systems as governed data stores, not informal collaboration tools.

How compliance APIs expand the attack surface

The OpenAI Compliance API described in the article exposes workspace-wide history, shared files, uploaded documents, user activity, and code snippets to administrators. That design supports retention and audit use cases, but it also concentrates control into a high-value administrative key. If that key is stolen, over-permissioned, or misused internally, the attacker does not need to break the platform’s core defenses; they can query legitimate endpoints and enumerate the workspace. This is an identity problem because the authority to inspect content is itself a privileged credential.

Practical implication: restrict administrative API access, monitor it like a privileged account, and rotate it as a high-risk secret.

How secrets move from chat to lateral movement

Once an attacker can export conversations and files, secret scanning becomes straightforward. The harvested material often includes cloud keys, database strings, SSH keys, and internal endpoints that can be replayed against production systems. That turns a collaboration-layer exposure into a broader compromise path: cloud access, code repositories, and identity providers can all become downstream targets. The technical lesson is that chat systems now sit inside the same credential lifecycle as any other secret-bearing platform, because users routinely deposit operational secrets there.

Practical implication: treat AI chat exports as a secrets discovery source and feed findings into revocation and access review workflows.

Threat narrative

Attacker objective: The attacker aims to turn centrally stored AI workspace content into reusable access for broader infrastructure compromise, data theft, and privilege escalation.

1. Entry occurs when an attacker obtains an OpenAI Compliance API key through a leaked GitHub secret, stolen admin credentials, or another compromise path.
2. Credential access follows through legitimate compliance endpoints that enumerate users, conversations, files, and metadata across the workspace.
3. Impact comes when exposed keys, tokens, and internal details are reused to pivot into cloud services, repositories, databases, or identity systems.

Breaches seen in the wild

• IOS app secrets leakage report — iOS apps leaking hardcoded secrets and credentials endangering user privacy.
• MongoBleed breach — MongoBleed exposed secrets across 87K MongoDB servers.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI workspace privacy is really a secrets-governance problem. The article’s core finding is that employees are treating chat systems as private spaces when the underlying control model is closer to a governed repository. That shifts the risk from “what the model remembers” to “who can retrieve stored content and reuse the secrets inside it.” For practitioners, the lesson is to govern AI workspaces as identity-bound data surfaces, not as casual productivity tools.

Centralized oversight creates a privileged access concentration point. The Compliance API is not failing to do its job. The failure mode is that compliance visibility and security exposure share the same administrative plane, which turns one key into workspace-wide authority. That is a classic NHI lesson applied to AI services: broad retrieval permissions amplify blast radius when the secret itself is the access path.

Secret sprawl inside AI chats is an identity lifecycle problem, not just a user behaviour issue. Secrets pasted into conversations often outlive the moment of use, then persist in searchable storage, exports, and audit trails. That means the governance assumption of “the secret only existed briefly in the session” no longer holds. The implication is that lifecycle controls must reach beyond source systems into collaboration and AI platforms where credentials are informally duplicated.

Ephemeral trust cannot be assumed in AI-enabled workspaces. The article shows that “private” enterprise chat is still a durable record with administrative visibility, so the trust boundary is not the conversation window but the storage and access layer behind it. That creates a governance gap across human IAM, NHI hygiene, and privileged admin control. Practitioners should treat every AI workspace as a potential discovery source for machine and human credentials alike.

From our research:

• 91% of former employee tokens remain active after offboarding, leaving organisations vulnerable to potential security breaches, according to The 2025 State of NHIs and Secrets in Cybersecurity.
• Our research also found that 62% of all secrets are duplicated and stored in multiple locations, which increases accidental exposure risk and makes revocation harder to complete cleanly.
• For a broader control lens, see Guide to the Secret Sprawl Challenge for how sprawl and duplication change the governance model.

What this signals

Secret sprawl now extends into AI collaboration layers, not just repositories. With 28% of secrets incidents originating outside code repositories and being 13% more likely to be critical, AI chat platforms should be folded into the same detection and revocation workflows used for Slack, Jira, and Confluence. The governance gap is no longer where secrets are created, but where staff casually store and reuse them.

Ephemeral trust debt is the right way to think about AI workspace exposure. If users keep pasting credentials into chat systems, the organisation accumulates access debt that survives the session and often survives the user. That means secret scanning, administrative access review, and offboarding must extend into AI platforms as part of ordinary lifecycle governance.

Programmes that already track workload identity and secrets rotation should extend those controls to collaboration surfaces that now behave like credential repositories. The next failure is likely to come from normal productivity behaviour rather than exotic compromise.

For practitioners

• Classify AI chat systems as governed data stores Map chat histories, file uploads, and compliance exports into your data classification and retention model so AI conversations are handled like email, Slack, and other durable records.
• Restrict compliance API access to a small privileged set Treat administrative keys for workspace-wide retrieval as high-risk secrets, enforce least privilege, and review all usage as part of privileged access management.
• Scan AI exports for live credentials Add conversation exports and uploaded files to secret-detection workflows so leaked API keys, tokens, and SSH credentials can be revoked before reuse.
• Update user guidance for safe prompt hygiene Tell staff to replace raw secrets with placeholders and to route sensitive operational details through approved secrets management and secure collaboration paths.
• Separate personal content from corporate workspaces Set explicit rules that personal, health, and job-search material must stay out of enterprise AI chats because retained content can be retrievable by administrators.

Key takeaways

• Enterprise AI chats become identity risks when users store secrets there and admins can retrieve them through broad compliance controls.
• The evidence points to a compounding exposure pattern: secrets move from conversations into cloud, code, and identity systems, where they can be reused quickly.
• Security teams should govern AI workspaces like durable data stores, with secret detection, privileged access limits, and revocation workflows built in.

Key terms

• Compliance API: An administrative interface that exposes stored workspace content for oversight, retention, and audit. In practice, it can become a high-value retrieval path because it centralises access to conversations, files, metadata, and activity logs under privileged credentials.
• Secret sprawl: The uncontrolled duplication of credentials across chats, tickets, repositories, documents, and other collaboration surfaces. It increases the number of places an attacker can find a reusable secret and makes revocation slower, harder, and more error-prone.
• Workspace-wide visibility: A permission model where a single administrative role can inspect large amounts of user content across an environment. It is useful for compliance, but it also expands blast radius if the credential is stolen, misused, or over-assigned.

What's in the full article

Token Security's full blog covers the operational detail this post intentionally leaves for the source:

• The step-by-step compliance API attack path, including the exact endpoints used to enumerate workspace users, conversations, and files
• Concrete examples of the secret types found in chat exports, including AWS keys, DB connection strings, API tokens, SSH keys, and passwords
• The article's recommended handling pattern for enterprise AI workspaces, including safe secret placeholders and admin access limits
• The vendor's explanation of how stored chat data can become searchable evidence and a downstream compromise source

👉 Token Security's full blog details the compliance API attack path, the secret types exposed, and the practical handling recommendations.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.