Public LLM use is mostly educational, not policy-breaking

At a glance

What this is: Knostic reports that publicly shared ChatGPT conversations were overwhelmingly educational and rarely policy-breaking, with only a small fraction containing sensitive material.

Why it matters: For IAM and AI governance teams, the finding shows that visibility changes behaviour, but it also reinforces the need to govern private enterprise AI use, data exposure, and acceptable-use boundaries.

By the numbers:

• 99.06% of analyzed conversations contained no policy violations.
• Over 80% of the analyzed conversations centered on educational topics.
• Only 0.94% contained sensitive material.

Context

Public LLM use creates a governance question that is easy to overstate in one direction and understate in the other. The issue is not whether all AI chats are unsafe, but which conditions actually change user behaviour, data exposure, and policy adherence. In this study, the relevant lens is public visibility, because it alters both what people say and what they choose not to say.

For IAM, NHI, and AI governance teams, that distinction matters. A public conversation can look safer because the user is self-censoring, while a private enterprise deployment can still expose sensitive data, unmanaged prompts, or policy gaps. The article is therefore less a claim that AI is inherently safe than evidence that accountability shifts behaviour, which is typical of human-facing controls but not sufficient for enterprise governance.

Key questions

Q: How should organisations interpret public LLM chats when setting AI governance policy?

A: Public chats are useful behavioural evidence, but they are not a full risk baseline. They show how people act when visible, which usually reduces obvious misuse. Organisations should therefore use them to inform training and policy design, while basing controls on private prompts, internal data access, and enterprise integrations.

Q: Why do visible AI conversations often look safer than private enterprise use?

A: Visibility changes behaviour because users self-censor when they expect an audience or accountability. That reduces obvious policy-breaking in shared chats, but it does not change the underlying exposure created by private access to data, tools, and workflows. Enterprise risk is usually higher where observation is weaker.

Q: What are the signs that LLM use is being governed more by self-presentation than by policy?

A: The main sign is a low rate of visible misuse paired with weak controls over private prompts, connected data, and user attribution. If teams can only assess public conversations, they are measuring social restraint, not governance maturity. Real governance shows up in access logging, data scoping, and enforcement.

Q: Should security teams trust public LLM studies to decide whether their AI controls are adequate?

A: No. Public studies are useful for understanding user behaviour, but they do not measure access to sensitive systems, enterprise data, or workflow automation. Teams should use them as directional context, then test controls against the environments where prompts, files, and permissions actually intersect.

Technical breakdown

Why public visibility changes LLM behaviour

Public sharing changes the operating context of an LLM interaction. When users know a conversation may be seen by others, they tend to edit for reputation, clarity, and social acceptability. That means the output set reflects both the model and the user’s self-presentation. In governance terms, the dataset is not a random sample of all AI use. It is a filtered interaction surface shaped by social pressure, which can reduce overt abuse without removing underlying risk in private or embedded deployments.

Practical implication: Treat public-chat datasets as behavioural signals, not as proof that enterprise LLM usage is safe.

What the 99.06% figure does and does not prove

A low policy-violation rate shows that visible misuse is uncommon in the sample, but it does not prove that policy-breaking is rare everywhere. Publicly shared conversations already exclude much of the private, operational, or sensitive work that employees are more likely to keep internal. The result is therefore useful for calibrating threat perception, not for downgrading control requirements. Governance has to account for the difference between observed behaviour and the full population of prompts, responses, and connected data sources.

Practical implication: Use the metric to challenge fear-driven assumptions, but keep controls focused on private and enterprise-facing AI workflows.

Why LLM safety is partly an identity and trust problem

This article intersects with identity governance because prompt behaviour is not only a model issue, it is also a user accountability issue. People behave differently when they expect to be identifiable, observed, or socially judged. That is similar to how access controls influence human conduct in IAM programmes: controls change incentives, not just permissions. For AI programmes, the same logic applies to logging, attribution, and policy enforcement around who can prompt, share, export, or connect sensitive data to tools.

Practical implication: Build AI governance around attribution, visibility, and access boundaries, not only around content filters.

Threat narrative

Attacker objective: The objective in the outlier cases was to bypass model safeguards or elicit sensitive responses, but the study found that such outcomes were uncommon.

1. Entry occurred through voluntarily shared public ChatGPT conversations, which created a visible sample of user behaviour rather than a covert attack path.
2. Credential or policy abuse was rare, with only one successful jailbreak attempt identified across the full dataset.
3. Impact was limited to rare sensitive disclosures and isolated policy circumvention cases, showing that public visibility can constrain misuse without eliminating it.

Breaches seen in the wild

• Samsung ChatGPT leak 2023: Samsung staff pasted chip source code and meeting notes into ChatGPT weeks after it was allowed, leading Samsung to restrict generative AI tools.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Public visibility is a control, but not a control plane: The study shows that being observed changes user behaviour, which is useful but limited. Transparency can suppress obvious misuse, yet enterprise AI governance cannot depend on self-policing alone. Organisations still need policy enforcement, logging, and data-boundary controls because private use is where exposure risk usually concentrates.

The real governance gap is between public caution and private workflows: The article’s strongest signal is not that AI is safe, but that safety claims based on public samples can mislead decision-makers. Private prompts, integrated copilots, and internal search surfaces create different risk conditions. Practitioners should assume the public sample underestimates the control burden in actual production environments.

Human self-presentation is now part of AI risk management: Users adapt to the presence of an audience, which means behaviour changes when accountability increases. That is a named governance concept worth carrying into AI policy design: visibility-driven restraint. It means the programme should measure how disclosure, logging, and identity attribution affect prompt hygiene, not just how filters classify output.

LLM governance should separate model safety from data governance: The article mixes model misuse, user intent, and sensitive-data exposure, but those are different control problems. Enterprises often over-focus on prompt toxicity and under-focus on what data the model can reach. The practical conclusion is straightforward: data access, not only model behaviour, determines whether an AI deployment becomes risky.

Public LLM research should not be mistaken for enterprise assurance: A dataset of voluntarily shared chats is valuable for understanding social behaviour, but it is not a proxy for internal access patterns, regulated-data handling, or agentic tool use. Security teams should treat it as a directional indicator only. The decision point is whether the organisation has governance over what the model can see and who can use it.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Visibility-driven restraint: public AI conversations show that accountability changes user behaviour, but that effect weakens once prompts move into private enterprise workflows. Security teams should therefore treat transparency as a helpful pressure on conduct, not as a substitute for access control or data governance.

The stronger programme signal is that LLM risk now sits at the intersection of identity, data access, and user attribution. If you cannot answer who prompted the model, what it could reach, and what it was allowed to reveal, you do not have governance, only observation.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. That context matters because the real control problem is usually the surrounding stack, not the public chat itself.

For practitioners

• Separate public behaviour from enterprise exposure Use public-chat studies as context for policy design, but validate controls against internal prompts, connected repositories, and workflow integrations.
• Add identity-aware logging for AI usage Record who initiated prompts, what data sources were reachable, and which permissions were exercised so accountability is tied to actual access.
• Restrict sensitive data paths into LLM tools Apply least-privilege access to copilots, search tools, and chat integrations so the model cannot surface regulated or confidential content by default.
• Measure policy adherence in private workflows Audit internal AI use for oversharing, prompt injection exposure, and unapproved data export, then compare it with public-chat behaviour to find the real gap.

Key takeaways

• Public LLM conversations can look reassuring because visibility changes how people behave, but that does not prove enterprise AI use is safe.
• The most important governance gap is the difference between public self-censorship and private access to data, tools, and workflows.
• For practitioners, the priority is to govern who can prompt, what the model can reach, and how AI activity is attributed and audited.

Key terms

• Public LLM Conversation: A publicly shared interaction with a large language model, often exported or posted by a user for others to see. These conversations are useful for studying user behaviour, but they are not a complete proxy for enterprise AI risk because private prompts and internal data access are usually absent.
• Policy violation: An AI response or action that crosses an organisation’s defined behavioural boundary, such as unsafe guidance, disallowed disclosure, or instructions that contradict operating rules. In practice, it is a governance failure as much as a model failure because it indicates enforcement did not hold.
• Jailbreak: A prompt or input transformation that bypasses a language model’s safety restrictions and causes it to produce output it would normally refuse. In operational settings, a jailbreak matters because the model may be embedded in a workflow, making the bypass a pathway to broader misuse, not just a bad response.
• Visibility-Driven Restraint: A behavioural effect where people become more careful, polished, or policy-aware when they expect their AI interactions to be seen by others. It is a governance-relevant signal because transparency can reduce obvious misuse while leaving private exposure, access control, and data leakage risks unchanged.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and AI agent identity. It helps security and identity practitioners align governance, access, and accountability across modern AI-enabled environments.