Public LLM use is mostly educational, not policy-breaking

At a glance

What this is: Knostic’s analysis of 13,455 public LLM conversations finds that public AI use is overwhelmingly educational, with 99.06% of chats showing no policy violations.

Why it matters: For IAM, NHI, and AI governance teams, the finding underscores that user behaviour, disclosure context, and access policy boundaries all shape how AI systems expose data and risk.

By the numbers:

• 99.06% of analyzed conversations contained no policy violations
• 80.1% of interactions involved educational or self-development topics
• 0.94% contained sensitive material

👉 Read Knostic's analysis of public LLM use, safety, and educational behavior

Context

Public LLM conversations are a useful lens on how people behave when they know a system or transcript may be visible to others. In this case, the primary security question is not whether AI is inherently unsafe, but how visibility, incentives, and policy boundaries affect what users choose to reveal. That has direct relevance for enterprise AI governance, where prompt content, retention, and access control determine whether a harmless interaction stays harmless.

The article sits at the intersection of AI governance and identity-adjacent risk because disclosure is shaped by accountability. When users self-censor in public, the system may look safer than it would in private enterprise workflows, but the governance lesson remains the same: visibility does not replace control. Teams responsible for AI access, data handling, and sensitive information boundaries still need policy, auditability, and enforceable need-to-know controls.

Key questions

Q: How should security teams govern AI assistants that can access enterprise data?

A: Security teams should govern AI assistants through least privilege, explicit data-scoping, and auditability. The key question is not only whether the model is safe to use, but whether the connected identities, tokens, and retrieval paths can expose more data than the user is allowed to see. Policy must cover prompts, connectors, and downstream sharing.

Q: Why do public LLM conversations understate enterprise AI risk?

A: Public conversations understate enterprise risk because users change behaviour when they know others may see the transcript. They ask safer questions, reveal less sensitive context, and avoid obvious misuse. Private enterprise workflows remove that social pressure, so the true control test is how the system behaves when visibility is absent.

Q: What breaks when AI access is governed only at the prompt layer?

A: Prompt-layer governance breaks when the model can still retrieve, summarise, or expose restricted data through connected systems. A compliant prompt does not prevent a privileged connector from returning sensitive content. Effective control requires data-access boundaries, identity governance, and logging across the full AI workflow.

Q: How do organisations know whether AI data controls are actually working?

A: They know controls are working when AI systems cannot retrieve data outside approved scopes, and when access reviews show that connectors, service identities, and OAuth grants match current business need. The best signal is not just blocked prompts, but consistently bounded retrieval and auditable access paths.

Technical breakdown

Public LLM transcripts and behavioural bias

Publicly shared chat transcripts are not neutral samples. They skew toward users who are willing to publish their interactions, which means the dataset captures behaviour under observation. That creates a strong selection effect: people are more likely to ask educational, polished, or socially acceptable questions when they expect others to see the result. In governance terms, this is similar to how logging and supervision change user behaviour in other systems. The finding does not prove all LLM use is safe, but it does show that transparency can suppress overt abuse and shift users toward safer interaction patterns.

Practical implication: treat public transcripts as indicator data, not a complete picture of enterprise AI risk.

Policy violations, jailbreaks, and model boundary testing

A policy violation occurs when a prompt or response crosses a platform rule, while a jailbreak is an attempt to bypass those rules through framing, role-play, or adversarial prompting. The rarity of successful jailbreaks in the study suggests that most public users are not actively trying to defeat safety systems. But rare does not mean irrelevant. In enterprise settings, a small number of boundary tests can still expose sensitive data handling gaps, especially when the model is connected to search, documents, or internal knowledge sources. The governance problem is therefore less about popularity of abuse and more about the consequences of the few cases that succeed.

Practical implication: focus testing on high-impact jailbreak outcomes, not just the frequency of attempted misuse.

AI data exposure is governed by access and context, not intent alone

The study points to a broader control issue in AI governance: content safety is only one layer. If a model can see or surface data that a user should not access, policy compliance in the conversation itself does not prevent exposure. That is where IAM-style concepts matter, especially least privilege, need-to-know, and auditability for AI-connected systems. Public behaviour may look disciplined, but enterprise risk is driven by whether the underlying AI environment respects data boundaries across prompts, retrieval, and downstream sharing. The security lesson is that safe user intent cannot compensate for overbroad data access.

Practical implication: enforce least-privilege access for AI-connected data sources and review what the model can retrieve.

NHI Mgmt Group analysis

Public AI safety is not the same as enterprise AI governance. The study shows that visible conversations are mostly educational and low risk, but that reflects user behaviour under observation rather than a complete risk model. Enterprise AI programs must govern both what users ask and what systems can access, because policy-compliant prompts can still expose restricted information through retrieval or sharing paths. The practitioner conclusion is that transcript safety is not a substitute for access governance.

Transparency creates safer behaviour, but it also masks private-use risk. When people know prompts may be shared, they curate their language and avoid obvious misuse. That effect is useful, but it means public conversation samples can understate what happens in private enterprise workflows, where users may feel less constrained. The practitioner conclusion is to validate controls against private and internal AI usage, not just public prompt datasets.

Need-to-know remains the right model for AI-connected data access. The article implicitly reinforces a core identity principle: if the AI system can retrieve more than the user should see, the conversation layer cannot fix that. This is where NHI governance intersects with AI governance, because service identities, connectors, and delegated tokens determine the blast radius of each prompt. The practitioner conclusion is to treat AI access paths like any other privileged integration and restrict them accordingly.

Prompt safety and data safety must be governed separately. A model can refuse harmful content and still expose sensitive information if the connected environment is over-permissioned. That separation is critical for AI search, copilots, and retrieval-augmented workflows, where the control plane may be compliant while the data plane remains overexposed. The practitioner conclusion is to assess both behavioural safety and information-access boundaries in the same programme.

Self-censorship is a weak security control, but it is a useful signal. The research suggests people behave more responsibly when they know they are visible, which may reduce obvious misuse in public contexts. But security programmes cannot depend on users behaving well, because control must survive when observation disappears. The practitioner conclusion is to use these findings as a signal for policy design, not as evidence that formal governance is unnecessary.

What this signals

Need-to-know for AI systems: the governance issue is not whether users behave responsibly in public, but whether connected identities and retrieval paths enforce the same boundary in private enterprise use. The closest control model is traditional IAM, extended to AI connectors, tokens, and delegated access. Teams should evaluate their AI programme against NIST AI Risk Management Framework and OWASP Agentic AI Top 10.

As enterprise AI adoption grows, prompt-level moderation will matter less than whether the underlying data plane can be bounded, audited, and revoked. That means identity teams need inventory of service accounts, OAuth grants, and retrieval connectors just as much as they need content policy review. The practical signal is simple: if an AI assistant can reach data the human requester cannot justify, the programme is already overexposed.

Public safety data can improve perception, but it should not change control priorities. Organisations should assume that visible conversations are the safest subset, then validate controls against less visible internal workflows where sensitive data, regulated content, and business context are more likely to appear.

For practitioners

• Map AI access paths to least privilege Inventory which AI chat, search, and retrieval systems can reach sensitive repositories, then reduce access to the minimum data sets required for each business use case. Prioritise connectors, delegated tokens, and service identities that can surface regulated or confidential content.
• Separate prompt safety from data access control Assess whether the model can refuse unsafe content while still retrieving restricted information. Build separate review checkpoints for behavioural policy enforcement and for information access, because one does not guarantee the other.
• Test private-use scenarios, not just public transcripts Run red-team exercises against internal AI workflows where users are not self-censoring. Focus on oversharing, indirect prompt leakage, and retrieval exposure through copilots, search tools, and connected documents.
• Audit delegated identities behind AI tools Review the service accounts, API keys, and OAuth grants that enable enterprise AI integrations. Confirm who issued them, what data they can reach, and whether those privileges are still justified.

Key takeaways

• Public LLM conversations look safer than much enterprise usage because people self-censor when transcripts may be visible.
• The real governance gap is not prompt quality alone, but whether AI-connected identities can retrieve data beyond the user’s authorised scope.
• Security teams should treat AI assistants as governed access paths, with least privilege, auditability, and separate controls for content safety and data exposure.

Key terms

• Public LLM transcript: A public LLM transcript is a conversation with a model that a user has intentionally shared or published. It is not a representative sample of all usage, because people often self-censor when they know others may see the content. That makes it useful for behavioural analysis, but limited for full risk assessment.
• Jailbreak: A jailbreak is an attempt to persuade or trick an AI system into ignoring its safety rules or policy boundaries. It often relies on role-play, indirect instructions, or framing changes. In governance terms, jailbreaks are important because even rare successes can expose gaps in content controls or connected data access.
• AI data access boundary: An AI data access boundary is the limit on what information a model or connected tool is permitted to retrieve, summarise, or expose. It depends on identity, policy, and connector design, not just on the model’s content filters. If the boundary is too broad, safe prompting still leaves exposure risk.
• Need-to-know: Need-to-know is an access principle that limits data exposure to the minimum required for a task. In AI systems, it applies not only to the human user but also to connectors, service identities, and retrieval workflows. It is one of the clearest ways to reduce overexposure in copilots and search tools.

What's in the full article

Knostic's full analysis covers the operational detail this post intentionally leaves for the source:

• Methodology notes on how the 13,455 public conversations were sampled and reviewed.
• The review rubric used to classify educational prompts, sensitive material, and jailbreak attempts.
• Examples of rare outlier prompts that reveal model boundary behaviour in practice.
• Additional discussion of why public visibility changes user behaviour and how that affects interpretation.

👉 Knostic's full article includes the methodology, outlier examples, and the underlying conversation analysis.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and workload identity. It helps security and identity teams apply stronger access control thinking to AI-connected systems, service identities, and delegated access paths.