What do security teams get wrong about using public LLMs in enterprise workflows?

Why Security Teams Misjudge Public LLM Use in Enterprise Workflows

Public LLMs are often adopted as if they were simple productivity tools, but they behave more like high-impact processing layers that can reshape instructions, content, and code. The main failure is not just data leakage. It is assuming the model will reliably separate safe inputs from unsafe ones. NHI Management Group’s research on incidents such as the McKinsey AI platform breach and the DeepSeek breach shows how quickly trust erodes when AI systems are connected to real business data without boundaries.

Current guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 points in the same direction: public models need governance around context, provenance, and downstream use, not just content filters. In practice, many security teams discover the problem only after a prompt containing sensitive material has already been reused in a workflow, rather than through intentional review.

How Public LLMs Should Be Controlled in Practice

The practical mistake is treating a public LLM like a blank worksheet. A better model is to treat it as an untrusted transformation service that may expose, rewrite, or amplify what it sees. That means the first control is data classification. Anything regulated, confidential, or operationally sensitive should be excluded unless there is a documented approval path and a business justification.

Security teams should then separate read access, prompt access, and output consumption. Users may be allowed to ask general questions, but not to paste customer records, internal code, secrets, or incident details. This is especially important because a model can produce plausible but incorrect guidance, and teams may copy it into production without review. NIST’s AI 600-1 Generative AI Profile and CSA’s CSA MAESTRO agentic AI threat modeling framework both support this sort of context-aware risk handling.

• Restrict what can be sent to public models, especially secrets, tokens, and customer data.
• Require human review before AI-generated code or instructions are deployed.
• Log prompts and outputs so risky reuse can be traced and investigated.
• Use approved gateways or brokers rather than direct employee access where possible.

When public LLMs are used in code generation, summaries, customer support drafting, or research workflows, the safest posture is to assume the output may be useful but unverified. The OWASP NHI Top 10 is also relevant here because AI workflows often depend on the same credentials, tokens, and service identities that attackers target. These controls tend to break down when employees can move from chat to production systems without an approval step, because the model becomes a fast path from untrusted input to operational change.

Where the Standard Advice Breaks Down

Tighter LLM controls often increase friction, requiring organisations to balance productivity against visibility, review effort, and developer convenience. That tradeoff becomes sharp in fast-moving teams that want broad access to public models for brainstorming, coding, or support drafting. The best practice is evolving, but there is no universal standard for when a prompt becomes a record that must be retained, reviewed, or blocked.

One common edge case is “safe” synthetic use that quietly becomes unsafe when real examples are inserted into prompts. Another is vendor-managed copilots that inherit enterprise documents, tickets, or repositories and blur the line between convenience and data exposure. The 12,000 Secrets Found in Public LLM Training Dataset example is a reminder that sensitive material can reappear in unexpected places when model inputs and training sources are not tightly controlled.

For organisations with regulated data, the right answer is rarely “ban all public LLMs” or “allow everything.” A more durable approach is to define approved use cases, classify permissible inputs, and require output validation for anything that influences code, policy, or customer communications. In high-risk environments, even well-intentioned users can turn a public LLM into a data exfiltration path if controls are too coarse.

Related resources from NHI Mgmt Group

• What do security teams get wrong about using LLMs for exact calculations?
• What do security and operations teams get wrong about using LLMs for summaries?
• What do security teams get wrong about using LLMs for policy enforcement at scale?
• What do security teams get wrong about enterprise auth readiness?