How should security teams implement AI governance in environments where developers use public LLMs and internal data sources?

Why This Matters for Security Teams

When developers use public LLMs alongside internal data sources, the risk is rarely limited to one bad prompt. Sensitive content can be copied into external systems, outputs can be reused without review, and tool-connected agents can turn a harmless query into an unauthorized data path. Governance has to cover data flow, access, logging, and model use at the point of interaction, which is why the NIST AI Risk Management Framework is useful as a governance baseline rather than a narrow AI policy checklist.

The practical problem is that public LLMs often sit outside traditional security controls, while internal data sources may already be classified and protected. Teams commonly focus on acceptable-use rules, but that does not stop prompt injection, data exfiltration through pasted context, or indirect leakage via model outputs. Security teams need to define what data can be used, where it can travel, who can invoke the model, and how interactions are retained for review. In practice, many security teams encounter leakage only after a developer has already pasted confidential material into a public model and the data has left the trusted boundary.

How It Works in Practice

Effective AI governance in this environment starts with control at the edge of use, not only in policy. Security teams should classify approved use cases, define data handling rules for each class, and route LLM access through approved gateways that can inspect prompts, redact secrets, and block disallowed content. That includes API keys, session tokens, customer records, source code fragments, and regulated data. Governance should also distinguish between experimentation in sandboxes and production use, because the acceptable risk profile is not the same.

Operationally, the strongest pattern is to combine identity, data, and monitoring controls:

• Require authenticated access and role-based restrictions for every LLM endpoint and connected tool.
• Mask or tokenize sensitive fields before prompts reach a public model.
• Block uploads and paste actions that would move restricted data outside approved environments.
• Log prompts, responses, model identifiers, and connected data sources for audit and incident response.
• Review outputs before they are used in code, decisions, or customer-facing workflows.

This is aligned with the broader control logic in the NIST Cybersecurity Framework 2.0, especially where governance, access control, and monitoring need to operate together. For AI-specific risk management, the NIST AI 600-1 Generative AI Profile helps translate model risk into concrete operating practices, while the OWASP Agentic AI Top 10 is valuable when LLMs are connected to tools or agents that can act on behalf of users. These controls tend to break down when developers bypass approved gateways and use public chat interfaces directly, because security teams lose visibility before data leaves the environment.

Common Variations and Edge Cases

Tighter governance often increases friction for developers, so organisations have to balance velocity against the risk of uncontrolled data movement. Current guidance suggests that one-size-fits-all restrictions are rarely effective, especially where teams use different models, different trust levels, and different data classes.

For low-risk experimentation, organisations may allow public LLMs with heavily sanitized prompts and no internal context. For higher-risk workflows, best practice is evolving toward private hosting, retrieval controls, and policy enforcement at the application layer rather than relying on user behaviour. The main edge case is agentic workflows that combine public LLMs with internal search, ticketing, or code repositories. In those environments, the model may not just expose data; it may also trigger actions, so the boundary between governance and access control becomes much tighter. That is where guidance from the MITRE ATLAS adversarial AI threat matrix and the CSA MAESTRO agentic AI threat modeling framework becomes especially relevant, because it helps teams model abuse paths, tool misuse, and chained failures. There is no universal standard for this yet, so organisations should document their own risk thresholds, review them regularly, and treat public-model usage as a governed privilege rather than a default developer convenience.

Related resources from NHI Mgmt Group

• How should security teams implement an AI governance policy in environments where employees use multiple AI tools and personal accounts?
• How should security teams govern AI workflows that use multiple tools and data sources?
• Why do open-source AI environments create a data-governance challenge for security teams?
• How should security teams implement governance for AI agents that can read and act on payment data through MCP?