How should security teams govern foundation models used in production AI applications?

Why This Matters for Security Teams

Foundation models change the governance problem because they are not just another application library. Once a model is embedded in production workflows, it can shape decisions, generate content, call tools, and expose sensitive data through prompts or outputs. That means the risk surface includes model provenance, training data quality, access paths, and the controls around every downstream system that consumes the model’s output. Governance has to start before deployment and continue through ongoing monitoring, review, and change management.

For security leaders, the key mistake is assuming the model is trustworthy because the application around it is authenticated. A model can still be manipulated through prompt injection, data poisoning, weak retrieval sources, or overly broad tool permissions. Current guidance suggests treating these systems as shared infrastructure that needs explicit ownership, logging, and review gates. The governance model should also define who can approve model changes, who can disable a risky integration, and how incident response works when outputs become unsafe or misleading.

That operational view fits well with the NIST Cybersecurity Framework 2.0, but AI-specific controls need to be layered on top rather than assumed. In practice, many security teams encounter model risk only after a production workflow has already leaked data or automated a bad decision, rather than through intentional model governance.

How It Works in Practice

Effective governance starts by defining the model’s approved use case, data boundaries, and acceptable failure modes. Security teams should require a review of where the model is sourced from, how it was trained or fine-tuned, what telemetry is available, and which applications can call it. If the model is accessed through an API, the security team should still apply identity controls, request throttling, secret management, and separation between development and production environments.

Operationally, governance usually spans four layers:

• Model layer: provenance, versioning, evaluation results, and change approval.
• Data layer: what data can be sent to the model, retained by the provider, or reused for training.
• Runtime layer: prompt filtering, output checks, logging, and abuse detection.
• Integration layer: tool permissions, downstream action limits, and human approval for high-impact actions.

Security teams should also establish red teaming and validation before release, then repeat testing after major prompt, retrieval, or model updates. Where the application uses retrieval-augmented generation, the governance boundary must include source document quality and access control, because the model often reflects the integrity of the retrieved content rather than just the base model. NIST’s NIST AI 600-1 Generative AI Profile is useful here because it translates AI risk management into controls that can be operationalised in production settings.

These controls tend to break down when teams connect the model to internal tools with broad write permissions, because a single compromised prompt can trigger real-world actions faster than existing approval processes can stop them.

Common Variations and Edge Cases

Tighter model governance often increases friction for product teams, requiring organisations to balance speed of deployment against safety, auditability, and business agility. That tradeoff becomes sharper when the model is used across multiple business units, because one team may want experimentation while another needs strict change control and evidence for audit.

There is no universal standard for every governance detail yet, especially for model provenance attestations, prompt logging depth, and how much output should be retained for review. Best practice is evolving. In high-regulation environments, the answer is usually to bias toward stronger documentation, stricter approval gates, and clear escalation paths when model behavior changes. In lower-risk use cases, teams may accept lighter controls, but only if the model cannot take actions, access sensitive data, or influence regulated decisions.

The identity intersection matters most when foundation models are given agent-like execution authority. At that point, governance should include privileged access review, credential scoping, and explicit control over which non-human identities can invoke tools or write to systems. If that link is not controlled, the model becomes a hidden operator rather than a supervised service.

Related resources from NHI Mgmt Group

• How should security teams govern LLM outputs in production AI applications?
• How should security teams govern access to premium AI models in production environments?
• How should security teams govern agentic AI environments when traditional posture tools only cover applications or models?
• How should security teams govern API keys used for generative AI access?