A common mistake is focusing only on model selection or initial testing while ignoring runtime controls. Once a model is in production, prompt injection, jailbreaks, data leakage, and unsafe outputs become ongoing risks. Teams also underestimate the need for continuous monitoring, policy enforcement, and review of connected tools, because the surrounding system often creates the real exposure.
Why This Matters for Security Teams
Foundation models are often treated like static software after go-live, but that framing misses the real risk surface. Production systems inherit new prompts, new users, connected tools, retrieval pipelines, and data flows that can change model behaviour without any code change. That makes post-deployment governance a security and resilience problem, not just a model quality problem. Guidance from the NIST AI 600-1 Generative AI Profile is useful here because it pushes teams to think about operational monitoring, misuse, and human oversight rather than one-time validation.
Teams most often get this wrong by assuming that pre-release red teaming, policy review, or a safe model selection decision is enough. In practice, the surrounding application layer creates the exposure: user prompts, retrieval content, external plugins, and downstream workflows can all produce outcomes that were never visible during testing. If those paths are not monitored, security teams lose visibility into how the model is actually being used. In practice, many security teams encounter unsafe model behaviour only after data has already been exposed or a business process has been influenced by a bad output, rather than through intentional monitoring.
How It Works in Practice
Securing a foundation model after deployment means controlling the full runtime path, not just the model endpoint. That includes who can prompt the model, what context it can retrieve, what tools it can invoke, and what outputs are allowed to leave the system. Strong deployments treat the model as one component in a larger control plane with logging, policy enforcement, and human escalation built around it.
Current guidance suggests four practical layers:
- Access control for prompts, tools, and administrative functions, with separate handling for internal users, customers, and agents.
- Input and output filtering to reduce prompt injection, leakage of sensitive data, and policy-violating responses.
- Telemetry for prompts, retrieved context, tool calls, refusals, and escalation events so investigators can reconstruct what happened.
- Change management for model versions, prompts, system instructions, and retrieval sources, because each can alter behaviour materially.
This is also where AI governance becomes operational. Security teams should define which outputs are advisory versus authoritative, when the model must defer to a human, and what constitutes an unacceptable response. If the system includes agentic workflows, the review bar should be higher because the model may act on its own output through tools or APIs. The NIST AI 600-1 Generative AI Profile helps anchor those controls to risk management language that governance teams can adopt without turning the model into an unbounded automation layer.
Security and platform teams should also test the full chain, not just the model. A clean model can still fail when retrieval content is poisoned, when a plugin over-privileges the agent, or when logging omits the exact prompt context needed for incident response. These controls tend to break down in highly dynamic environments with frequent prompt changes, weak tool governance, and no consistent owner for AI runtime policy because responsibility becomes fragmented across platform, product, and security teams.
Common Variations and Edge Cases
Tighter runtime control often increases latency, review overhead, and integration cost, so organisations have to balance safety against product velocity. That tradeoff becomes more visible as foundation models move from internal assistance to customer-facing or workflow-executing roles.
There is no universal standard for this yet, especially for systems that blend retrieval, tool use, and autonomous actions. Some teams can rely on coarse policy controls for low-risk summarisation use cases, while others need granular approvals, per-tool permissions, and stronger human-in-the-loop review. Best practice is evolving for agentic deployments, where the model is not just generating text but also triggering downstream actions.
Two edge cases deserve special attention. First, models exposed to sensitive or regulated data need stricter output controls because even a harmless-looking response can leak context. Second, fine-tuned or vendor-managed models may create a false sense of separation: if the application layer, retrieval layer, or prompt orchestration is weak, the deployment remains vulnerable regardless of the model brand. Teams should therefore measure post-deployment risk by the behaviour of the full system, not by the training pedigree of the model itself.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack surface, NIST AI RMF and NIST AI 600-1 set the technical controls, and EU AI Act define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF centers governance and monitoring across the model lifecycle. | |
| NIST AI 600-1 | The GenAI Profile maps directly to post-deployment monitoring and misuse controls. | |
| OWASP Agentic AI Top 10 | Agentic AI risks include tool abuse, prompt injection, and unsafe autonomous actions. | |
| MITRE ATLAS | ATLAS helps model adversarial tactics such as manipulation and evasion. | |
| EU AI Act | The AI Act reinforces lifecycle governance and post-market responsibility. |
Map likely attack paths to detections for prompt abuse, poisoning, and inference attacks.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 1, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
