Both, but runtime governance is the decisive layer when the model is already in production. A model may be aligned in isolation and still fail once generation begins inside an agentic workflow. That is why policy enforcement, monitoring, and intervention points have to sit inside the execution path, not only around the prompt.
Why LLM safety becomes a runtime governance problem in production
Once an LLM is embedded in a live workflow, the safety question shifts from “is the model generally well behaved?” to “what constraints exist at the moment of action?” A model can pass offline evaluation and still produce harmful output when it is connected to tools, retrieval, memory, or downstream automation. The operational unit of control is the execution path, not the prompt alone.
That is why production safety depends on policy enforcement that can shape, block, log, or terminate behaviour as it unfolds. Runtime controls also let teams separate benign generation from high-impact actions, which matters when the same model output can become a search query, a ticket update, a code change, or a privileged request.
For agentic systems, this distinction is even sharper because the model is not just answering, it is acting through chained steps. A runtime control point is the only place where an organisation can inspect the current context, apply guardrails consistently, and intervene before a step crosses an acceptable boundary.
What the model layer can and cannot prove
Model safety matters, but it mostly proves capability under test conditions. Alignment work, red-teaming, and dataset curation can reduce obvious failure modes such as toxic generation, prompt sensitivity, or instruction-following errors. They do not, by themselves, guarantee that the system will remain safe once tool access, external data, or autonomous planning are introduced.
This is the main limitation of treating safety as a model-only property: the model does not own the surrounding trust boundary. The surrounding application decides what the model can see, what it can call, which outputs are accepted, and whether a human must approve an action. Those decisions are governance decisions, not model-weights decisions.
Teams should therefore think in layers. The model sets a baseline capability profile, while runtime governance determines whether that capability can actually cause impact. NIST AI Risk Management Framework is useful here because it frames AI risk as something that must be managed across the lifecycle, not only during training.
Where runtime governance belongs in an agentic workflow
In an agentic workflow, runtime governance sits at the points where the system selects tools, consumes data, and commits side effects. That is where organisations can require approval for sensitive actions, enforce scope boundaries, and record enough telemetry to reconstruct what happened. If those checks are absent, the model can remain “safe” in isolation while the system as a whole becomes unsafe in execution.
Practically, this means governing the things that change the blast radius: permissions, tool routing, context injection, memory writes, and escalation paths. For example, an agent that can read records, call an API, and post a change ticket needs runtime constraints on each hop, not just a pre-deployment approval of the base model.
Runtime governance also gives teams a way to contain failure after deployment. If the output quality drops, or the model begins to drift into unsafe behavior, the control point can pause execution, force review, or fail closed. That is much harder to do if safety has been treated as a static property of the model artifact alone.
For agentic AI systems, the relevant control questions are often about identity, privilege, and tool use. NHIMG’s Agentic AI Security Guide and AI Security Platform Buyer’s Guide both reinforce the same operational point: guardrails are only meaningful when they are enforced where the system actually acts.
Risk and Threat Considerations
When organisations rely on model-only safety claims, the main risk is false confidence. The model may look acceptable in evaluation, yet still be exploitable once it is placed inside a workflow with retrieval, memory, APIs, or delegated actions. That creates exposure to prompt injection, tool misuse, data leakage, and unsafe autonomous behaviour.
Failure mechanism: the model’s output is treated as trustworthy even when the surrounding runtime has changed the context, the authority, or the available actions. Attackers and accidental misuse then succeed by abusing the execution layer, not by defeating the benchmark.
Impact: the organisation can see unauthorized data disclosure, harmful external actions, privilege misuse, or cascading operational failures, especially when the same model instance is connected to multiple tools or environments.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI Risk Management Framework | This subject is about governing AI risk across deployment and operation. |
| Recommendation — Apply AI RMF controls to manage model, workflow, and runtime safety together. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Runtime governance in agentic workflows depends on controlling delegated authority and privilege use. |
| ASI02 — Tool Misuse | The question centers on controls at the execution path where tools are invoked. | |
| Recommendation — Constrain agent authority and verify every high-impact action before execution. Restrict tool access and validate tool calls at the runtime boundary. | ||
| CSA MAESTRO | MAESTRO agentic AI security framework | The topic is agentic runtime governance, orchestration, and control placement. |
| Recommendation — Use MAESTRO to place governance controls at orchestration and execution points. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Runtime governance must limit what the model and its delegated processes can do. |
| AU-2 — Event Logging | The answer relies on monitoring and intervention inside the execution path. | |
| Recommendation — Enforce least privilege on every service, tool, and delegated action path. Log model actions, tool calls, and overrides for runtime detection and review. | ||
Practitioner Guidance
What to prioritise: treat runtime policy, observability, and intervention points as the primary safety controls for any LLM that can act beyond text generation. If the system can call tools, write memory, or trigger side effects, the control boundary must sit there.
What to verify: test the exact production path, not just the model endpoint. A safe sandbox, a red-team prompt set, or an offline benchmark does not prove that the live workflow will block unsafe tool use or detect suspicious chaining.
Common mistake: teams often certify the model and then assume the application is safe. In practice, the model verdict should be treated as input to governance, not as the final control decision.
Practitioner takeaway: if the LLM can influence a real action, safety is no longer a model question alone, it is an operational control question that must be governed at runtime.