Model-only protection leaves organisations exposed to unsafe prompts, malicious API activity, and behavior that emerges only after deployment. A model may scan clean, yet still be abused in production through tool calls, poisoned inputs, or agent actions. Effective AI security has to cover the full path from build time to runtime enforcement.
Why Model-File Security Alone Leaves Runtime Exposure Unchecked
Securing only the model file protects a static artefact, but it does not govern how the model behaves once it is called, prompted, connected to tools, or placed inside an agentic workflow. That distinction matters because the real exposure often appears in the interaction layer: unsafe instructions, over-broad tool access, and untrusted inputs can all reshape outcomes after deployment. For teams building or governing AI systems, the security question is not whether the model binary is intact, but whether execution is constrained at the point where decisions and actions occur. The OWASP Non-Human Identity Top 10 is useful here because runtime AI behaviour often depends on machine identities, tokens, and delegated access that a file scan never evaluates. In practice, many security teams discover this gap only after production integrations have already given the model permission to act.
What Actually Breaks Once the Model Is Deployed
At runtime, the model becomes part of a live control flow rather than a sealed artefact. That means the attack surface shifts from code integrity to behavioural integrity. Prompt injection can steer outputs, retrieval can supply hostile context, and connected tools can turn a bad instruction into an external action. If the system includes agents, the failure is larger still: the model may chain tool calls, pass data between systems, or trigger actions with authority that was never present in the original file.
Several common breakdowns follow from that shift:
- Input trust breaks when unvalidated prompts, documents, or API payloads influence the model.
- Tool trust breaks when the model can call functions, send messages, query systems, or change records without tight runtime policy.
- Identity trust breaks when the model inherits secrets, tokens, or service permissions that outlive the file itself.
- Monitoring breaks when teams watch model hashes or scan results but do not inspect prompt flows, tool invocations, or abnormal agent behaviour.
In AI security, the model file is only one control point. A clean artefact does not prevent a compromised session, a poisoned retrieval source, or a legitimate agent from being induced into unsafe action. NIST AI Risk Management Framework guidance is relevant because it treats AI risk as lifecycle risk, not just artefact risk. The practical boundary is the execution environment, where guardrails, authorisation checks, and logging determine whether the model can be safely used at all. Once those runtime checks are weak, the system can fail even when the trained model has not been altered.
Where organisations go wrong is assuming deployment is a packaging step rather than a new trust boundary. That assumption breaks down fastest in systems that connect to internal APIs, approval workflows, or external data sources.
Where the Model-Only Approach Frays in Real Deployments
Tighter model assurance often increases operational overhead, requiring organisations to balance artefact control against runtime governance. The hard part is that different deployment patterns fail in different ways, so the answer is not always “add another scan.” It is to decide which runtime conditions must be controlled before the model is trusted to act.
Model-only security frays in a few predictable situations. First, retrieval-augmented systems can ingest hostile text that the model file cannot anticipate. Second, tool-enabled assistants can be tricked into taking actions that are technically authorised but operationally unsafe. Third, agentic workflows can accumulate risk across steps, where each individual call looks normal but the sequence creates an unacceptable outcome. Fourth, secrets and session tokens may be reused in ways that create persistent access even if the model itself remains unchanged.
Guidance vs consensus: there is broad agreement that runtime controls are necessary, but teams still disagree on how much autonomy should be delegated to agents versus forced through human approval. That decision depends on the sensitivity of the action, the blast radius of the tool, and whether the system can prove what it attempted and why.
The approach also breaks down when teams cannot observe prompt content, retrieval sources, and tool execution together. Without that joined-up view, they may know the model exists, but not how it was influenced or what it tried to do. In that case, file-level protection becomes a false signal of safety.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, OWASP Non-Human Identity Top 10 and MITRE ATT&CK address the attack surface, NIST AI RMF and CIS Controls v8 set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | GOVERN — Govern | AI runtime governance must cover deployed behaviour, not just model artefacts. |
| Recommendation — Define runtime accountability and policy checks for AI behaviour before allowing production use. | ||
| OWASP Agentic AI Top 10 | A1 — Agentic Access Control | Agent tool use and delegated action are the core runtime exposure here. |
| Recommendation — Restrict agent actions to least-privilege tool access and enforce approval for high-impact calls. | ||
| OWASP Non-Human Identity Top 10 | NHI-01 — Secrets and Credential Management | Runtime AI systems often rely on tokens and machine credentials absent from model files. |
| Recommendation — Protect and rotate runtime credentials so AI components cannot abuse standing access. | ||
| CIS Controls v8 | 6 — Access Control Management | Runtime AI risk is driven by over-broad access to tools, APIs, and downstream systems. |
| Recommendation — Apply least privilege to AI service accounts and remove unnecessary runtime permissions. | ||
| MITRE ATT&CK | T1204 — User Execution | Prompt and instruction abuse can induce a legitimate system to execute attacker-influenced actions. |
| Recommendation — Map unsafe AI actions to abuse paths and detect when prompts drive unexpected execution. | ||
Practitioner Guidance
What to prioritise: Treat runtime authorisation as the primary control plane for any AI system that can query, decide, or act. If the model can only generate text, the control problem is narrower; if it can invoke tools or agents, the control problem includes identity, privilege, and transaction safety.
What to verify: Check whether prompts, retrieved content, tool calls, and output handling are policy-enforced at execution time. A model that passes pre-deployment review is not trustworthy if it can still be steered by untrusted inputs or operate under unchecked permissions.
Decision rule: If a runtime action can create business impact, require a control that can block, limit, or evidence that action at the moment it occurs. If you cannot do that, treat the system as higher risk than the model-file review suggests.
Common mistake: Teams often invest in secure model storage, signing, or scanning while leaving function calling, API access, and agent permissions effectively open. That produces compliance-looking assurance without behavioural control.
Practitioner takeaway: The security boundary is not the model file; it is the live execution path where prompts, tools, identities, and permissions combine into action.
Related resources from NHI Mgmt Group
- What breaks when organisations only secure the model layer of agentic AI?
- What breaks when model file validation is weak in AI platforms?
- What do organisations get wrong when they secure AI only at the model layer?
- When should organisations prioritise runtime guardrails over model-focused AI controls?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 7, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
