What breaks when organisations secure AI only at the model file and not at runtime?

Model-only protection leaves organisations exposed to unsafe prompts, malicious API activity, and behavior that emerges only after deployment. A model may scan clean, yet still be abused in production through tool calls, poisoned inputs, or agent actions. Effective AI security has to cover the full path from build time to runtime enforcement.

Why This Matters for Security Teams

Securing only the model file assumes the real risk is in the weights, when the operational risk often starts after deployment. Once an AI system can accept prompts, call tools, read data, or trigger workflows, the attack surface shifts to runtime decisions. That is where prompt injection, malicious API use, and poisoned inputs turn a “clean” model into an unsafe system. Guidance from the NIST Cybersecurity Framework 2.0 reinforces that protection has to span the full lifecycle, not just the artifact.

For NHI teams, this is the same mistake seen in secrets programs that focus on storage while ignoring exposure. NHIMG research shows the average time to remediate a leaked secret is 27 days, even though 75% of organisations express strong confidence in their secrets management capabilities, according to The State of Secrets in AppSec by GitGuardian & CyberArk. Runtime AI risk behaves similarly: what looks controlled in review can become exploitable in production. In practice, many security teams encounter abuse only after an agent has already made an unsafe tool call, rather than through intentional runtime testing.

How It Works in Practice

Model files are static assets. Runtime AI systems are not. The security question changes once a model is embedded in an application that accepts live inputs, exchanges context, and performs actions. A malicious prompt can steer an assistant into revealing data, chaining tools, or approving a workflow it should never touch. A poisoned retrieval source can alter answers without changing the model at all. An agent with delegated authority can do real damage even when the underlying model remains unchanged.

Effective controls therefore move from artifact scanning to runtime enforcement. That usually includes:

• Input validation and content filtering before prompts reach the model.
• Context boundaries so the model cannot freely consume all available data.
• Tool allowlists and per-action authorization for API calls, file access, or transactions.
• Short-lived credentials and scoped tokens so a single session cannot persistently expand access.
• Logging and policy checks on model outputs, tool calls, and downstream side effects.

This is where build-time assurance and runtime policy must work together. The DeepSeek breach illustrates how large-scale exposure can begin with the broader data and deployment environment, not just the model object. For implementation detail, current best practice is to treat the model as only one component inside a governed system, with runtime controls evaluated on each request rather than assumed from pre-deployment testing. These controls tend to break down when legacy applications expose broad backend privileges to the AI layer because the model can then turn one valid action into many unintended ones.

Common Variations and Edge Cases

Tighter runtime controls often increase engineering overhead, requiring organisations to balance safety against latency, usability, and integration complexity. That tradeoff is most visible in low-latency assistants, multi-agent workflows, and systems that must act on behalf of users in real time. There is no universal standard for exactly how much context an agent should receive, so current guidance suggests starting with least privilege and expanding only when a use case is proven.

Edge cases usually appear where the AI is not just answering questions but orchestrating work. A copilot that drafts text may need content filtering; an agent that edits records, sends messages, or executes code needs request-time authorization, auditability, and revocation. The Schneider Electric credentials breach is a reminder that when credentials or tokens are exposed, the attacker often bypasses the original security boundary entirely. The same principle applies to AI runtime: if the model is allowed to inherit standing access, the system can fail even when the model file itself is intact.

Best practice is evolving toward policy-as-code, short-lived permissions, and continuous runtime evaluation. Model-only security is useful for integrity and provenance, but it does not stop unsafe behaviour that emerges from prompts, tool chaining, or environmental access.

Related resources from NHI Mgmt Group

• What breaks when organisations only secure the model layer of agentic AI?
• What breaks when model file validation is weak in AI platforms?
• What do organisations get wrong when they secure AI only at the model layer?
• When should organisations prioritise runtime guardrails over model-focused AI controls?