Why do model files and live AI behaviour create different security risks?

Model files can hide malicious code or tampering in serialized artifacts, so the risk is in what gets loaded. Live AI behaviour is different because the same model can produce unexpected, unsafe, or manipulated outputs depending on the prompt and context. That makes one control suitable for static integrity checks, and the other necessary for behavioural risk testing.

Why model artefacts and runtime outputs need different controls

Model files and live behaviour sit at different points in the AI lifecycle, so the security question changes with the object you are evaluating. A file on disk is a static artefact: you care about tampering, provenance, hidden payloads, and whether the artifact you load is the artifact you intended. Live behaviour is dynamic: you care about prompt sensitivity, unsafe generation, policy bypass, tool misuse, and whether the system behaves safely under realistic inputs. The NIST Cybersecurity Framework 2.0 is useful here because it separates governance, protection, detection, response, and recovery concerns instead of treating every AI issue as the same kind of control problem.

Practitioners often get this wrong by assuming that a clean file scan means a safe model, or that behavioural testing can substitute for verifying the artifact itself. Those are different assurance questions, and they fail in different ways. In practice, many security teams discover the gap only after a trusted model file has been altered in the supply chain or after a benign-looking model starts producing unsafe outputs in deployment.

How the two risk surfaces work in practice

For model files, the core issue is integrity. The file format may contain weights, configuration, tokenizer data, or other serialized components that are loaded into an execution environment. If an attacker can alter the artifact, they may change what is deployed, embed malicious logic, or create a backdoored model that behaves normally in most cases but diverges under a trigger condition. Even without active malice, uncontrolled reuse of artifacts can create version drift, provenance loss, and accidental deployment of the wrong model.

For live AI behaviour, the risk is not whether the file is intact but whether the system behaves safely when exposed to real prompts, context, tools, and downstream automation. The same model can be safe in one setting and unsafe in another because the surrounding application changes what it can see, what it can call, and what it is allowed to do. Behavioural risk therefore includes prompt injection, instruction hierarchy confusion, unsafe tool invocation, policy evasion, and output that is technically valid but operationally harmful.

Static checks answer: did we load the right artifact, from the right source, with the right integrity?
Behavioural tests answer: does the system remain safe under adversarial or unexpected interaction?
Deployment review answers: what extra context, tools, and permissions change the model’s real-world risk?

Those controls complement each other rather than duplicate each other. A strong artifact pipeline does not prove safe runtime behaviour, and a good red-team evaluation does not prove the file has not been tampered with. Where teams connect the model to external tools, retrieval layers, or automated actions, runtime behaviour becomes more consequential because unsafe output can cross from prediction into action. This guidance breaks down when organisations treat an AI model as a single static asset and ignore the application layer that gives the model operational power.

Where the boundary blurs in real deployments

Tighter assurance often increases operational overhead, because teams must validate both the artifact and the behaviour, and those checks serve different purposes. The distinction becomes less clean when the model is packaged with prompts, policies, adapters, fine-tuning layers, or retrieval sources, because each of those can alter the effective security posture even when the base model file is unchanged. That means the question is not just whether the model is “safe,” but which layer is responsible for which failure mode.

There is also an active debate about how much behavioural testing is enough. Consensus is still forming across the industry, but the practical rule is simple: if the model can influence decisions, generate content for users, or trigger tools, it needs runtime evaluation in addition to artifact integrity checks. If the model is merely stored, versioned, or transferred, then file integrity and provenance become the dominant concern.

In edge cases, the same weakness can exist in both layers. A poisoned model file may carry a dormant behaviour that only appears under specific prompts, while a clean model may still become unsafe because the prompt chain or surrounding orchestration is compromised. That is why the control set should follow the risk surface, not the label on the asset.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and CIS Controls v8 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
NIST CSF 2.0	GV.1 — Cybersecurity Governance	The question is about differentiating AI security assurance domains.
	PR.DS.1 — Data-at-Rest Protection	Model files are stored artifacts whose integrity and protection matter before execution.
	DE.CM.8 — Anomalous Activity Detection	Live model behavior needs monitoring for unsafe or unexpected outputs in operation.
Recommendation — Use governance to separate artifact integrity controls from runtime behavior testing. Protect model artifacts at rest and verify their integrity before deployment. Monitor deployed AI behavior for anomalous or unsafe runtime patterns.
NIST AI RMF	MAP-1 — Context and Intended Use	The answer depends on distinguishing artifact assurance from behavioral assurance in AI systems.
	MEA-1 — Measure and Evaluate AI Risk	Behavioural testing is a direct AI-risk evaluation concern.
Recommendation — Define the model’s intended use and test both the asset and its deployment context. Evaluate deployed model behavior under realistic and adversarial prompts.
CIS Controls v8	2.2 — Software Asset Inventory	Model files are software artifacts that need inventory, versioning, and provenance control.
	8.2 — Audit Log Management	Runtime AI systems need traceability for prompts, outputs, and downstream actions.
Recommendation — Inventory model artifacts so only approved versions are deployed. Log AI interactions and actions so unsafe behavior can be investigated and contained.
MITRE ATT&CK	T1027 — Obfuscated Files or Information	Tampered or hidden payloads in model files align with concealed artifact abuse.
Recommendation — Hunt for hidden content or tampering in model artifacts before execution.

Practitioner Guidance

What to prioritise: Separate assurance for artifact integrity from assurance for runtime behaviour. Treat them as different control objectives, and do not accept one as evidence for the other.

What to verify: Confirm the model’s source, version, and hash before loading, then test the deployed system under realistic prompts, context shifts, and tool-access conditions before allowing production use.

What practitioners underestimate: The surrounding application often creates more risk than the base model itself. Retrieval, plugins, orchestration logic, and action permissions can turn a moderately risky model into a materially dangerous system.

Practitioner takeaway: The most reliable posture is to govern the artifact as a supply-chain object and the deployed model as an interactive system, because those two assurance problems fail in different places and require different evidence.