How can organisations prove what AI model actually ran?

Why This Matters for Security Teams

Proving what AI model actually ran is not just a deployment hygiene problem. It is an evidence problem. If a model can be swapped, rolled forward silently, or pulled from an untrusted source after approval, security teams lose the ability to tie a runtime event to a specific artifact, signature, and review state. That weakens incident response, auditability, and regulatory defensibility.

The control objective is similar to software supply chain verification, but with a sharper edge because model weights, adapters, prompts, and serving containers can each change independently. Security teams should treat the model artifact as a governed asset and verify it against signed provenance, approved hashes, and release metadata before any workload is allowed to serve traffic. That is consistent with broader control expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls and aligns with the evidence-driven posture discussed in The State of Secrets in AppSec.

NHIMG research also shows how quickly trust gaps become operational incidents: in DeepSeek breach, more than one million sensitive records were exposed after secrets and data handling failures. In practice, many security teams only discover model drift or unapproved model promotion after an incident review has already begun, rather than through intentional pre-deployment verification.

How It Works in Practice

The strongest approach is to create an end-to-end chain of custody for the model and the environment that serves it. That usually starts before deployment, when the model is packaged with immutable identifiers such as digest, version, training lineage, approval ticket, and signer identity. At deploy time, the serving platform should verify those claims against trusted metadata and refuse to start if the artifact does not match the approved record.

For runtime evidence, teams commonly combine signed provenance with workload identity, attestation, and policy checks. In practical terms, this means the serving system proves not only that a model file exists, but that the specific runtime instance is the one that was authorised. That evidence may include:

• Signed provenance records showing source, build, and approval history
• Certificate chains binding the model package to a trusted signing authority
• Integrity checks on the deployed artifact, container image, and adapter layers
• Workload identity or attestation that ties the serving process to a known host or enclave
• Policy-as-code gates that block unverified versions from entering production

For teams mapping this to control design, NIST SP 800-53 Rev 5 Security and Privacy Controls is useful for framing audit evidence, integrity, and configuration management. Current guidance also increasingly favors provenance systems that can be queried after the fact, because auditors need proof of the exact model that handled a request, not just a claim that the environment was “approved.” The same logic is reinforced in NHIMG’s State of Secrets in AppSec research, where fragmented control and weak operational hygiene undermine trust in what is actually running.

These controls tend to break down when teams allow hot-swaps, self-updating agent loops, or multi-region rollouts without immutable release records because the runtime state no longer matches the approval state.

Common Variations and Edge Cases

Tighter provenance controls often increase release overhead, requiring organisations to balance rapid model delivery against stronger proof of identity and integrity. That tradeoff becomes most visible in environments with frequent fine-tuning, canary releases, or model routing across multiple providers.

There is no universal standard for this yet. Some organisations treat the base model, adapter, and prompt bundle as separate attestable artifacts, while others bind them into one signed release object. The best practice is evolving, but the operational principle is stable: the more independently replaceable the components are, the more granular the evidence chain must be.

Edge cases matter. A model may be technically verified but still not be the one that generated a given response if a routing layer shifted traffic to a fallback model, an autoscaler recreated pods from a different image, or an internal agent selected a tool-backed model at runtime. In those cases, proof must extend beyond deployment records to request-time telemetry and policy logs. Where organisations rely on third-party inference endpoints, they should insist on verifiable attestations from the provider or treat the response as untrusted for high-risk use cases. That is especially important when secret exposure and rapid attacker action are in scope, as shown by DeepSeek breach and the broader risk trends described in The State of Secrets in AppSec.

Related resources from NHI Mgmt Group

• How can organisations tell whether their AI security model is actually working?
• How can organisations prove their AI controls are actually working?
• How can organisations know whether AI model registration is actually working?
• How should organisations prove AI systems are safe when the model changes continuously?