They should do both, but provenance comes first when the concern is whether the deployed artefact is the one that was approved. Prompt filtering is still necessary, but it cannot compensate for an altered model with weakened internal safeguards. The first trust decision is what model is allowed into production.
Why model provenance belongs before prompt filtering
Provenance answers the first trust question: whether the model in production is the one that was reviewed, approved, signed off, and expected to behave within known guardrails. If that trust chain is broken, prompt filtering is only operating on an untrusted artefact. That makes provenance the higher-order control when the concern is integrity of the deployed model itself.
This matters because model behaviour is not only shaped by prompts. Training data, fine-tuning, weights, system prompts, tool access, and post-training modifications can all change how the model responds before any user input is filtered. If the artefact has been altered, the organisation may be relying on a control that is trying to contain a different risk than the one it actually faces.
For teams building provenance controls, the most useful baseline is to treat the model package like a release artifact that needs traceability from source to deployment. The AI Model Provenance and Watermarking Guide is a practical reference for the lineage, signing, and watermarking questions that establish whether a model can be trusted before it is exposed to live traffic.
What prompt filtering does well, and where it stops
Prompt filtering is still valuable because it can reduce exposure to obvious jailbreaks, unsafe requests, data exfiltration attempts, and malformed inputs that try to steer the model outside policy. It is a front-line interaction control, not a substitute for artefact integrity. Used well, it reduces the chance that a clean model is immediately pushed into unsafe behaviour by malicious or accidental user content.
The limitation is that prompt filtering only sees what comes in at inference time. It does not prove that the deployed model is the approved model, nor does it restore trust if the model has been swapped, poisoned, or weakened upstream. If the internal safety posture of the model has changed, the filter may still allow harmful outputs, miss subtle prompt attacks, or create a false sense of control.
That is why provenance and filtering solve different problems. One protects the supply chain and deployment trust boundary, while the other constrains runtime interaction risk. When organisations choose only one, they usually end up protecting the wrong layer.
How to sequence the control decision in practice
Start by verifying the model identity, provenance metadata, and release approval path, then confirm that the runtime instance matches what was approved. Only after that should you tune prompt filtering to the model’s actual behaviour and the application’s abuse cases. If the artefact cannot be trusted, prompt rules should be treated as a temporary containment measure, not the primary assurance mechanism.
Teams should also align provenance checks with the rest of the AI supply chain, especially where the model is built, packaged, or distributed through reproducible pipelines. SLSA is relevant because it frames build provenance and integrity verification for software artefacts, which is the same governance instinct needed when deciding whether an AI model is the one that was actually authorised.
For runtime hardening, prompt filtering works best when it is one layer in a broader control set that also includes logging, abuse monitoring, and change control over the model itself. NIST AI 600-1 GenAI Profile is useful here because it ties provenance, pre-deployment testing, and governance together rather than treating prompt safety as a standalone control.
Risk and Threat Considerations
When provenance is weak, the organisation can approve one model and unknowingly run another, which turns every downstream control into a partial defence at best. Adversaries or internal misuse do not need to defeat prompt filters if they can introduce a model with degraded safety behaviour, hidden backdoors, or altered tool-response patterns.
Failure mechanism: The deployment pipeline, registry, or release process accepts an unverified or modified model, and runtime filtering is then applied to an artefact whose internal behaviour no longer matches the reviewed baseline.
Impact: Unsafe outputs, policy bypass, data leakage, and silent loss of assurance can persist even though prompt controls appear to be functioning.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
SLSA, NIST AI 600-1 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| SLSA | Supply-chain Levels for Software Artifacts | Model provenance depends on integrity and traceability of built artefacts. |
| Recommendation — Apply SLSA provenance controls to verify what artefact was approved and deployed. | ||
| NIST AI 600-1 | GenAI Profile | Directly addresses GenAI governance, provenance, and pre-deployment testing. |
| Recommendation — Use the GenAI Profile to govern provenance checks before runtime prompt controls. | ||
| NIST SP 800-53 Rev 5 | CM-3 — Configuration Change Control | Model changes must be controlled before relying on runtime filtering. |
| SI-7 — Software, Firmware, and Information Integrity | Integrity verification is central when deciding whether the deployed model is trusted. | |
| AU-2 — Event Logging | Runtime prompt filtering needs auditability to support abuse detection and review. | |
| Recommendation — Require formal change approval for any model update before production use. Verify model artefact integrity before allowing it into production. Log prompt and response events to support investigation of unsafe model behaviour. | ||
Practitioner Guidance
What to prioritise: Confirm model identity and approval status first, especially for any model that can reach production, call tools, or influence downstream decisions. If provenance is uncertain, treat the environment as an integrity problem before treating it as a prompt-safety problem.
What to verify: Be able to show which exact artefact was deployed, who approved it, what changed since approval, and whether the runtime instance matches the intended release. Prompt filters should be validated against the deployed model, not against an assumed baseline.
Decision rule: If the question is “is this the right model?”, provenance comes first. If the question is “how do we reduce abuse of a trusted model?”, prompt filtering becomes a major control, but still not the first trust decision.
Practitioner takeaway: The strongest control is the one that protects the trust boundary you are actually crossing, and for production AI that boundary is model integrity before user input.
Related resources from NHI Mgmt Group
- Should organisations prioritise agent sandboxes or prompt filtering first?
- Should organisations prioritise data scoping or prompt filtering first?
- Which controls should organisations prioritise first to reduce prompt injection risk in production?
- Should organisations prioritise audit trails or model accuracy first in regulated AI?