Because the malicious change can live in the model artefact after the training session ends. If an attacker can poison a model through fine-tuning, the altered behaviour can keep influencing later outputs, even when the original access path is gone. That is persistence through the asset, not through the session.
Why model-shaping access creates persistence beyond the training session
Model-shaping permissions are different from ordinary runtime access because they can alter the artefact that future workflows depend on. If an attacker can change weights, adapters, prompts, or other model state, the effect may survive the original session and reappear every time that model is reused. The risk is not just immediate abuse, but durable behaviour change.
That persistence matters because AI workflows often treat the model as trusted infrastructure. Once a poisoned artefact is promoted into a shared training, evaluation, or inference path, later users may inherit the altered behaviour without needing the original attacker connection to remain open. In practical terms, the model becomes the persistence vehicle.
Persistence is stronger when model updates are loosely controlled, retraining outputs are automatically promoted, or rollback is weak. In those cases, a malicious fine-tune can look like a legitimate improvement, which makes the change harder to notice and easier to retain through normal operations.
How persistence shows up across AI lifecycle and access boundaries
The key issue is that model-shaping actions affect a long-lived asset, not a short-lived request. If training data, adapters, checkpoints, or merge artefacts are writable by an overprivileged actor, the attacker can embed malicious behaviour that remains available to downstream inference systems long after the authoring account is gone.
This is especially dangerous when multiple environments share the same base model or when teams reuse tuned artefacts across projects. A single compromise can then propagate into production, test, and internal tools, creating repeatable exposure that is hard to separate by session logs alone.
Controls that matter most are the ones that constrain who can modify model state, who can approve promotion, and how changes are validated before reuse. If the workflow cannot distinguish benign tuning from adversarial shaping, the organisation may preserve the compromise as if it were a standard model update.
Why the risk is more than a one-time compromise
Model-shaping permissions can turn a temporary foothold into a durable influence channel. Even if the original path is revoked, the attacker may have already changed how the model answers, filters, routes, or prioritises outputs, which means the compromise continues through the artefact itself.
That makes provenance, versioning, and rollback central. Teams need to know which model version is running, what changed in that version, and whether the change was expected. Without those checks, malicious behaviour can blend into normal model drift and survive routine deployment cycles.
For a useful companion view on the broader identity and access control side of this problem, Authorisation Models Guide explains how fine-grained access decisions help constrain who can shape shared assets. When model artefacts are treated as sensitive controlled resources, the same logic applies to tuning, merge, and promotion rights.
Risk and Threat Considerations
When model-shaping permissions are too broad, the main risk is durable compromise of the model artefact itself. That creates a persistence path that survives credential revocation, because the malicious change is stored in the tuned model, not only in the attacker’s session.
Failure mechanism: An attacker with fine-tuning or merge rights poisons the model state, then relies on normal reuse, promotion, or deployment to carry the altered behaviour into later workflows.
Impact: The organisation may continue to produce manipulated outputs, expose hidden backdoors, or carry forward unsafe behaviour across environments even after the initial access vector is closed.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-05 — Overprivileged NHI | Model-shaping rights are a high-risk privilege that can outlast a session and alter future behaviour. |
| NHI-01 — Improper Offboarding | Persistence survives when access is revoked but altered model artefacts remain in use. | |
| Recommendation — Restrict model-write privileges to the smallest approved set and require separate promotion approval. Revoke and revalidate model-shaping access when roles change or a change source is no longer trusted. | ||
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Workflows become persistent when an actor abuses change privileges over agent or model state. |
| Recommendation — Separate runtime use rights from state-changing rights and require approval for durable changes. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Limiting write access to model artefacts directly reduces the chance of durable poisoning. |
| CM-5 — Access Restrictions for Change | Model shaping is a controlled change activity that needs explicit authorization. | |
| Recommendation — Apply least privilege to model training, merge, and promotion permissions. Authorize and log all changes to model artefacts before promotion to shared workflows. | ||
Practitioner Guidance
What to prioritise: Treat model-writing and model-promotion rights as high-impact change privileges, not routine developer convenience. The most important decision is whether a user can alter the artefact that later workflows trust.
What to verify: Confirm that every tuned or merged model has an attributable source, an approved change record, and a rollback path. If you cannot prove which version introduced the behaviour, you do not yet have defensible control over persistence.
Common mistake: Teams often protect inference endpoints better than model artefacts. That leaves the durable object unguarded, which is exactly where persistence lives in this attack pattern.
Practitioner takeaway: Preventing persistence in AI workflows is mainly a matter of controlling who can rewrite the model, who can approve its reuse, and how quickly an unsafe artefact can be withdrawn.