A training or fine-tuning pipeline turns data, code and compute into a model that will later run in production, so anyone who can influence any stage of that pipeline can change what the model does without touching the serving system. Most organisations secure the model registry and the inference endpoint, then leave the stages in between (data snapshots, training images, distributed jobs, checkpoints, evaluation and promotion) with build-server levels of trust and none of the build-server controls. This guide treats the pipeline as a build system: it defines a gate and an evidence record for each stage, from data intake to the deployed artefact, and maps those gates to NIST SP 800-218A, SLSA and MITRE ATLAS. It is written for ML platform engineers, security architects and assurance teams who own or approve pre-training, continued training or fine-tuning pipelines. For the broader lifecycle overview, see the MLOps Security Guide; for method-level detail on adapters and safety regression, see the Fine-Tuning and LoRA Security Guide.
Key takeaways
- Every stage needs a gate and a record. NIST SP 800-218A asks producers to document the entire training process, track provenance of models, derivatives, libraries and pipelines, and generate attestations of training dataset integrity; a pipeline without per-stage evidence cannot prove what it built.
- Distributed training is a trusted-network protocol. PyTorch states that its distributed primitives (c10d, RPC, TCPStore) have no authorisation and send messages unencrypted, so anyone on the training network can execute code as the training user.
- Checkpoints are executable state. PyTorch says loading a distributed checkpoint may execute arbitrary code and that the weights_only protection does not apply, so only checkpoints your own infrastructure produced and stored should ever be resumed.
- The training job is a privileged non-human identity. It reads the most sensitive data and writes the artefact production will run; give it one scoped, short-lived identity and keep humans away from weights by default, as SP 800-218A suggests.
- Promotion is the control that matters most. Evaluate the exact artefact, verify its signature and provenance, and require an approver who did not run the training job before anything reaches the registry's production stage.
The pipeline as a build system
A model pipeline has the same shape as a software build: inputs (datasets, base models, code, container images), a build step (training) and an output artefact (weights or an adapter). The SLSA Build track gives useful vocabulary. At Build L1 provenance exists but is "trivial to bypass or forge"; at L2 a hosted platform generates and signs provenance; at L3 the platform isolates runs so they cannot influence one another and keeps signing secrets out of reach of user-defined steps. Applied to training, that means the orchestration platform, not the training script, should record which dataset version, base model digest, image digest and code commit produced each checkpoint, and sign that record. See the SLSA and Build Provenance Guide and the in-toto Attestations Guide.
The difference from ordinary software is that the output cannot be reviewed line by line. A poisoned dataset or a tampered checkpoint produces weights that look like any other weights. Integrity therefore has to be proven from the inputs and the process, and behaviour has to be tested afterwards.
Stage-by-stage gates and evidence
The table is the core of this guide: one gate per stage, the evidence that gate should produce, and the main threat it addresses. ATLAS IDs were checked against ATLAS 2026.10.
| Stage | Gate before the next stage | Evidence to keep | Main threat (MITRE ATLAS / ATT&CK) |
|---|---|---|---|
| 1. Data intake | Source on an allowlist; licence and lawful basis recorded; secret and PII scan passed | Source register, scan reports, dataset hash | AML.T0010.002 Data Repository; AML.T0020 Training Data Poisoning |
| 2. Data preparation | Filtering, deduplication and anomaly checks run on an immutable snapshot; changes reviewed | Transformation log, snapshot digest, reviewer | AML.T0020 Training Data Poisoning |
| 3. Base model and dependency intake | Base model verified by hash or signature; safe format; packages pinned by hash from a private mirror | Base model digest, AI-BOM entry, lock file | AML.T0010.003 Model Repository; AML.T0010.001 Software; T1195.002 |
| 4. Training environment | Image built in CI and pinned by digest; job network isolated; one scoped job identity | Image digest and SBOM, network policy, role definition | AML.T0012 Valid Accounts; T1190 Exploit Public-Facing Application |
| 5. Training run | Run launched only by the orchestrator from approved inputs; checkpoints written to a store only the job can write | Signed run provenance (inputs, code commit, hyperparameters, image) | AML.T0018.000 Poison AI Model |
| 6. Checkpoint handling | Checkpoints hashed on write; resume only from the job's own store; old checkpoints expired | Checkpoint manifest with hashes and retention | AML.T0018 Manipulate AI Model; AML.T0035 AI Artifact Collection |
| 7. Evaluation | Held-out evaluation set access-controlled and versioned; safety, task and regression tests on the exact artefact | Evaluation report bound to the artefact digest | AML.T0018 Manipulate AI Model |
| 8. Signing and promotion | Artefact signed; provenance verified; separate approver; registry version immutable | Signature bundle, approval record | AML.T0010.003 Model Repository |
| 9. Deployment | Serving verifies signature and loads only registered versions; rollback criteria defined | Deployment log, verification result | AML.T0011 User Execution |
What NIST SP 800-218A asks for at each stage
NIST SP 800-218A, final since July 2024, is the SSDF Community Profile for generative AI and dual-use foundation models. It adds AI-specific recommendations to SSDF tasks. The ones that bear directly on the pipeline are summarised below; read the publication for exact wording, and see the NIST SP 800-218A SSDF Community Profile for Generative AI Guide for the full profile.
| SP 800-218A item | Pipeline stage | What it asks for (paraphrased) |
|---|---|---|
| PW.3.1 (R1, R2) | Data intake and preparation | Verify provenance and integrity of training, fine-tuning and alignment data before use; apply anomaly detection, filtering and sanitisation |
| PW.3.2 | Data intake | Track provenance of all training, testing, fine-tuning and aligning data and document data whose provenance is unknown |
| PW.4.4 (R1, R2) | Base model and dependency intake | Verify and scan acquired models, datasets, reward models, adaptation layers and configurations before use |
| PO.5.1 (R2, R3, R4, R6) | Training environment | Keep sensitive data in approved environments, apply least privilege to pipelines and registries, monitor training activity and registry changes, and monitor for plaintext secrets |
| PS.1.1 and PS.1.3 | Training run and checkpoints | Keep models, weights, pipelines and reward models in secure storage; minimise direct access; store reward models separately; keep weights apart from training data; add encryption, hashes, signatures, multiparty authorisation or air-gapping by risk |
| PS.3.1 and PS.3.2 | Whole pipeline | Version the tools used for dataset creation and training, document the entire training process, and track provenance of models, derivatives, libraries and pipelines |
| PW.8.2 (R1, R2) | Evaluation | Test models for vulnerabilities and retest when retrained or when new data sources are added |
| PS.2.1 (R1, R2) | Signing and promotion | Provide hashes or signatures for the model and its components, and sign model changes |
| RV.2.2 (R2) | Deployment | Define criteria for stopping use of a model or rolling back to a previous version |
Securing the training environment
Distributed training traffic
PyTorch's security policy is direct: PyTorch Distributed features "are intended for internal communication only", none of the primitives include an authorisation protocol, messages are sent unencrypted, and anybody with network access can execute arbitrary code with the privileges of the user running PyTorch. Treat the training fabric as a closed segment. Put workers in a dedicated subnet or namespace with network policy that allows only peer-to-peer traffic between job members, keep rendezvous endpoints off shared networks, and never expose them across tenants. The same reasoning applies to cluster control planes: the ShadowRay 2024 campaign used internet-reachable Ray job interfaces to run code on AI clusters and take cloud credentials, AI tokens and compute. See the Ray and AI Compute Cluster Security Guide and GPU and AI Cluster Multi Tenancy Security.
Managed training network isolation
The major cloud ML platforms each document a way to stop training jobs reaching the internet or data outside an approved boundary. Platforms are listed alphabetically; each is a customer-side setting.
| Platform | Setting | What the documentation says it does | Operational consequence |
|---|---|---|---|
| Amazon SageMaker AI | EnableNetworkIsolation on CreateTrainingJob (optionally with VpcConfig) | Containers cannot make outbound calls to any service, including Amazon S3, and no AWS credentials are made available to the container runtime; SageMaker AI moves input and output data outside the container using the execution role; multi-instance traffic is limited to training peers | Training code reads and writes local paths; dependencies must be in the image |
| Google Cloud Gemini Enterprise Agent Platform | VPC Service Controls perimeter | Training data for AutoML or custom models stays inside the perimeter, and default internet access from the Google-managed environment is blocked | Public PyPI installs are not possible; use custom containers or install packages from Artifact Registry |
| Microsoft Azure Machine Learning | Workspace managed virtual network, isolation mode allow_only_approved_outbound | Outbound traffic from compute instances, clusters, serverless compute and managed online endpoints is blocked except approved service tags, FQDNs and private endpoints, with data exfiltration protection enabled automatically | Microsoft notes that added outbound rules, especially FQDN rules, are not guaranteed exfiltration protection; once set, the isolation mode cannot be switched or disabled |
As of November 2026. For platform-specific depth see the Amazon SageMaker Security Guide, the Google Vertex AI Security Guide and the Azure Machine Learning Security Guide.
Images, dependencies and secrets
- Build images in CI, not on the cluster: training images should be built from a reviewed Dockerfile, scanned, signed and referenced by digest in the job spec. A tag such as :latest lets the image change between runs without a record.
- Pin Python dependencies by hash: the PyTorch torchtriton attack showed how a public package with the same name as an internal dependency was preferred by pip and stole SSH keys and environment variables from nightly users. Use a private mirror and hash-pinned lock files.
- No secrets in job specs or notebooks: SP 800-218A PO.5.1 calls for monitoring environments for plaintext secrets. Inject short-lived credentials at runtime through workload identity instead.
Data, checkpoints and the training run
Data controls (source allowlists, provenance, poisoning defences) are covered in depth in the CISA AI Data Security Guidance Guide and the AI Data Poisoning and Backdoors Guide. The pipeline-specific points are these.
- Train from snapshots, not live locations: point jobs at an immutable, versioned dataset snapshot whose digest is recorded in run provenance. A job that reads a mutable bucket prefix can train on data that changed after review.
- Scan for secrets before training: the 12,000 Live Secrets in LLM Training Data 2025 case found 11,908 working credentials in a Common Crawl snapshot used for model training. Revoke anything found; removing it from the dataset does not undo the exposure.
- Protect checkpoints like releases: PyTorch states that distributed checkpoints are meant to be restored only from storage you control, that loading one "may execute arbitrary code", and that the weights_only protection of torch.load does not apply. Write checkpoints to a location only the job identity can write, hash them on write, and never resume from a checkpoint copied in from elsewhere.
- Keep reward models and evaluation sets separate: SP 800-218A recommends storing reward models separately from models and data. Apply the same to held-out evaluation sets: if the training identity can read or write them, results can be contaminated or gamed.
- Expire what you no longer need: intermediate checkpoints are full copies of model weights. Set retention so they do not become an unmanaged store of valuable artefacts. See Securing AI Model Weights.
Where identity fits
A training pipeline typically holds several non-human identities with unusual reach: the orchestrator's service account, the training job's cloud role, the hub token used to pull base models, the experiment tracker API key and the registry credential used to publish. Each should be distinct, scoped to its stage and short-lived.
| Identity | Should be able to | Should not be able to |
|---|---|---|
| Data ingestion pipeline | Write new raw data to a landing zone | Approve snapshots or read training outputs |
| Training job role | Read one approved snapshot, write checkpoints and outputs to staging | Write training data, overwrite registry versions, reach the internet |
| Model hub or registry pull token | Read allowlisted base models | Publish or delete models |
| Promotion identity | Sign and move a verified artefact to production stage | Launch training jobs |
| Human engineers | Launch jobs through the orchestrator, read metrics | Download production weights directly (SP 800-218A PS.1.1 suggests considering blocking human access to weights) |
Use cloud-native workload identity or federation rather than static keys in job definitions. See the Workload Identity Federation Guide and the CI/CD Pipeline Identity Security Guide.
How to implement
- Inventory pipelines and identities (weeks 1 to 2): list every training and fine-tuning pipeline, where it runs, its data sources, base models, images and the identities it uses. Record them in the AI-BOM. See the AI Supply Chain and AI-BOM Guide.
- Close the network (weeks 2 to 4): turn on managed network isolation or equivalent policy for one pilot pipeline, move dependencies into images or a private mirror, and confirm the job still completes. Roll back by reverting the job definition, not by opening the network permanently.
- Split identities (weeks 3 to 6): replace shared credentials with one role per stage, remove standing human access to weights, and log every assumption of the training role.
- Add provenance (weeks 4 to 8): have the orchestrator record dataset digest, base model digest, image digest, code commit and hyperparameters for every run and sign the record. Start in record-only mode, then make missing provenance a promotion blocker.
- Gate promotion (weeks 6 to 10): require a signed artefact, verified provenance, an evaluation report bound to the artefact digest and an approver who did not run the job. Serving verifies the signature before load. See the Model Signing and OpenSSF Model Signing Guide.
- Extend and rehearse: roll the pattern to remaining pipelines, then rehearse a tampered-checkpoint or poisoned-snapshot scenario. The NIST, NSA and CISA Key Practices for Incident Readiness in AI Training and Evaluation Guide covers response preparation.
Decision guidance
- Managed fine-tuning API versus self-managed training: a provider-hosted fine-tuning service removes the cluster, network and checkpoint stages from your scope but leaves data handling, job permissions and who can deploy or call the custom model. Self-managed training gives full control and full responsibility for every gate above.
- Small teams: start with the three highest-value gates: immutable dataset snapshots, network-isolated jobs with one scoped role, and signed promotion. Manual approval is acceptable at low volume.
- Large or frontier-scale programmes: automate provenance at SLSA Build L3-style isolation, separate training and promotion duties formally, and apply SP 800-218A's multiparty authorisation and air-gapping options to the most valuable weights.
- Continuous or online training: treat each data refresh as a new input that triggers re-evaluation, as SP 800-218A PW.8.2 expects when new data sources are added.
Detection and measurement
Log job submissions (who, which inputs, which image), training role assumptions, reads of evaluation sets, writes to checkpoint stores, registry stage transitions and signature verification failures at serving. Alert on jobs launched outside the orchestrator, checkpoint writes from identities other than the job, and promotions without a matching provenance record.
- Provenance coverage: share of production models with a signed run record linking dataset, base model, image and code digests.
- Isolated-job ratio: share of training jobs running with network isolation or an approved-outbound boundary.
- Standing access to weights: number of human identities able to read production weights directly (target near zero).
- Unverified loads: serving loads that skipped or failed signature verification (target zero).
- Re-evaluation latency: time between a data source change and a completed evaluation of the retrained model.
Framework mapping
| Control objective | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | SP 800-218A |
|---|---|---|---|
| Record data, model and pipeline provenance | SR-4 Provenance, CM-8 | ID.AM-08 | PW.3.2, PS.3.2 |
| Isolate training networks | SC-7 Boundary Protection | PR.IR-01 | PO.5.1 |
| Least privilege for job and promotion identities | AC-6 Least Privilege | PR.AA-05 | PO.5.1, PS.1.1 |
| Integrity of checkpoints and released models | SI-7, SI-7(6), SC-28 | PR.DS-01 | PS.1.3, PS.2.1 |
| Baseline training images and configuration | CM-2 Baseline Configuration | PR.PS-01 | PS.3.1 |
| Evaluate before promotion and after retraining | SA-11 Developer Testing and Evaluation | PR.PS-06 | PW.8.2 |
| Monitor training activity and registry changes | AU-12, SI-4 | DE.CM-09 | PO.5.1, PO.5.3 |
Common failure patterns
- Open training fabric: distributed workers or cluster dashboards reachable from shared or public networks, despite PyTorch and Ray both assuming a trusted network, as the ShadowRay campaign showed.
- Resuming from a borrowed checkpoint: a checkpoint copied from a shared drive or another team's bucket was resumed, although distributed checkpoint loading can execute code.
- Training on a moving target: jobs read a live bucket prefix, so the data trained on was not the data reviewed, and no digest exists to prove otherwise.
- One role for everything: the same service account ingests data, trains, writes to the registry and promotes, so one leaked key gives control of the whole chain.
- Evaluation set inside the blast radius: the training identity can write the held-out set, so evaluation results cannot be trusted.
- Promotion by tag: a registry stage was changed by name without signature or provenance checks, so a swapped artefact would pass.
Practitioner checklist
- Define a gate and an evidence record for each pipeline stage, from data intake to deployment.
- Train only from immutable, versioned dataset snapshots whose digests are recorded in run provenance.
- Scan training data for secrets and personal data, and revoke any credential found.
- Run training jobs network-isolated, with dependencies baked into digest-pinned images or a private mirror.
- Keep distributed training traffic and rendezvous endpoints on a closed segment limited to job peers.
- Write checkpoints to a store only the job can write, hash them, expire old ones, and never resume from external checkpoints.
- Give each stage its own short-lived non-human identity and remove standing human access to production weights.
- Bind evaluation reports to the artefact digest and keep evaluation sets out of the training identity's reach.
- Promote only signed artefacts with verified provenance, approved by someone who did not run the job.
- Define rollback criteria and rehearse a tampered checkpoint or poisoned snapshot scenario.
Standards and references
- NIST: SP 800-218A Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
- OpenSSF: SLSA Build Track Basics
- PyTorch: Security Policy (Using PyTorch Securely)
- NCSC: Guidelines for secure AI system development
- AWS: Network isolation for SageMaker AI training and inference containers
- Google Cloud: VPC Service Controls for Agent Platform
- Microsoft Learn: Workspace managed virtual network isolation for Azure Machine Learning
- OpenSSF and Sigstore: model-transparency (model signing)
Related NHI Mgmt Group resources: MLOps Security Guide · Fine-Tuning and LoRA Security Guide · If your model registry is compromised · LLM Quantisation Explained