Fine-Tuning and LoRA Security Guide: Protecting Custom Models

Fine-tuning changes what a model does, so whoever controls the training data, the training job or the adapter file controls the behaviour of the model you ship. Supervised fine-tuning and parameter-efficient methods such as LoRA (Low-Rank Adaptation) have made custom models cheap enough for any team to build, and adapters are now passed around as small files that can be loaded into an inference server at runtime. That creates a pipeline with its own data, credentials, artefacts and release process, and each stage can weaken safety, leak sensitive data or plant a backdoor. This guide covers the fine-tuning threat model, how to secure training data, job permissions and adapter artefacts, how to evaluate a model before release, and what to monitor. It is written for ML platform engineers, AI security leads and application owners who commission or approve custom models. Poisoning techniques in general are covered in the AI Data Poisoning and Backdoors Guide; this page focuses on controls for the fine-tuning pipeline.

Key takeaways

• Fine-tuning can remove safety alignment. Researchers broke GPT-3.5 Turbo's guardrails with 10 adversarial training examples for under $0.20, and found that benign datasets also degrade safety, so every fine-tuned model needs its own safety evaluation.
• Small amounts of poison are enough. Research published in October 2025 found around 250 poisoned documents compromised models from 600M to 13B parameters, and reported the same pattern for fine-tuning; percentage-based data sampling will not catch it.
• Adapters are deployable code paths. A LoRA adapter changes model behaviour without touching the base weights; treat adapters as release artefacts with provenance, signing and an allowlist, and keep runtime adapter loading off in production.
• Training jobs are privileged non-human identities. The service role that reads training data and writes the model needs scoped storage access, confused-deputy conditions, encryption and private networking.
• Safety training does not reliably remove backdoors. The Sleeper Agents study showed trigger-based behaviour surviving supervised fine-tuning, reinforcement learning and adversarial training, so provenance and data controls matter more than after-the-fact clean-up.

Fine-tuning methods and what they change

PEFT's modules_to_save option trains extra modules (for example a classifier head) in full and saves them with the adapter, so an "adapter" can carry more than low-rank matrices. Check the adapter configuration, not just the file size, when reviewing what an adapter changes.

The fine-tuning threat model

• Safety regression: Qi and colleagues (2023) showed that fine-tuning an aligned model, even with non-malicious data, can make it comply with harmful requests. A fine-tuned model inherits none of the base model's safety evaluation results.
• Training data poisoning and backdoors: manipulated examples can plant a trigger that changes behaviour only when a specific phrase or context appears. OWASP LLM04:2025 Data and Model Poisoning lists fine-tuning as one of the lifecycle stages at risk, and MITRE ATLAS tracks AML.T0020 Poison Training Data and AML.T0018.000 Poison AI Model.
• Malicious or vulnerable adapters: OWASP LLM03:2025 Supply Chain lists vulnerable LoRA adapters and collaborative model merge services as distinct risks, including a scenario where a compromised supplier's adapter is merged into a deployed model. ATLAS AML.T0058 Publish Poisoned Models covers the distribution route.
• Benchmark gaming: OWASP LLM03 also describes a model fine-tuned to score well on safety benchmarks while carrying targeted triggers, which is why public benchmark results are not evidence of safety for a third-party fine-tune.
• Sensitive data memorisation: Carlini and colleagues extracted verbatim training examples, including names, phone numbers and email addresses, from GPT-2 by querying it, and found larger models more vulnerable. Fine-tuning on tickets, emails or code puts that content in reach of anyone who can query the model (OWASP LLM02:2025 Sensitive Information Disclosure; ATLAS AML.T0057 LLM Data Leakage and AML.T0024 Exfiltration via AI Inference API).
• Theft of the custom model: a fine-tuned checkpoint or adapter encodes proprietary data and effort. Stolen weights give full access for offline analysis (ATLAS AML.T0044 Full AI Model Access).
• Persistence through the model: a backdoor in a fine-tuned model is a persistence mechanism that survives host rebuilds and credential rotation, because it lives in an artefact you redeploy. The Sleeper Agents study found such behaviour can survive later safety training. See the Persistence Techniques Guide.

Securing fine-tuning data

• Record provenance for every example: source system, collection date, owner and licence for each dataset, with immutable versions (for example data version control or object versioning) so you can reproduce exactly what a model was trained on. OWASP LLM04 recommends tracking data origins and transformations, citing CycloneDX and ML-BOM. See the AI Training Data Provenance and Copyright Exposure Guide.
• Restrict who can write training data: separate the people and pipelines that contribute data from those that approve a dataset version for training, and log every write. A single contributor with write access is enough to insert a small, targeted poison set.
• Treat user-generated content as untrusted: chat logs, support tickets, reviews and scraped pages can be shaped by outsiders. Filter, deduplicate and review them, and weight human-curated data above anything an external party can influence.
• Remove secrets and personal data before training: run secret scanning and PII detection on the dataset, not only on source repositories. Truffle Security found 11,908 live secrets in a December 2024 Common Crawl archive used for LLM training, as covered in 12,000 Live Secrets in LLM Training Data 2025. Revoke any secret found; scrubbing the dataset does not undo the exposure.
• Establish a lawful basis for personal data: confirm you are allowed to use the data for training and record the decision. See the AI and Privacy Guide and the Privacy Programme Guide for Security Teams.
• Use data the model needs, no more: OWASP LLM04 suggests keeping volatile user-supplied information in a retrieval store rather than training it in, so it can be changed or deleted without retraining. See the RAG Security Guide.

Fine-tuning privileges and job identity

A training job is a non-human identity with unusual reach: it reads the most sensitive datasets in the organisation and writes an artefact that will later run in production. Managed platforms make this explicit. Amazon Bedrock model customisation jobs run under an IAM service role trusted by bedrock.amazonaws.com, and AWS recommends aws:SourceAccount and aws:SourceArn conditions in the trust policy to prevent confused-deputy use, read-only access to training and validation buckets, write access only to the output bucket, s3:prefix conditions to narrow folders, KMS keys for the job, data and custom model, and a vpcConfig so job traffic stays in your VPC.

• Use short-lived credentials: notebooks and training clusters often hold long-lived Hugging Face, registry and cloud keys. Replace them with workload identity and scoped, expiring tokens. See the AI Infrastructure Workload Identity Guide.
• Protect publishing tokens as supply chain credentials: in the Hugging Face API tokens exposure, Lasso found 655 leaked tokens with write permission and said it could have modified 14 heavily downloaded datasets. A write token to your model or dataset repository is a direct route to poisoning your own fine-tunes. See the Hugging Face Security Guide.
• Isolate the training environment: restrict egress from training clusters, so a compromised dependency cannot exfiltrate data or weights, and keep training nodes off general corporate networks.

Adapter and model artefact security

• Use safe serialisation: save and accept adapters and checkpoints in safetensors, which stores tensors without executable content, and reject pickle-based files from outside the organisation. Hugging Face's pickle documentation notes that its Hub scanning is not foolproof.
• Sign and hash artefacts: record the hash of the base model, adapter, training data version, training code commit and hyperparameters in a lineage record, and sign it. Verify the signature at deployment. See the AI Model Provenance and Watermarking Guide and the AI Supply Chain and AI-BOM Guide.
• Bind adapters to their base: an adapter tested on one base model revision is a different system on another. Pin the base model revision in the deployment and refuse mismatches.
• Keep runtime adapter loading off in production: vLLM's LoRA documentation says VLLM_ALLOW_RUNTIME_LORA_UPDATING (which enables /v1/load_lora_adapter, /v1/unload_lora_adapter and in-place replacement with load_inplace) should not be used in production unless the environment is isolated and fully trusted, and that the Hugging Face Hub resolver's remote downloads are not intended for production. Register adapters at start-up with --lora-modules instead. See the Self-Hosted LLM Security Guide.
• Vet third-party adapters like third-party code: downloading a community LoRA from a model hub is equivalent to adding an unreviewed dependency that changes runtime behaviour. Require an approved publisher, a pinned revision and your own evaluation before use. See the Open-Weight Model Security Guide.
• Record merges: once an adapter is merged with merge_and_unload(), the resulting checkpoint looks like any other. Keep the pre-merge adapter and the lineage record, or you lose the ability to trace a behaviour change to its source.

Evaluating a fine-tuned model before release

Every fine-tune is a new model for evaluation purposes. Managed services increasingly enforce this: Microsoft Foundry's fine-tuning safety evaluation checks training data for harmful content before training and fails the job above a severity threshold, then simulates conversations with the trained model and marks it as not deployable if harmful output exceeds an acceptable rate, with thresholds customers cannot change. Self-managed pipelines need an equivalent gate.

1. Baseline: run the same safety, refusal and capability suites on the base model so regressions are measurable.
2. Safety regression: run harmful-request, jailbreak and policy suites relevant to the use case against the fine-tune and compare with the baseline. Block release on any regression above an agreed tolerance.
3. Memorisation and leakage: probe for verbatim reproduction of training records, secrets and personal data, including canary strings inserted into the training set for this purpose.
4. Backdoor screening: test behaviour across varied contexts, dates, languages and rare tokens, and compare outputs with the base model on the same prompts. Treat this as risk reduction, not proof: the Sleeper Agents results show triggers can evade standard testing.
5. Task and misuse red teaming: test the specific use case, including prompt injection through the content the model will process. See the AI Red Teaming Guide.
6. Release decision: a release approver who did not train the model signs off on the evidence and the lineage record.
7. Rollback plan: keep the previous model or adapter version deployable, so a failing fine-tune can be withdrawn by configuration change.

Implementing a secure fine-tuning pipeline

1. Inventory: list every fine-tuned model and adapter in use, with base model, data sources, owner and where it is deployed. Include adapters loaded by individual teams on shared inference servers.
2. Freeze runtime loading: disable runtime adapter updates and hub resolvers on production servers first; this closes the quickest path for an unreviewed adapter.
3. Move training into a pipeline: versioned datasets, a dedicated job identity, private networking, encrypted storage and a staging registry. Pilot with one project before mandating it.
4. Add the evaluation gate: start in report-only mode, publishing evaluation results without blocking, then enforce once thresholds are calibrated on a few releases.
5. Sign and verify: sign lineage records at the end of training and verify them at deployment; reject unsigned artefacts once coverage is complete.
6. Monitor and review: monitor deployed fine-tunes and re-evaluate after any change to base model, data or adapter.

Managed API versus self-managed training

A managed fine-tuning service handles the training environment and often adds safety screening, so the work is mostly data governance, job permissions and controlling who can deploy and call the custom model. Self-managed training with PEFT on your own GPUs gives full control of weights and adapters, but you own the environment, artefact security and the evaluation gate. Small teams usually get a better security result from a managed service with strict data controls; large organisations with many adapters need a central model registry and promotion process whichever route they choose.

Detection and measurement

• Data plane: writes to training datasets, new data sources added, and dataset versions approved, with the identity responsible.
• Job plane: fine-tuning jobs started, by whom, on which dataset version, and from which network; GPU spend per project.
• Artefact plane: uploads to and downloads from the model registry, overwrite attempts, unsigned artefacts and bulk weight exports.
• Serving plane: adapter load and unload events, adapters requested that are not registered, and output drift against the evaluation baseline.

Framework mapping

NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (March 2025), provides the shared vocabulary for poisoning and privacy attacks. See the MITRE ATLAS Guide and the NIST AI RMF Guide.

Common failure patterns

• Assuming the base model's safety carries over: releasing a fine-tune after only task accuracy testing, when research shows even benign fine-tuning can weaken refusals.
• Training on raw exports: support tickets, chat logs or code repositories used without secret scanning or PII removal, so the model can reproduce credentials and customer data on request.
• Hot-loading adapters in production: runtime LoRA updates or hub resolvers left on, so anyone with access to the management endpoint, or a compromised hub repository, can change model behaviour without a release.
• Leaked repository write tokens: a Hugging Face or registry token with write access in a notebook or repository lets an outsider replace a dataset or adapter your pipeline pulls, as the Lasso findings showed was possible at scale.
• No lineage after merging: adapters merged into checkpoints with no record of which data and adapter produced them, so a suspected backdoor cannot be traced or scoped.
• Trusting public benchmark scores: adopting a community fine-tune because it ranks well, when OWASP LLM03 describes models tuned to pass benchmarks while carrying targeted triggers.

Practitioner checklist

• Inventory all fine-tuned models and adapters with base model, data sources, owner and deployment location.
• Version every training dataset immutably and separate data contribution from dataset approval.
• Scan training data for secrets and personal data, revoke any secret found and record the lawful basis for personal data.
• Run training jobs under a dedicated, scoped identity with confused-deputy conditions, encryption and private networking.
• Store artefacts in safetensors, sign lineage records and verify signatures and base model revision at deployment.
• Disable runtime adapter loading and hub resolvers on production inference servers.
• Gate every release on safety regression, memorisation and task red-team results compared with the base model.
• Replace long-lived hub and registry write tokens with scoped, short-lived credentials.
• Log dataset writes, job starts, registry changes and adapter loads, and alert on unregistered adapters.
• Keep the previous model version deployable and rehearse rollback.

Standards and references

• OWASP: Top 10 for LLM Applications 2025
• NIST: AI 100-2 E2025 Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
• MITRE ATLAS: Poison Training Data
• vLLM: LoRA Adapters
• AWS: Access and security for model customization in Amazon Bedrock
• Microsoft Learn: Safety evaluation for fine-tuning
• arXiv: Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To
• arXiv: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Related NHI Mgmt Group resources: AI Data Poisoning and Backdoors Guide · AI Model Security Guide · LLM Security Guide · Open-Weight Model Security Guide