How should organisations reduce security risk when fine-tuning code generation models on mixed-quality training data?

Why This Matters for Security Teams

Fine-tuning on mixed-quality code is not a harmless efficiency step. If the dataset includes vulnerable patterns, deprecated APIs, hardcoded secrets, or insecure defaults, the model can learn those patterns as normal and reproduce them at scale. That turns training-data curation into a model-risk control. NIST Cybersecurity Framework 2.0 is a useful anchor here because it treats governance, identification, and protection as operational disciplines, not one-time checks. NHIMG’s research on the 12,000 Secrets Found in Public LLM Training Dataset shows how quickly training data can inherit avoidable exposure when curation is weak.

The practical issue is that code generation models do not just copy syntax. They absorb style, library usage, and implicit trust boundaries from the examples they see. If low-quality code is blended with high-quality code, the resulting model can become inconsistent in ways that static tests miss until after deployment. Security teams should therefore treat dataset selection, validation, and redaction as part of secure model development, alongside code review and release gating. In practice, many teams discover unsafe model behaviour only after the first developer starts using the fine-tuned system in production workflows.

How It Works in Practice

The safest pattern is to build a curation pipeline before fine-tuning begins. Start by filtering out obvious risk classes such as secrets, known vulnerable snippets, abandoned dependencies, and code that fails baseline linting or unit tests. Then score the remaining corpus against security and maintainability rules so the model trains on examples that reflect the organisation’s approved engineering standard. This aligns with the spirit of the OWASP NHI Top 10, especially where secret leakage and insecure tool use are symptoms of poor upstream hygiene.

Operationally, teams get better results when they separate data quality checks into layers:

• Source filtering: exclude repositories, files, or commits with known security issues.
• Static analysis: scan for vulnerable constructs, secrets, and policy violations.
• Human review: sample borderline cases where automation is uncertain.
• Dataset versioning: record what was removed, why it was removed, and who approved it.
• Evaluation: test the fine-tuned model against prompts that should trigger unsafe code patterns.

For organisations that want a broader control baseline, the NIST Cybersecurity Framework 2.0 supports this as a governance and protection activity, while NHIMG’s Top 10 NHI Issues highlights how weak secret handling and overexposure often recur when teams do not enforce lifecycle controls. These controls tend to break down when training data is pulled from multiple product teams with inconsistent code review standards because the curation rules become impossible to apply uniformly.

Common Variations and Edge Cases

Tighter dataset filtering often increases cost and reduces training volume, so organisations have to balance model breadth against security confidence. That tradeoff is real: aggressive removal can improve safety while also shrinking the variety of patterns the model learns. Best practice is evolving, and there is no universal standard for how much low-quality code can remain without materially affecting risk.

Mixed-quality data becomes especially difficult when the objective is domain adaptation rather than general code generation. In those cases, some older or weaker code may be necessary to preserve niche API coverage, but it should be isolated, labelled, and evaluated separately rather than blended into the main corpus. The Ultimate Guide to NHIs — Key Research and Survey Results is a useful reminder that weak governance is common, so assumptions about “trusted” internal code should be verified, not assumed. Where sensitive codebases are involved, current guidance suggests pairing curation with post-training red-team tests that probe for insecure completions, unsafe refactoring, and accidental secret reproduction.

Related resources from NHI Mgmt Group

• How can organisations reduce the risk of secrets in AI training data?
• How should organisations reduce the security risk of ROT data in cloud and SaaS environments?
• How can organisations reduce the risk of source code, credentials, and regulated data leaking into generative AI tools?
• How should security teams reduce identity risk when employees use large language models with sensitive enterprise data?