Poor training data teaches models to reproduce unsafe or buggy patterns at scale. In code generation, that means insecure libraries, weak error handling, and flawed logic can be amplified into many outputs. The control point is the dataset itself. If the model learns from cleaner examples, its generated code is more likely to be secure and reliable.
Why This Matters for Security Teams
Poor training data is not just a model-quality issue. For code-generating systems, it becomes a security issue because the model learns what “normal” code looks like and then reproduces that pattern at scale, including insecure imports, weak exception handling, hard-coded secrets, and brittle authentication logic. NIST’s NIST Cybersecurity Framework 2.0 treats trustworthy development as a control objective, not a side effect, and that logic applies directly to AI-generated code.
NHIMG research shows why this matters in practice: the 12,000 Secrets Found in Public LLM Training Dataset finding demonstrates that sensitive patterns can be absorbed from training material and later resurfaced in generated output. Once those patterns are embedded, downstream teams inherit a defect factory rather than a productivity gain. In practice, many security teams encounter the risk only after unsafe code has already been copied into multiple repositories and release branches.
How It Works in Practice
The security problem begins upstream. Training data that contains vulnerable code, deprecated crypto, insecure defaults, or exposed credentials teaches the model to treat those patterns as acceptable. During generation, the model does not “understand” security intent in a human sense; it predicts the most likely continuation. If the dataset is noisy, the output is often noisy in exactly the same ways.
That is why the control point is the dataset itself, along with the curation pipeline around it. Security teams should look for three practical safeguards:
- Data hygiene before training, including secret scanning, license checks, malware filtering, and removal of known vulnerable snippets.
- Provenance and labeling, so high-trust sources can be separated from scraped or community-contributed code.
- Post-training evaluation, using secure coding test suites to measure whether the model keeps reproducing unsafe constructs.
This also affects AI-assisted development workflows. If an organisation feeds internal repositories into fine-tuning without cleaning them first, it can amplify old mistakes such as weak input validation or privilege misuse across new codebases. Guidance from the State of Secrets in AppSec is relevant here because 43% of security professionals are already concerned about AI systems learning and reproducing sensitive information patterns from codebases. That concern is well founded, and it aligns with broader software assurance thinking in the NIST Cybersecurity Framework 2.0 and current secure-by-design guidance.
Current best practice is to treat model training data like production code input: validate it, classify it, remove sensitive material, and continuously test the output for regressions. These controls tend to break down when teams fine-tune on large, inherited code archives because the provenance is unclear and insecure patterns are already deeply embedded.
Common Variations and Edge Cases
Tighter data curation often increases cost and slows model development, so organisations have to balance output quality against the operational burden of cleaning and maintaining datasets. That tradeoff is real, especially for teams using large internal code corpora or vendor-supplied base models.
There is no universal standard for how much cleaning is enough. Current guidance suggests a risk-based approach: high-impact systems should use stricter filtering, stronger provenance controls, and more aggressive secret removal than low-risk internal assistants. A model used to draft infrastructure, auth, or payment code deserves more scrutiny than one used for generic documentation tasks.
Edge cases matter. A dataset can look “safe” but still encode anti-patterns through repetitive examples, outdated framework versions, or insecure workaround code that passes linters yet fails security review. This is why organisations should pair dataset governance with human review and secure coding benchmarks. The Ultimate Guide to NHIs — Key Research and Survey Results is also relevant when training data includes automation scripts, tokens, or service account material, because security gaps in machine identities often appear as code-generation failures later on.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | LLM07 | Training data quality drives unsafe code reproduction and insecure outputs. |
| CSA MAESTRO | T1 | Covers data supply-chain risks that shape agent and code-gen behaviour. |
| NIST AI RMF | MAP | Risk mapping is needed to identify training-data-driven security failure modes. |
| NIST CSF 2.0 | ID.AM-6 | Data and software inventory supports visibility into model training inputs. |
| OWASP Non-Human Identity Top 10 | NHI-01 | Secret leakage in training data can become reusable credential exposure in outputs. |
Filter training corpora for insecure patterns, then test generated code for security regressions.
Related resources from NHI Mgmt Group
- Why do training data changes create security risk in AI systems?
- Why do AI-generated code pipelines create more security risk than traditional development?
- Why do SaaS, cloud, and generative AI environments create harder data security problems?
- Why do AI-generated code and security review at scale create new risk even when individual outputs improve?
