How should security teams govern custom foundation model training on proprietary data?

Why This Matters for Security Teams

Custom foundation model training turns proprietary data into model behaviour, so the security problem is not just access to files but control over how data is selected, labeled, mixed, and reused. That makes the workflow closer to identity governance than a conventional ML project. If teams do not control training inputs and run privileges, they can unintentionally encode sensitive content, biased signals, or malicious poison into a model that later appears trustworthy. NIST’s NIST AI 600-1 Generative AI Profile treats these risks as lifecycle issues, not one-time approval events. For NHI governance, the same pattern shows up in training orchestration, dataset access, experiment tracking, and model registry permissions. The organisation needs to know who can trigger a run, which service accounts are used, and whether those credentials are rotated and scoped correctly. NHIMG’s Top 10 NHI Issues highlights how weak lifecycle control and over-privileged non-human access become root causes when workflows scale faster than governance. In practice, many security teams discover training contamination only after a model has already been promoted into production, not during the approval step.

How It Works in Practice

Security teams should govern training as a chain of controlled actions, with each step tied to an accountable identity and a reviewable data lineage record. That starts with defining approved data sources, then enforcing who can request a training job, which datasets can be mounted, and what secrets or tokens the training platform may use. A model trained on proprietary data should inherit only the permissions needed for the job, not broad platform access. A practical control pattern looks like this:

• Require explicit approval for each training dataset, including source, owner, retention, and sensitivity classification.
• Bind every training run to a non-human identity with least privilege, short-lived credentials, and strong audit logging.
• Separate raw data access from training execution so developers cannot casually move data into experiments.
• Validate reward signals, labeling pipelines, and synthetic data sources before they influence the model.
• Record lineage from input data to checkpoint to deployment so later investigations can trace behaviour back to a specific run.

This is where guidance from the NIST Cybersecurity Framework 2.0 and the NHIMG Ultimate Guide to NHIs — Lifecycle Processes for Managing NHIs becomes operational: treat training access as a governed lifecycle, not a standing entitlement. Teams should also test post-training behaviour against leakage, memorisation, prompt injection carryover, and unsafe retrieval paths before deployment. These controls tend to break down in federated training environments because data ownership, identity scope, and audit logging are often split across teams and platforms.

Common Variations and Edge Cases

Tighter training governance often increases friction for research teams, so organisations have to balance speed against assurance. Current guidance suggests that the right control level depends on the sensitivity of the data and the blast radius of a bad model, but there is no universal standard for this yet. Public, internal, and highly sensitive datasets should not be governed the same way. For lower-risk corpora, lightweight approval and automated scanning may be enough. For regulated or proprietary data, training should include stronger segregation, documented lineage, and formal review of reward signals and fine-tuning sources. This is especially important when teams use third-party labs, managed platforms, or shared GPU environments, where NHIs and service credentials can be exposed in ways that resemble broader NHI failures described in NHIMG research and in the Ultimate Guide to NHIs — Regulatory and Audit Perspectives. Another edge case is continuous training. If models are updated frequently, approval cannot rely on a one-time intake review because the training set changes with every run. In those environments, governance should be policy-driven, with automated checks for dataset drift, secret exposure, and ownership changes. Security teams also need to account for “shadow” experimentation, where developers copy sensitive data into notebooks or ad hoc pipelines outside the official training path. That is where a reference like the DeepSeek breach is useful as a warning signal, because training-time exposure can become a systemic data-governance failure, not just an ML quality issue.

Related resources from NHI Mgmt Group

• How should security teams govern unstructured data for GenAI use cases?
• How should security teams govern non-human identities at scale?
• How should security teams govern non-human identities for compliance?
• How should security teams govern non-human identities in Salesforce?