AI systems can encode, retain, and regenerate information in ways that do not map to classic file or database controls. Once data enters prompts, training sets, or stored conversations, it may persist outside normal deletion and access revocation paths. That creates a governance problem, not just an encryption problem, because exposure can reappear through later prompts or outputs.
Why This Matters for Security Teams
AI systems change the protection problem because sensitive data is no longer confined to a file, table, or application record. Prompts, embeddings, conversation memory, fine-tuning corpora, and tool outputs can all become secondary data stores, which means access control, deletion, and retention rules no longer behave the way they do in traditional applications. That is why this is not only a confidentiality issue but a lifecycle and governance issue.
NHI Management Group has documented how AI-adjacent exposure shows up in real environments, including cases where secrets and sensitive records were left in training or operational data paths, such as the DeepSeek breach. The broader pattern is consistent with current guidance from the NIST Cybersecurity Framework 2.0: asset control and data governance must extend beyond the original system boundary. In practice, many security teams encounter AI data exposure only after a model has already memorised, reproduced, or redistributed sensitive content rather than through intentional data classification.
How It Works in Practice
Traditional applications usually protect sensitive data by controlling where it is stored, who can query it, and when it is deleted. AI systems introduce additional paths where data can persist or reappear. A user may paste regulated content into a prompt, a retrieval layer may index it, a logging pipeline may store it, and a model may later echo it back in an output. Each step creates a new control point, but also a new exposure path.
Security teams usually need to treat AI workflows as data processing chains, not single applications. That means combining classification, minimisation, and access controls with model-specific safeguards. Current practice commonly includes:
- Preventing sensitive data from entering prompts unless there is a clear business need and approved retention policy.
- Filtering logs, traces, and conversation histories so secrets and regulated data are not retained by default.
- Separating training, evaluation, and production contexts so one environment does not inherit data from another.
- Using policy checks for retrieval and tool use, so the model cannot surface data it should not see.
- Testing for prompt leakage and memorisation risk as part of assurance, not as an afterthought.
NHIMG research on The State of Secrets in AppSec shows why this matters operationally: 43% of security professionals are already concerned about AI systems learning and reproducing sensitive information patterns from codebases, which reflects the gap between intent and actual behaviour. The challenge is amplified by the speed of abuse; NHIMG notes in LLMjacking: How Attackers Hijack AI Using Compromised NHIs that exposed AWS credentials can be attempted within minutes. These controls tend to break down when AI tools are granted broad workspace access because the model can aggregate and re-emit data across sources faster than reviewers can detect it.
Common Variations and Edge Cases
Tighter AI data controls often increase operational friction, requiring organisations to balance model usefulness against privacy, compliance, and developer speed. That tradeoff becomes most visible when teams want copilots to answer from internal documents, customer records, or source code.
There is no universal standard for this yet, but current guidance suggests a few important distinctions. First, not all AI systems pose the same risk: a chat assistant with no memory is easier to govern than an agent connected to email, ticketing, and storage systems. Second, sensitive data can be exposed even when the model itself is not compromised, because the retrieval layer, telemetry store, or prompt history may be the weak point. Third, deletion is harder than in traditional applications because data may exist in multiple derived forms, including cached embeddings, logs, and generated outputs.
For that reason, organisations should avoid assuming that encryption alone solves the problem. The stronger pattern is to limit what enters the AI workflow, restrict what the model can retrieve, and define retention rules for every downstream copy. The Ultimate Guide to NHIs – Key Research and Survey Results is useful here because it frames AI-connected identities and machine access as a governance surface, not just an application feature. In practice, the hardest cases are systems that mix long-lived memory with broad tool access and unreviewed external data sources.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-01 | AI data paths expand NHI exposure across prompts, logs, and retrieval layers. |
| OWASP Agentic AI Top 10 | A2 | Agentic systems can reproduce or expose sensitive data through tool use and memory. |
| NIST AI RMF | AI RMF addresses governance for data risk, retention, and unintended disclosure. |
Use AI RMF governance to inventory AI data stores and define retention and disclosure rules.
