Application teams should redact or tokenize sensitive fields before the prompt is sent to the model, then only pass the minimum context needed for the task. This reduces exposure of personal data, limits unintended retention in downstream systems, and lowers compliance risk. The control works best when built into the application flow, not left to developers to apply manually each time.
How to keep customer data out of prompts without breaking the workflow
The practical goal is to separate task context from raw customer content. Teams should classify which fields are actually required for the model to complete the job, then strip, redact, or tokenize everything else before the prompt is assembled. That means treating prompt construction as a controlled application step, not a developer habit or a late-stage cleanup task.
The strongest implementations push this filtering into shared middleware, prompt builders, or service-layer policy so the rule is enforced consistently across features. That matters because ad hoc handling tends to fail first in edge cases, new endpoints, and fast-moving product changes, exactly where sensitive data is most likely to leak into logs, traces, caches, or downstream model output.
One useful design pattern is to pass a stable internal identifier or synthetic token instead of the original customer value, then resolve only the minimum needed detail outside the model path when the workflow genuinely requires it. Where the model does not need the original field, do not reconstruct it for convenience. McKinsey AI platform breach and 12,000 Secrets Found in Public LLM Training Dataset both reinforce how quickly high-value data can spread once it enters an LLM-adjacent workflow.
Risk and Threat Considerations
If sensitive customer data reaches prompts, it can be copied into logs, telemetry, vendor retention stores, cached completions, evaluation datasets, or human review queues. The exposure is often wider than the original business use case, because the prompt becomes a transport layer for data that was never meant to leave the application boundary.
Failure mechanism: Weak prompt assembly, insufficient field-level filtering, or overbroad context enrichment allows personal data, payment-related details, or other regulated fields to be sent to the model even when the task only needs a summary, classification, or lookup token.
Impact: The result is avoidable privacy exposure, greater breach blast radius, and a harder compliance position because the organisation can no longer credibly claim that the model only saw minimum necessary data. Once the data is in the prompt path, it can also be duplicated into places that are much harder to purge than the source system.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST AI RMF, NIST AI 600-1, CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 — Prompt Injection and Data Exfiltration | Prompt handling must prevent sensitive data from reaching model inputs. |
| Recommendation — Filter and minimize prompt content before model submission. | ||
| NIST AI RMF | GOV — GOVERN | Prompt data minimisation is an AI governance control for accountable deployment. |
| Recommendation — Define governance rules for what data may enter model prompts. | ||
| NIST AI 600-1 | MAP — Map | Mapping data flows is necessary to locate where sensitive data enters GenAI workflows. |
| Recommendation — Document prompt data flows and identify sensitive-field handling points. | ||
| CIS Controls v8 | 16 — Application Software Security | Embedding sanitisation into application flow is a secure-development control. |
| Recommendation — Build prompt sanitization into the application layer and test it continuously. | ||
| NIST CSF 2.0 | PR.DS — Data Security | Keeping customer data out of prompts protects data confidentiality and minimization. |
| Recommendation — Apply data-security controls to restrict what reaches LLM prompts. | ||
Practitioner Guidance
What to verify: Check the assembled prompt, not just the source code, and prove that blocked fields cannot re-enter through templates, retries, debug output, or fallback paths. The control is only trustworthy when the final payload is inspected at the boundary where it is actually sent.
Implementation sequence: Start with a field inventory, map each field to a business necessity, then encode the rule in a shared prompt-construction layer with tests for permitted and forbidden data. Add monitoring for prompt-size anomalies and high-risk field names so regressions surface quickly when product teams change workflows.
Practitioner takeaway: The key decision is whether the model truly needs the original customer value; if it does not, remove it before assembly and make the reduction enforceable by design, not dependent on manual discipline.
Related resources from NHI Mgmt Group
- How should security teams govern sensitive data in LLM workflows?
- How should security teams prevent sensitive data from leaking through AI prompts and copilots?
- How should security teams prevent sensitive data from reaching SIEM and storage in cleartext?
- How should security teams prevent LLM memory from leaking sensitive data?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 20, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
