How do you test a trained LLM for patient data leakage?

How to tell whether the model is leaking memorised patient data

Testing starts with a controlled prompt set that tries to elicit memorised clinical text rather than general medical knowledge. Use canary strings, direct extraction prompts, and variants that target rare phrases, identifiers, and record-specific combinations. The goal is to see whether the model reproduces training examples, not whether it can answer clinical questions plausibly.

Pair that with a held-out clinical set and an explicit leakage threshold defined before testing begins. That gives you a repeatable benchmark for deciding whether a response is merely similar to training data or crosses into disclosure of patient-linked material.

Because leakage often shows up as partial recall, test for both exact and near-exact reproduction. A model may not output a full chart note verbatim, but it can still expose enough unusual detail to identify a patient or a specific encounter.

What to probe for in outputs and evaluations

Look for direct identifiers first, then the subtler signals that indicate memorisation. Names, dates, locations, case numbers, uncommon comorbidities, and rare wording are all useful indicators, especially when they appear together in a response that should have been generic. When you compare outputs against your held-out set, focus on overlap in distinctive phrases and structured facts, not just topical similarity.

A practical test suite should include prompts that ask for summaries, restatements, continuation of a chart fragment, and “what was the exact wording” style requests. These are useful because leakage often appears when the model is nudged to continue a remembered sequence rather than generate a fresh answer from general training.

If your system uses retrieval or long-context augmentation, also test the boundaries between what came from the model and what came from the retrieved source. A response can look like model leakage when it is actually an over-shared retrieval result, so the evaluation needs to separate training-memory disclosure from retrieval overexposure.

How to interpret leakage results without overreacting

Not every suspicious match means the system is unsafe, but repeated reproduction of rare patient-linked text is a red flag. A single exact phrase may be coincidence; multiple distinctive overlaps, especially across different prompts, usually indicate that memorised data is surfacing in a way you should treat as material leakage. The most important question is whether the model can reveal information that was not supposed to be recoverable by an ordinary user prompt.

That is why the threshold matters. Predefining what counts as a leak avoids ad hoc judgment after the fact, and it makes it easier to compare model versions, fine-tuning runs, or prompt policies over time. If the model crosses the threshold, the issue is not just quality, it is exposure of sensitive clinical content that may require investigation and containment.

Risk and Threat Considerations

Patient data leakage is risky because an LLM can disclose sensitive clinical facts even when the caller does not know the underlying record. The practical danger is not only verbatim memorisation, but also re-identification through rare combinations of symptoms, timing, treatment details, and wording that point back to a specific person.

Failure mechanism: The model has absorbed low-frequency patient text during training or fine-tuning, then reproduces it when prompted with extraction, continuation, or canary-style probes. If the system also over-shares retrieved context, the same test can surface leakage from the surrounding data layer rather than from the model weights alone.

Impact: You can expose protected health information, violate data-minimisation expectations, and create downstream privacy and compliance risk. In practice, confirmed leakage usually means you need to review the training corpus, fine-tuning data, prompt history, and any retrieval path that could have amplified the disclosure.

Practitioner Guidance

What to verify: Make sure the test set includes canaries, rare phrases, and realistic patient-style prompts that reflect how users actually interact with the model. If your evaluation only uses generic medical questions, you will miss the failure mode that matters most.

Decision rule: If the model reproduces patient-linked text above the pre-set leakage threshold, treat it as a control failure and investigate the training data, not just the prompt. If the result is ambiguous, rerun the same prompts with perturbations and compare consistency across multiple attempts.

What good looks like: The model answers clinically without echoing unique patient text, and repeated probing does not increase the amount of recovered sensitive detail. A stable model should show low overlap with held-out records and no uncontrolled disclosure of direct identifiers or distinctive chart language.

Practitioner takeaway: The safest evaluation is one that assumes the model may remember more than it should, then proves that memory is bounded before you trust it with clinical workflows.

Related resources from NHI Mgmt Group

• Guide: Permission-Aware RAG Guide
• Why do LLM applications create new data leakage risks for identity teams?
• Who is accountable for LLM data leakage in an organisation?
• How should security teams test for LLM data poisoning before deployment?
• Why do AI agents and LLM applications increase the risk of unauthorized access and data leakage?