AI systems collect, memorise and infer personal data in ways that ordinary data protection controls were never designed to handle. Scraped training sets contain names and opinions, models can regurgitate what they were trained on, chatbots collect prompts full of customer data, and models increasingly shape decisions about people. This guide covers the privacy questions security, privacy and engineering teams face across the AI lifecycle: lawful basis for training, whether a trained model is still personal data, privacy attacks such as membership inference and training data extraction, data subject rights and machine unlearning, automated decision-making, and how the EU AI Act sits alongside the GDPR. It is written for practitioners who build, buy or govern AI systems in the EU and UK.
Key takeaways
- A model trained on personal data is not anonymous by default. The EDPB says anonymity must be shown case by case, with the likelihood of extracting personal data, directly or through queries, being insignificant.
- Legitimate interest can work for AI training, but only with evidence. Regulators expect a documented three-step test, a real necessity case for web scraping and mitigations such as opt-outs and source exclusions.
- Privacy attacks are a security problem with privacy consequences. Membership inference, attribute inference and training data extraction turn a model or its API into a disclosure channel, so test for them before release.
- Erasure has to reach every copy. Source tables, training snapshots, embeddings, vector indexes, fine-tuned weights and prompt logs all hold personal data; plan suppression, deletion and retraining as one process.
- The AI Act adds to the GDPR, it does not replace it. Prohibitions, transparency duties and high-risk obligations run alongside lawful basis, DPIAs and Article 22, and the Digital Omnibus has moved several dates.
Where personal data enters the AI lifecycle
Map each stage separately, not just the training set, because the EDPB treats development and deployment as potentially separate purposes with separate lawful bases.
| Stage | Typical personal data | Main legal question | Key controls |
|---|---|---|---|
| Collection and web scraping | Names, contact details, posts, images, special category data in public pages | Lawful basis (Art. 6), Art. 9 conditions, transparency under Art. 14 | Source exclusions, robots.txt and ai.txt respect, opt-out list, collection criteria published |
| Data preparation | Raw records, labels, annotator notes | Minimisation (Art. 5(1)(c)), retention | PII detection and masking, deduplication, pseudonymisation, documented retention |
| Training and fine-tuning | Information memorised in weights | Is the model itself personal data? | Differential privacy, regularisation, extraction and inference testing |
| Retrieval (RAG) and embeddings | Documents, tickets, emails converted to vectors | Purpose limitation, access control, erasure | Permission-aware retrieval, index deletion on erasure |
| Inference and prompts | Prompts, uploaded files, conversation history | Transparency, retention, processor terms (Art. 28) | Input DLP, no-training contract terms, short log retention |
| Outputs and decisions | Inferences, scores, generated text about people | Accuracy (Art. 5(1)(d)), Art. 22, AI Act Art. 86 | Output filters, human review that can change outcomes, explanations |
Lawful basis and legitimate interest for training
Consent rarely works for large training sets, so most developers rely on legitimate interest under Article 6(1)(f). The EDPB's Opinion 28/2024 (adopted 17 December 2024) sets out how supervisory authorities should test that claim, and makes clear legitimate interest is not a default.
- Step 1, legitimate interest: the interest must be "clearly and precisely articulated" and "real and present, not speculative". The EDPB's own example is developing a conversational agent to assist users.
- Step 2, necessity: show there is "no less intrusive way" of pursuing the purpose.
- Step 3, balancing: weigh the impact on individuals, including their reasonable expectations. The EDPB states that meeting GDPR transparency requirements "is not sufficient in itself" to make processing expected.
The UK ICO reached a similar position in the outcomes of its generative AI consultation series. It describes web scraping as a "large-scale processing activity that often occurs without people being aware of it", says "it is for developers to demonstrate the necessity of web scraping", and warns developers "may struggle to demonstrate" that they pass the balancing test without evidence of effective mitigations.
Mitigations regulators recognise
- Collection limits: exclude categories of data or sources, exclude content that could put people at risk, and respect robots.txt or ai.txt signals (Opinion 28/2024, para. 105).
- Rights beyond the minimum: an "unconditional 'opt-out' from the outset", an opt-out list managed by the controller before collection, erasure even where Article 17(1) grounds do not strictly apply, and a reasonable gap between collection and training (paras. 102 and 106).
- Data treatment: pseudonymisation that prevents combining data on individual identifiers, and masking or substituting personal data with fake values in the training set (para. 101).
- Transparency: publish collection criteria and datasets used, model cards and transparency labels (para. 103).
Special category data: scraped pages often reveal health, religion or sexual orientation, which needs an Article 9 condition as well as an Article 6 basis. For bias work specifically, the Digital Omnibus on AI (Regulation (EU) 2026/1744) deleted Article 10(5) of the AI Act and added Article 4a, which lets providers of high-risk systems "exceptionally process special categories of personal data, subject to appropriate safeguards" where strictly necessary for bias detection and correction. The safeguards include showing synthetic or anonymised data would not work, pseudonymisation, access controls, no onward transfer, deletion once the bias is corrected and records explaining the necessity. See the Privacy-Enhancing Technologies Guide.
Is a trained model personal data?
Models can memorise. In "Extracting Training Data from Large Language Models" (Carlini et al., 2020), researchers recovered hundreds of verbatim sequences from GPT-2, including names, phone numbers and email addresses, some of which appeared in only one training document. The EDPB therefore says "AI models trained on personal data cannot, in all cases, be considered anonymous" (para. 34).
A model can be treated as anonymous only where the likelihood of direct, including probabilistic, extraction of personal data about training subjects is insignificant, and so is the likelihood of obtaining it through queries (para. 43). Authorities will look at evidence in five areas:
- Source selection and steps to avoid or limit collection of personal data.
- Data preparation and minimisation, including filtering and deduplication.
- Methodological choices, such as privacy-preserving techniques like differential privacy.
- Measures on the model itself, such as output filters.
- Structured testing against attacks including attribute and membership inference and exfiltration, with documentation of the model's resistance to re-identification.
Why it matters downstream: if a model was developed unlawfully and is not anonymous, the problem travels. The EDPB says a deployer that obtains such a model should assess how it was built and "could not ignore that the initial processing was unlawful" (para. 129). Corrective powers include erasure of the dataset or the model and ordering retraining (para. 114). If the model is genuinely anonymised before deployment, the GDPR does not apply to its later operation. Ask vendors for this evidence during procurement.
Privacy attacks on models
NIST AI 100-2 E2025 (Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations) groups these as privacy attacks. The ICO distinguishes white box attacks, which need the model itself, from black box attacks, which only need query access; an API removes the first risk but not the second.
| Attack | What it reveals | Access needed | Main mitigations |
|---|---|---|---|
| Membership inference | Whether a specific person's record was in the training set (sensitive if the dataset is, say, a clinic's patients) | Black box queries, more effective with confidence scores | Differential privacy, do not return confidence values, regularisation, query monitoring |
| Attribute inference and model inversion | Sensitive attributes of people the attacker partly knows | Black or white box | Remove or generalise sensitive features, differential privacy, output limits |
| Training data extraction and regurgitation | Verbatim personal data from training text | Black box prompting | Deduplication, PII scrubbing before training, output filters, canary testing |
| Data reconstruction | Individual records rebuilt from aggregate outputs | Repeated queries on released statistics or models | Differential privacy with a tracked budget, query limits |
| Model extraction | The model's parameters or behaviour, which enables white box attacks later | High-volume queries | Rate limits, anomaly detection on API usage, contractual controls |
| Prompt and context leakage | Another user's data, system prompt contents, retrieved documents | Ordinary use or prompt injection | Session isolation, permission-aware retrieval, OWASP LLM02:2025 controls |
The ICO also warns that some model types, such as support vector machines and k-nearest neighbours, contain parts of the training data "in its raw form" by design, so publishing those models publishes personal data. OWASP LLM02:2025 notes that system prompt instructions restricting what a model returns "may not always be honored and could be bypassed via prompt injection", so prompts are not a privacy control on their own. See the Permission-Aware RAG Guide.
Where identity fits
RAG pipelines and AI agents read personal data through service accounts, API keys and OAuth grants. If that identity can read the whole CRM, the model can surface any of it to any user, and a stolen key becomes a bulk export path. Scope agent and retrieval identities to the user's own permissions, log every data access by the agent identity, and include agent credentials in incident response. The Spain AEPD AI agent breach is an early example of an agent incident involving credentials and personal data being notified to a data protection authority. See the Agentic AI Identity Guide.
Data subject rights and machine unlearning
Access (Art. 15), rectification (Art. 16), erasure (Art. 17) and objection (Art. 21) all apply where a model or its pipeline processes personal data. The ICO expects controllers to "design and build systems that implement the data protection principles effectively", and says organisations relying on Article 11 (cannot identify the data subject) must show that reliance is justified.
- Output suppression: filters stop a name or identifier appearing in outputs. Fast, but the data stays in the weights, so treat it as interim.
- Index and log deletion: for RAG and chat history, delete the source document, its embeddings and cached responses. This is complete and cheap, a strong reason to prefer retrieval over fine-tuning for personal data.
- Retraining without the data: the only complete remedy for memorised data in weights. Sharded training, where each shard trains a sub-model, limits how much must be retrained.
- Approximate unlearning and post-training suppression: techniques that adjust weights to remove or suppress specific data; the EDPB lists post-training techniques among deployment mitigations (para. 107(b)). Verify results with the same extraction tests used before release.
Automated decisions, profiling and explanations
Article 22(1) GDPR gives people the right "not to be subject to a decision based solely on automated processing, including profiling," with legal or similarly significant effects. Exceptions are contract necessity, Union or Member State law, and explicit consent; for the first and last, the controller must provide at least the right to obtain human intervention, express a point of view and contest the decision. Special category data can only be used under Article 9(2)(a) or (g). In SCHUFA (C-634/21, 7 December 2023) the Court of Justice treated an automatically generated credit score as a decision in its own right where a lender draws strongly on it, so upstream scoring vendors cannot assume Article 22 is someone else's problem.
- UK position: the Data (Use and Access) Act 2025 replaced Article 22 in the UK GDPR with Articles 22A to 22D. The ICO's draft guidance (updated 31 March 2026) lists the Article 22C safeguards: information about the decision, the ability to make representations, human intervention and the ability to contest, with responses due within one month.
- Meaningful human review: the reviewer must have authority and information to change the outcome. A reviewer who only confirms the model's output leaves the decision solely automated.
- AI Act Article 86: people affected by decisions based on listed high-risk systems can obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure".
- Impact assessments: under Article 27 of the AI Act, public bodies, private entities providing public services and deployers of credit scoring and life and health insurance pricing systems must carry out a fundamental rights impact assessment, which "shall complement" the Article 35 DPIA rather than replace it.
The AI Act alongside the GDPR
Article 113, as amended by the Digital Omnibus on AI (in force 27 July 2026), sets staggered dates. The provisions with the most privacy relevance:
| Provision | Privacy relevance | Applies from |
|---|---|---|
| Art. 5(1)(e), (f), (g) prohibitions | Untargeted scraping of facial images to build recognition databases; emotion inference at work and in education (except medical or safety); biometric categorisation to infer race, politics, union membership, religion, sex life or orientation | 2 February 2025 |
| Art. 5(1)(ba) and (bb), added by the Omnibus | Systems generating non-consensual intimate imagery of identifiable people, and child sexual abuse material | 2 December 2026 |
| Chapter V, general-purpose AI models | Providers must document models and publish a summary of training content | 2 August 2025 |
| Art. 50 transparency | Tell people they are interacting with AI; machine-readable marking of synthetic content; inform people exposed to emotion recognition or biometric categorisation; disclose deepfakes | 2 August 2026 (the Omnibus gives generative systems already on the market until 2 December 2026 for marking) |
| High-risk systems in Annex III | Data governance (Art. 10), logging, human oversight, FRIA (Art. 27) | 2 December 2027 |
| High-risk systems in Annex I products | As above, for AI in regulated products | 2 August 2028 |
Article 10(3) requires high-risk training data to be "relevant, sufficiently representative, and to the best extent possible, free of errors and complete", which pulls against minimisation, so record why each attribute is needed. See the Agentic AI Compliance Guide.
Choosing an approach
| Scenario | Lean towards | Trade-off |
|---|---|---|
| Staff using a hosted chatbot or copilot | Enterprise terms with no training on customer data, Art. 28 processor terms, short retention, input DLP | Less flexibility than consumer tools; blocking consumer tools without an alternative drives shadow use |
| Answering questions over internal or customer records | Retrieval with permission checks rather than fine-tuning | More infrastructure, but erasure and access control stay at document level |
| Fine-tuning on support transcripts or CRM data | Legitimate interest with a written assessment, masking, opt-out, extraction testing | Erasure may require retraining; budget for it |
| Training on scraped web data | Only with a documented necessity case and the full para. 105 mitigation set | Highest regulatory exposure in both the EU and UK |
| Releasing a model or statistics derived from sensitive data | Differential privacy with a tracked epsilon, or no public release | Accuracy loss; NIST notes differentially private models may be less accurate |
| Decisions about individuals (credit, hiring, benefits) | Human decision-maker with model as input, Art. 22 safeguards, DPIA and FRIA | Slower; still needed where the AI Act classes the use as high-risk |
Small organisations are usually deployers: focus on vendor terms, DPIAs and log retention. Model builders also need a privacy red team, dataset lineage and an unlearning pipeline.
Implementing an AI privacy programme
- Inventory first: list every AI use case, model, vendor and dataset, including shadow tools found through proxy and SaaS logs. See the Shadow AI Discovery Guide.
- Assign roles: GDPR controller or processor, and AI Act provider or deployer, for each use case.
- Document lawful basis per stage: separate assessments for development and deployment, with the three-step test where legitimate interest is used.
- Run a DPIA for any use case involving profiling, special category data, large-scale scraping or decisions about people, and an Article 27 FRIA where required.
- Prepare data: apply source exclusions, PII detection and masking, deduplication and retention limits before training; snapshot and version datasets.
- Test before release: run membership inference and extraction tests, including inserted canary strings, and gate the release on agreed thresholds. Deploy output filters in log-only mode to a pilot group first, review what they catch, then switch to blocking.
- Operate: rate limit queries, remove confidence scores from external APIs and restrict prompt log access and retention.
- Handle rights: a runbook that suppresses outputs within days, deletes from indexes and logs, and queues retraining where weights are affected.
- Plan rollback: keep the previous model version and dataset snapshot. If testing or a complaint shows a model leaking personal data, withdraw that version, revert, retrain without the data and assess notification under Articles 33 and 34.
Detection and measurement
- Log: API query volumes and patterns per client (systematic, high-volume queries suggest extraction or membership inference), output filter hits, DLP detections in prompts and uploads, retrieval access by agent identities, and model version per response so incidents can be traced.
- Protect the logs: prompt and output logs are personal data in their own right; restrict access and set short retention.
Metrics that show the programme is working:
- Percentage of AI use cases with a current DPIA and lawful basis record (target 100% for production).
- Extraction and membership inference test pass rate per model release.
- Median days from an erasure request to suppression, and to full removal from indexes and retrained models.
- PII detections per 1,000 prompts sent to external AI services, trending down after training and DLP.
- Share of AI vendors with contractual no-training, retention and sub-processor terms.
- Number of decisions about individuals where human reviewers changed the model's recommendation (zero over a long period suggests rubber-stamping).
Framework mapping
| Control area | GDPR / UK GDPR | EU AI Act | NIST SP 800-53 Rev. 5 | Other |
|---|---|---|---|---|
| Lawful basis and purpose | Art. 5(1)(b), 6, 9 | Art. 4a (bias data) | PT-2, PT-3 | EDPB Opinion 28/2024 |
| Impact assessment | Art. 35 | Art. 27 | RA-8 | ISO/IEC 27001:2022 A.5.34 |
| Data quality and minimisation | Art. 5(1)(c), (d), 25 | Art. 10 | PM-22, SI-12 | CIS Controls v8.1 Control 3 |
| Model privacy attacks | Art. 32 | Art. 15 (robustness, high-risk) | SI-4 | NIST AI 100-2 E2025, OWASP LLM02:2025 |
| Rights and erasure | Art. 15 to 17, 21 | n/a | SI-18 | ICO AI and data protection guidance |
| Automated decisions | Art. 22 (UK: 22A to 22D) | Art. 14, 86 | n/a | NIST CSF 2.0 GV.OC-03 |
| Transparency | Art. 13, 14 | Art. 50 | PT-5 | Model cards |
Common failure patterns
- Assuming the model is anonymous: teams delete the training set and declare the model out of scope, without extraction testing or documentation. The EDPB expects evidence, not assertion.
- Staff pasting personal and confidential data into public tools: the Samsung ChatGPT leak led to a ban rather than a governed alternative, which tends to push use underground.
- Unprotected conversation stores: chat histories are a concentrated personal data set. The DeepSeek database exposure leaked chat logs from an open ClickHouse database.
- Erasure that stops at the source system: the record is deleted from the CRM but survives in embeddings, cached answers, fine-tuned weights and prompt logs.
- Confidence scores in public APIs: the ICO notes models can be "disproportionately confident" about people in the training data, which is exactly what membership inference exploits.
- Rubber-stamp human review: reviewers without time, information or authority to overturn outputs, which leaves the decision solely automated under Article 22.
Practitioner checklist
- Maintain an inventory of AI use cases, models, datasets and vendors, with controller and provider roles recorded.
- Record a lawful basis per lifecycle stage, with a three-step legitimate interest assessment where relied on.
- Run a DPIA for every AI use case involving profiling, special category data, scraping or decisions about people.
- Apply source exclusions, PII masking and deduplication before training, and version every dataset.
- Test each model release for membership inference and training data extraction, and keep the results.
- Remove confidence scores and rate limit external model APIs; monitor for systematic querying.
- Prefer permission-aware retrieval over fine-tuning for personal data that may need correcting or erasing.
- Build one erasure runbook covering sources, indexes, logs, caches and model weights.
- Give human reviewers of automated decisions the authority and information to change outcomes.
- Track AI Act dates for prohibitions, Article 50 transparency and high-risk obligations against your use cases.
Standards and references
- EDPB: Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models
- EUR-Lex: Regulation (EU) 2024/1689 (Artificial Intelligence Act)
- EUR-Lex: Regulation (EU) 2026/1744 (Digital Omnibus on AI)
- ICO: Response to the consultation series on generative AI
- ICO: How should we assess security and data minimisation in AI?
- ICO: What are the ADM safeguards?
- NIST: AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- OWASP: LLM02:2025 Sensitive Information Disclosure
Related NHI Mgmt Group resources: GDPR Compliance Guide for Security Teams · Privacy-Enhancing Technologies Guide · Enterprise AI Copilot Security Guide · Agentic AI Compliance Guide · Shadow AI Discovery Guide. This guide summarises GDPR, UK GDPR and EU AI Act provisions relevant to AI and personal data and is not legal advice.