Federated Learning Privacy: Gradient Leakage, Secure Aggregation and Real Protections

Federated learning keeps training data where it is, but the model updates it shares can still leak that data, so "the data never left the device" is not a privacy guarantee on its own. Federated learning is a way of training one machine learning model across many devices or organisations, where each participant trains locally and sends only model updates to be combined. Federated learning privacy is the set of risks and protections around those updates and the final model: gradient leakage, membership inference, a curious or malicious aggregator, and the techniques that counter them, such as secure aggregation and differential privacy. This guide explains how the attacks work in plain terms, then gives practitioners the design choices, failure modes and checks needed to deploy it safely. It goes deeper than the federated learning section of the Privacy-Enhancing Technologies Guide and is written for privacy engineers, ML platform teams and data protection officers.

Key takeaways

• Updates are data. Research since 2019 shows that shared gradients can be inverted to reconstruct training examples, in some cases pixel-accurate images and token-matching text.
• Two separate problems need two controls. Input privacy (hiding individual updates from the aggregator) comes from secure aggregation, homomorphic encryption or enclaves; output privacy (limiting what the final model reveals) comes from differential privacy.
• Federated learning alone is not enough. The UK ICO states that using it alone may not achieve appropriate protection of personal information, and recommends combining it with other PETs.
• Secure aggregation has a cost. It hides each participant's contribution, but it also stops the server inspecting updates for poisoning, so robustness has to be designed in separately.
• Treat it as personal data processing. Run a DPIA, define controller roles across participants, and document the threat model, privacy budget and aggregation design.

What it is

The term and the standard algorithm come from a 2016 paper by McMahan and colleagues at Google, "Communication-Efficient Learning of Deep Networks from Decentralized Data". It described leaving training data on mobile devices, having each device compute an update to a shared model and combining those updates by iterative model averaging (now usually called Federated Averaging, or FedAvg). The authors reported 10 to 100 times fewer communication rounds than synchronised stochastic gradient descent.

Two deployment patterns are common:

Data can also be split in two ways. In horizontal federated learning every participant holds the same kinds of fields for different people, so updates can be combined by simple addition. In vertical federated learning participants hold different fields about the same people (for example a bank and a retailer with overlapping customers), which NIST notes makes aggregation much more complex and needs different input-privacy techniques.

How it works

A single training round in a horizontal setting runs like this:

1. Distribute: the coordinator (aggregation server) sends the current global model to a selected set of participants.
2. Train locally: each participant runs a few steps of training on its own data and computes an update: the difference between its new weights and the global weights, or its gradients.
3. Protect the update: optionally, the participant clips and adds noise (local or distributed differential privacy) and masks or encrypts the update for secure aggregation.
4. Aggregate: the coordinator combines updates, ideally learning only their sum or average.
5. Update and repeat: the new global model is formed and the next round begins. Central differential privacy noise can be added at this step.

Where privacy leaks

The landmark result is "Deep Leakage from Gradients" (Zhu, Liu and Han, 2019), which recovered training data from shared gradients and described the reconstruction as pixel-wise accurate for images and token-wise matching for text. NIST's 2024 blog series with the UK Centre for Data Ethics and Innovation also cites attacks that reconstruct training data from a finished model's parameters, including extraction from an image-generation diffusion model and GPT-2, and notes that memorisation is a bigger problem for larger models. For the inference side in more depth, see Membership Inference Attacks.

Protecting the updates (input privacy)

• Secure aggregation: participants mask their updates with shared secrets so the server learns only the sum. Bonawitz and colleagues' protocol (Cryptology ePrint 2017/281) tolerates participants dropping out at any time, is proven secure against honest-but-curious and malicious settings, and reported communication expansion of 1.73 times for 2¹⁰ users with 2²⁰-dimensional vectors. NIST notes secret sharing increases communication between participants and that more efficient deployed variants exist. See Secure Multi-Party Computation.
• Homomorphic encryption: participants encrypt updates under a common public key and the aggregator adds ciphertexts. A separate key holder must not collude with the aggregator; threshold schemes split the key among participants to remove that extra party. See the Homomorphic Encryption Guide.
• Trusted execution environments: aggregation runs inside a hardware enclave, which is simpler and faster than cryptography, but security then depends on trusting the CPU and its maker. See the Confidential Computing Guide.

Protecting the model (output privacy)

Differential privacy is the most comprehensive output protection NIST names. In training this usually means clipping each contribution to a maximum norm and adding calibrated Gaussian noise, with the privacy loss tracked as epsilon across all rounds. In federated settings the unit of privacy matters: protecting each record differs from protecting each participant's whole dataset. NIST SP 800-226 explains how to evaluate such guarantees; see the Differential Privacy Guide.

Why it matters

• Regulatory position: model updates derived from personal data can themselves be personal data where individuals can be identified from them. Federated learning supports data minimisation and data protection by design (GDPR Article 25) but does not remove the need for a lawful basis, Article 32 security or a DPIA under Article 35 where risk is high. See the AI and GDPR Guide and the DPIA Guide.
• Collaboration that would otherwise not happen: NIST's running example is a consortium of banks training a fraud model without pooling transactions. The UK-US PETs Prize Challenges produced solutions in this space; for example, winning teams Scarlet Pets and Visa Research used secure aggregation, while PPMLHuskies, ILLIDANLAB and MusCAT used threshold homomorphic encryption.
• Security of the system around the maths: every participant is a client holding credentials to the coordinator. If those credentials are weak or shared, an attacker can enrol fake clients, observe rounds or inject updates, which undermines both privacy and integrity.

Choosing a design

Small organisations joining a consortium should insist on seeing the threat model and privacy budget rather than building their own protocols. Large platform operators should treat the aggregator as a high-value system with separate administration, logging and change control.

Common misconceptions

• "Raw data never leaves, so it is anonymous": updates can be inverted, and the final model can memorise. The ICO's position is that federated learning alone may not provide appropriate protection.
• "Secure aggregation solves privacy": it hides individual updates from the server, but the aggregated model can still leak through membership inference or extraction. You still need output privacy.
• "Differential privacy makes secure aggregation unnecessary": central DP added by the server protects the published model, but the server still sees raw individual updates unless they are also protected.
• "Averaging over many clients hides everyone": with few participants per round, or with a malicious server that controls which clients are selected, individual contributions can be isolated.
• "Poisoning is someone else's problem": once updates are hidden, the server cannot inspect them, so norm bounding and robust aggregation must be designed alongside secure aggregation. See the AI Data Poisoning and Backdoors Guide.

Detection and measurement

• Privacy budget spent: cumulative epsilon and delta per training run, against the documented limit.
• Participants per aggregation round: the minimum observed, against the threshold below which rounds are discarded.
• Memorisation test results: canary exposure or membership inference advantage measured before each model release.
• Anomalous update rate: share of updates exceeding norm bounds or rejected by robust aggregation.
• Client identity hygiene: percentage of participants authenticating with unique, rotated credentials or attested devices.

Related concepts

Practitioner checklist

• Write down the threat model: who might be curious or malicious (coordinator, participants, outsiders) and what each can see.
• Run a DPIA and agree controller and processor roles across all participants before training starts.
• Use secure aggregation, threshold homomorphic encryption or an attested enclave so no party sees individual updates.
• Apply differential privacy with a stated unit of privacy and a fixed epsilon budget, and track it across rounds.
• Set a minimum number of participants per round and discard rounds below it.
• Bound update norms and use robust aggregation to limit poisoning.
• Give each participant a unique, rotated credential or device attestation, and log enrolment and round participation.
• Test the final model for memorisation and membership inference before publishing or exposing it as an API.
• Record which PETs, parameters and libraries were used so the guarantee can be audited.

Standards and references

• NIST: Privacy Attacks in Federated Learning
• NIST: Protecting Model Updates in Privacy-Preserving Federated Learning
• NIST: SP 800-226 Guidelines for Evaluating Differential Privacy Guarantees
• ICO: What PETs are there?
• Bonawitz et al.: Practical Secure Aggregation for Privacy-Preserving Machine Learning
• Zhu, Liu and Han: Deep Leakage from Gradients
• McMahan et al.: Communication-Efficient Learning of Deep Networks from Decentralized Data
• MITRE ATLAS: Exfiltration via AI Inference API

Related NHI Mgmt Group resources: Privacy-Enhancing Technologies Guide · Differential Privacy Guide · Membership Inference Attacks · Secure Multi-Party Computation