Join our Newsletter — 33% off our NHI Course
Home› Guides› Differential Privacy Guide: Set Budgets, Choose Mechanisms and…
Guide Privacy & Data Protection

Differential Privacy Guide: Set Budgets, Choose Mechanisms and Avoid Hazards

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 13 November 2026 15 min read
On this page

Differential privacy is often sold internally as a checkbox, yet a release with the wrong unit of privacy, an unbounded budget or a home-made noise function gives almost no protection. This guide goes deeper than our overview of privacy-enhancing technologies: it explains how to choose epsilon, delta and the unit of privacy, how to pick between the central and local models, which mechanisms and open source libraries to use, and how to evaluate a deployment against NIST SP 800-226. It uses the 2020 US Census and published device telemetry programmes as worked examples, and covers analytics, synthetic data and machine learning training. It is written for privacy engineers, data platform teams, security architects and the reviewers who must sign off a differentially private release.

Key takeaways

  • The guarantee is about releases, not storage. Differential privacy limits what an output reveals about any one person; it does nothing for the raw data during collection, storage or analysis, so access control and security still carry that risk.
  • Three numbers define the promise. Epsilon, delta and the unit of privacy must be stated together; NIST SP 800-226 recommends user-level privacy as the default and treats delta above 10-5 as suspicious.
  • Budgets add up. Privacy loss composes across every query on the same people, so a programme needs an accountant and a hard stop, not per-query reviews.
  • Use tested libraries. Randomised code is easy to get wrong; floating point, timing and caching side channels have broken naive implementations, which is why SP 800-226 points practitioners to well-tested libraries.
  • Accuracy is a design input. The Census Bureau set accuracy targets with data users first and then tuned parameters to meet them; doing it the other way round produces releases nobody can use.

What differential privacy promises, and what it does not

Differential privacy (DP) is a mathematical definition: an analysis is differentially private if its outcome is nearly equally likely whether or not any one person's data is included. For two neighbouring datasets that differ by one individual, the probability of any output can differ by at most a factor of eepsilon. NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (final, March 2025), frames this as a measure of how much privacy a person gives up by being included.

  • It protects against: inferences that depend on one individual's record, including linkage, differencing (subtracting two aggregates to isolate one person) and reconstruction attacks that rebuild microdata from many published tables.
  • It does not protect against: conclusions drawn from population patterns. If a study shows smokers in a region have high cancer rates, that finding applies to a smoker whether or not they took part, and DP is not designed to stop it.
  • It is not automatically anonymisation: the ICO's guidance says DP output can be anonymous information if enough noise is added and the residual risk is remote, but the source data held by the curator remains personal data. See the Anonymisation and Re-identification Risk Guide.
  • It survives post-processing: any computation on a DP output stays DP. SP 800-226 warns that forcing a DP release to agree with other, non-DP figures is not post-processing in this sense and breaks the guarantee.

For where DP sits beside clean rooms, homomorphic encryption and federated learning, see the Privacy-Enhancing Technologies Guide.

Choosing the parameters: epsilon, delta and the unit of privacy

SP 800-226 places these at the top of its "differential privacy pyramid", because every lower layer inherits whatever they promise. There is no formula for choosing epsilon: NIST calls it an open research question that needs expert judgement, and recommends publishing the parameters so others can hold the release to account.

ParameterWhat it controlsNIST SP 800-226 guidanceReview question
Epsilon (privacy loss)Maximum ratio by which one person can shift output probabilities; smaller means more noise and stronger privacy0.1 generally strong; below 1 considered reasonable; many deployments use values between 1 and 20, and 10 or more can leave outliers exposedIs this the total for the whole release programme, or per query, per day or per donation?
DeltaSmall probability that the epsilon bound fails outrightAt most 1/n2 (or 1/(n log n)); above 10-5 needs justification; use only when a mechanism needs itCould a delta failure release a record without noise?
Unit of privacyWhat "one person's data" means: an event, a user, a user-dayUser-level is the recommended default; event-level can be surprisingly weakHow many records can one person contribute, and is that bounded over the whole period?
Neighbouring definitionBounded (dataset size fixed) or unbounded (add or remove a person)Bounded DP does not protect dataset size; some mechanisms need itIs the total count itself sensitive?
VariantPure DP, approximate (epsilon, delta), Rényi DP, zero-concentrated DP (zCDP), Gaussian DPPrefer RDP, zCDP or GDP where possible and report the native parameters; conversion to (epsilon, delta) is lossyHas a small epsilon been produced by quietly choosing a large delta?

Watch the unit: a guarantee of epsilon 1 per user-day can add up to 365 over a year for a daily active user. Apple's published local DP programme states budgets per donation, for example epsilon 4 with at most two donations a day for Lookup Hints and epsilon 8 with two a day for QuickType, so a reviewer must multiply by the donation rate and retention period (Apple states up to three months) to see the effective loss.

Trust models: central, local and the deployment architecture

Where the noise is added decides who must be trusted and how much accuracy you get. The ICO calls these global and local DP; SP 800-226 calls them the central and local trust models.

ModelWho sees raw dataAccuracy for the same epsilonTypical useSecurity work it shifts
Central (trusted curator)The curator's platformHigh: noise added once to aggregatesOfficial statistics (2020 US Census), internal analytics, DP SQL over a warehouseCurator store needs strict access control, logging and breach protection, because a raw data leak voids the guarantee for those records
LocalNobody: each device randomises before sendingLow: needs very large populationsDevice telemetry (Apple QuickType, emoji and Safari usage)Client code integrity, no identifiers in reports, collection endpoint hygiene
Interactive query serviceCurator; analysts see only noisy answersDepends on query mixAnalyst sandboxes over sensitive dataBudget enforcement per dataset and per analyst, collusion between analysts
Non-interactive releaseCurator; public sees fixed tables or synthetic dataPlanned in advancePublished statistics, open data, synthetic microdataRelease review and versioning of the whole budget

The ICO notes that an interactive system must stop answering once the budget is spent, and that colluding parties could pool their answers, so per-analyst budgets alone are not enough. SP 800-226's bottom layer (query model, trust model, side channels, security and access control, data collection exposure) is where most real deployments fail, because it is ordinary security engineering that the mathematics assumes is done.

Mechanisms and the privacy budget

Core mechanisms

  • Laplace mechanism: adds Laplace noise scaled to the L1 sensitivity of the query divided by epsilon, giving pure (epsilon, 0) DP. Often the more accurate choice for low-dimensional outputs such as one count.
  • Gaussian mechanism: adds Gaussian noise scaled to L2 sensitivity, giving approximate DP or zCDP but not pure DP. Often better for high-dimensional outputs such as many cells or model gradients.
  • Discrete mechanisms: the Census Bureau used a discrete Gaussian distribution for the 2020 Census because its thinner tails make large errors rarer than the geometric (discrete Laplace) alternative for comparable disclosure risk.
  • Randomised response and sketches: the building blocks of local DP. Apple's Count Mean Sketch hashes input into a matrix, flips each bit with a probability tied to epsilon and sends one row; its Hadamard variant sends a single bit.

Sensitivity and contribution bounding

Noise must cover the most one person can change the result. That requires clipping values (a salary sum needs an upper bound) and bounding contributions (at most k rows per user). OpenDP documents this as identifier truncation. Bounding moves an event-level system towards user-level protection, but SP 800-226 notes that per-day or per-month bounds are weaker than a bound over the whole dataset.

Composition and accounting

Each release spends budget, and the spends add up: ten analyses at epsilon 0.1 each cost epsilon 1 in total under basic composition. Tighter accounting under zCDP or Rényi DP gives more useful releases for the same guarantee, which is why the 2020 Census budget was expressed as rho (the zCDP parameter) rather than epsilon. The Census Bureau's Data Stewardship Executive Policy Committee set that budget, which is a governance lesson in itself: someone accountable must own the total.

Worked example: the 2020 US Census TopDown Algorithm

The Census Bureau's disclosure avoidance system for 2020 is the largest published central DP deployment and shows how the abstract choices become engineering decisions.

  1. Measure: tabulations are built from the Census Edited File microdata and noise is added to their cells within the privacy-loss budget.
  2. Follow a hierarchy: processing runs down a geographic spine from nation to regions, divisions, states, counties, tracts, block groups and blocks. Blocks get the most relative noise; tracts and above are relatively more accurate. Off-spine areas such as school districts can be noisier.
  3. Hold invariants: state total populations, housing units per block and counts of occupied group quarters facilities by type were published without noise. Every invariant is a deliberate exception to the guarantee and must be justified.
  4. Post-process: optimisation makes counts non-negative and consistent (counties sum to states). The Bureau acknowledged that clipping and small-area noise create bias, which SP 800-226 also discusses.
  5. Tune to accuracy targets: targets were agreed with internal and external data users, demonstration data built from 2010 records was released for testing, and some tables were dropped because publishing more requires more noise.

Status note: the Census Bureau's differential privacy page now carries a banner saying it is no longer current and that the Bureau is evaluating alternative approaches to comply with a Department of Commerce administrative order prohibiting noise infusion. Treat the 2020 system as a documented reference design, not as a statement of current Bureau policy.

Analytics, synthetic data and machine learning

Use caseApproachTooling examples (open source)Main pitfall
Counts, sums, averages over a warehouseDP aggregations with per-user contribution boundsGoogle DP building block libraries (C++, Go, Java), ZetaSQL DP command line, PipelineDP4j on Beam or Spark, Privacy on Beam (Go)Averages and min/max need clipping bounds chosen without looking at the data
Grouped statistics with unknown keysPrivate partition (key) selection, which needs deltaSame libraries; OpenDPPublishing a key that only one person has, which reveals that person exists
Synthetic microdataFit DP marginals, then sample recordsOpenDP synthetic data guidesAssuming synthetic means safe; non-DP synthesisers can memorise records. See the Synthetic Data Risk Guide
Model trainingDP-SGD: clip each example's gradient, add Gaussian noise, track cumulative loss with an accountantOpacus for PyTorch (PrivacyEngine, make_private with noise_multiplier and max_grad_norm); dp_accounting libraryTuning epsilon after training for accuracy, and fine-tuning a public model on private data without counting the private steps
Text, images, videoUnit of privacy is hard to defineResearch stageOne person appears in many documents or frames

For model training, the threat DP addresses is membership inference and memorisation, catalogued in MITRE ATLAS as AML.T0137.000 Infer Membership. DP training bounds that risk for the training set; it does not stop prompt-time leakage of data placed in a context window. See the AI and Privacy Guide.

Evaluating a deployment and rolling it out

  1. Write the privacy statement first: epsilon, delta, variant, unit of privacy, neighbouring definition, time period and what is invariant. If the team cannot write it in one paragraph, the design is not ready.
  2. Set accuracy targets with users: agree the error tolerances (for example, relative error at the 95% confidence level for each published cell) before tuning, as the Census Bureau did.
  3. Choose a library, not a formula: OpenDP (Python, R, Rust), Google's libraries or a platform's built-in DP SQL. Record the version; libraries fix side-channel and floating point issues over time.
  4. Pilot on old or synthetic data: run the pipeline on a past dataset, publish demonstration outputs internally and let analysts test fitness for use.
  5. Test the implementation: use statistical testers (Google's stochastic tester, DP Auditorium) to catch regressions. SP 800-226 is clear that empirical auditing can find bugs but cannot prove the guarantee.
  6. Lock down the bottom of the pyramid: restrict who can query raw data, log every query and budget spend, separate the curator role from analysts, and remove raw extracts after the release.
  7. Release with a ledger: record budget spent per dataset and per release; a new request that exceeds the total goes to the budget owner, not the engineer.
  8. Roll back cleanly: you cannot unpublish a release, so the rollback is to stop further releases, revoke query access and re-plan the remaining budget. Version every published artefact so a flawed release can be identified and withdrawn from distribution.

Decision guidance

  • Central or local: choose central when you already hold the data and can secure it; choose local only for very large populations where you do not want to hold raw values at all.
  • Small organisations: a DP query layer over a warehouse with a fixed annual budget per dataset is achievable; custom mechanisms and synthetic data programmes need specialist staff.
  • Large organisations: centralise the accountant and budget ownership; otherwise each team spends the same people's budget independently.
  • When not to use DP: the ICO notes it suits broad trends but not anomaly detection or detailed patterns; fraud investigations and individual-level decisions need other controls.

Detection, measurement and framework mapping

Monitor the budget ledger, queries against raw tables, and any release that bypassed the DP pipeline. Useful metrics:

  • Budget consumed: cumulative epsilon (or rho) per dataset against the approved total.
  • Coverage: share of external statistical releases produced by the DP pipeline rather than ad hoc extracts.
  • Accuracy: share of published cells meeting the agreed error target.
  • Raw access: number of people and service accounts with direct read access to curator data, trending down.
  • Exceptions: number of invariants or non-DP side releases, each with a recorded justification.
FrameworkReferenceHow DP relates
NIST SP 800-53 Rev. 5SI-19 De-identification; SI-19(6) Differential PrivacyControl enhancement specifically for adding noise to query outputs
NIST SP 800-53 Rev. 5SI-19(4) Removal, Masking, Encryption, Hashing, or Replacement of Direct IdentifiersComplementary, weaker technique often wrongly treated as equivalent
NIST SP 800-53 Rev. 5PM-25 Minimization of PII Used in Testing, Training, and Research; AC-21 Information SharingDP releases and DP training as a way to meet minimisation and sharing decisions
NIST CSF 2.0PR.DS-01, PR.DS-10; GV.OC-03Protecting the curator store and data in use; privacy obligations understood
MITRE ATLASAML.T0137.000 Infer MembershipThreat that DP-SGD bounds
NIST Privacy FrameworkDisassociated processing outcomesSee the NIST Privacy Framework 1.1 Guide

Common failure patterns

  • Per-event budgets presented as per-person: a small epsilon per transaction says little about a customer with thousands of transactions.
  • Unbounded composition: each team or dashboard refresh spends budget with no global ledger, so the effective epsilon grows without anyone noticing.
  • Home-made noise: sampling Laplace noise with standard floating point arithmetic can leak information through the pattern of representable values; use a maintained library.
  • Tuning on the private data: choosing clipping bounds, bins or hyperparameters by looking at the sensitive data spends privacy that is never counted.
  • Forgetting the raw store: DP release pipelines sit on top of a database holding everything. In the 23andMe breach, attackers who reached accounts could see profile data of genetic relatives, close to 7 million people; no statistical release control would have helped, because the exposure came through account access to raw records.
  • Side releases of exact figures: publishing a non-DP total next to DP cells can let readers difference out the noise; SP 800-226 notes that releasing non-DP statistics does not become safer because DP ones sit beside them.

Practitioner checklist

  • Write and publish a privacy statement with epsilon, delta, variant, unit of privacy and time period for every release programme.
  • Default to user-level privacy and bound each person's contributions over the whole dataset.
  • Keep delta at or below 1/n2 and justify anything above 10-5.
  • Assign a named budget owner and keep a ledger of every spend per dataset.
  • Use a maintained DP library, record its version and run a statistical tester in CI.
  • Agree accuracy targets with data users before tuning parameters.
  • Treat the curator store as high-value data: least privilege, query logging, short retention of extracts.
  • List every invariant and every non-DP side release with a justification.
  • For DP-SGD, fix the privacy target before training and account for every private fine-tuning step.
  • Assess with your DPO whether the output is anonymous information or still personal data.

Standards and references

Related NHI Mgmt Group resources: Privacy-Enhancing Technologies Guide · Anonymisation and Re-identification Risk Guide · Synthetic Data Risk Guide · AI and Privacy Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 13 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.