Differentially Private SGD

• DPSGD is a stochastic optimization method that enforces (ε, δ)-differential privacy by clipping per-example gradients and adding calibrated Gaussian noise.
• It introduces clipping bias and privacy-induced variance, with theoretical guarantees of almost sure convergence under both nonconvex and strongly convex settings.
• Recent research improves DPSGD through algorithmic variants, system optimizations, and fairness-aware techniques to balance privacy, utility, and computational efficiency.

Searching arXiv for foundational and recent DPSGD papers. Differentially Private Stochastic Gradient Descent (DPSGD, often written DP-SGD) is a stochastic optimization method for training machine learning models under formal differential privacy guarantees. In its standard form, DPSGD modifies stochastic gradient descent by clipping per-example gradients to bound sensitivity and then adding Gaussian noise before the parameter update. It has become the standard algorithm for training machine learning models with rigorous privacy guarantees (Mukherjee et al., 20 Nov 2025), following the formulation introduced in "Deep Learning with Differential Privacy" (Abadi et al., 2016). Contemporary research treats DPSGD simultaneously as a privacy mechanism, an optimization algorithm with clipping-induced bias and privacy-induced variance, a systems problem dominated by per-example gradient computation, and an inference problem in which privacy, utility, stability, fairness, and uncertainty quantification interact in nontrivial ways (Mukherjee et al., 20 Nov 2025, Xu et al., 11 Nov 2025, Xia et al., 28 Jul 2025).

1. Core algorithm and mathematical formulation

For unconstrained minimization, the objective is

$\min_{x \in \mathbb{R}^d} f(x),$

with two regimes often distinguished in the literature: smooth nonconvex objectives bounded below, and $\mu$-strongly convex objectives with unique minimizer $x^*$ (Mukherjee et al., 20 Nov 2025). In empirical risk minimization, the objective takes the form

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$

where $D=\{\xi_i\}_{i=1}^n$ is the training dataset (Xu et al., 11 Nov 2025).

Vanilla stochastic gradient descent updates

$$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$

with $\nabla f(x_t)=\mathbb{E}[g_t]$ under the stochastic oracle model (Mukherjee et al., 20 Nov 2025). DPSGD alters the gradient computation in two stages. First, each per-sample gradient is clipped at threshold $q$ or $C$: $$\operatorname{clip}_q\big(\nabla f(x_t;\xi_t)\big) = \min\left(1,\frac{q}{\|\nabla f(x_t;\xi_t)\|}\right)\nabla f(x_t;\xi_t),$$ or equivalently, in minibatch notation,

$\mu$0

This enforces a norm bound on each contribution (Mukherjee et al., 20 Nov 2025, Xu et al., 11 Nov 2025, Abadi et al., 2016).

Second, Gaussian noise is added. In one common notation,

$\mu$1

and the update becomes

$\mu$2

In minibatch form, the private averaged gradient is

$\mu$3

followed by

$\mu$4

(Mukherjee et al., 20 Nov 2025, Xu et al., 11 Nov 2025).

A central quantity in recent theory is the clipping probability

$\mu$5

which captures how often the stochastic gradient exceeds the clipping threshold (Mukherjee et al., 20 Nov 2025). This makes explicit that DPSGD differs from non-private SGD in two mathematically distinct ways: clipping introduces systematic bias, while Gaussian perturbation contributes zero-mean noise.

A recurrent misconception is that DPSGD is merely SGD with additive noise. The literature summarized here does not support that simplification. The clipping operator is nonlinear, and several analyses identify clipping bias as an independent and often dominant effect relative to the Gaussian perturbation (Xu et al., 11 Nov 2025, Watson et al., 2023).

2. Differential privacy mechanism and privacy accounting

The standard privacy definition used in DPSGD is $\mu$6-differential privacy: $\mu$7 for all neighboring datasets $\mu$8 differing in at most one entry and all measurable sets $\mu$9 (Mukherjee et al., 20 Nov 2025). The Gaussian mechanism calibrates noise to the $x^*$0-sensitivity of the clipped query. In one statement recalled for DPSGD,

$x^*$1

while in another formulation,

$x^*$2

for a single Gaussian release (Mukherjee et al., 20 Nov 2025, Xu et al., 11 Nov 2025).

In practice, DPSGD privacy accounting is not based on a single iteration but on composition across many noisy updates. The literature in the data block emphasizes amplification by subsampling together with moments accountant or Rényi differential privacy accounting, in the spirit of Abadi et al. (Xu et al., 11 Nov 2025, Abadi et al., 2016). One sufficient condition quoted for a $x^*$3-step subsampled procedure with sampling probability $x^*$4 is

$x^*$5

for constants $x^*$6, ensuring overall $x^*$7-DP when $x^*$8 (Xu et al., 11 Nov 2025).

Recent work also studies alternative privacy formalisms layered onto SGD. One line develops local differential privacy and Gaussian differential privacy for one-pass online noisy SGD, rather than the central-DP, multi-epoch model characteristic of standard DPSGD (Xie et al., 13 May 2025). Another develops output-specific individual $x^*$9-DP for DP-SGD and shows that most examples can enjoy stronger privacy guarantees than the worst-case bound (Yu et al., 2022).

The distinction between central and local privacy is substantive. Central DPSGD assumes a trusted training mechanism that sees raw data and privatizes the released model; local-DP variants privatize each user contribution before it reaches the server (Xie et al., 13 May 2025). By contrast, the standard DPSGD literature represented here follows the central model (Abadi et al., 2016, Mukherjee et al., 20 Nov 2025).

3. Optimization theory and convergence properties

Recent theory has substantially sharpened the asymptotic understanding of DPSGD. "Almost Sure Convergence Analysis of Differentially Private Stochastic Gradient Methods" proves that DP-SGD converges almost surely under standard smoothness assumptions, both in nonconvex and strongly convex settings, provided the step sizes satisfy standard decaying conditions (Mukherjee et al., 20 Nov 2025). For nonconvex objectives, the target is gradient vanishing,

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$0

while under strong convexity the goal is

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$1

The main step-size schedule used in that analysis is

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$2

implying the Robbins–Monro conditions

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$3

These are also the explicit assumptions in the last-iterate almost-sure convergence theorem (Mukherjee et al., 20 Nov 2025).

A key analytical device is the energy-like quantity

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$4

and, in the strongly convex case,

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$5

These encode both true optimization progress and clipping effects via $f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$6 (Mukherjee et al., 20 Nov 2025).

Under $f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$7-smoothness, directional invariance, and decaying step sizes, the paper proves

$f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$8

in the nonconvex case, and an analogous result for $f(x) = \frac{1}{n} \sum_{\xi \in D} f(x,\xi),$9 under strong convexity (Mukherjee et al., 20 Nov 2025). For the momentum variants DP-SHB and DP-NAG, it further proves last-iterate almost sure convergence: $D=\{\xi_i\}_{i=1}^n$0 which implies $D=\{\xi_i\}_{i=1}^n$1 and $D=\{\xi_i\}_{i=1}^n$2 in the strongly convex setting (Mukherjee et al., 20 Nov 2025).

These results are stronger than earlier guarantees in expectation or with high probability. The distinction matters because almost sure convergence is pathwise: it concerns the behavior of single random trajectories rather than averages over runs (Mukherjee et al., 20 Nov 2025).

The convex low-noise literature studies a different regime. "Differentially Private Stochastic Gradient Descent with Low-Noise" analyzes gradient-perturbed SGD under $D=\{\xi_i\}_{i=1}^n$3-DP for convex pointwise and pairwise learning and derives fast excess risk rates in realizable settings (Wang et al., 2022). In the smooth pointwise case with $D=\{\xi_i\}_{i=1}^n$4, it gives

$D=\{\xi_i\}_{i=1}^n$5

and analogous fast rates for pairwise learning, including non-smooth losses via Hölder smoothness (Wang et al., 2022). This suggests that, in favorable low-noise convex regimes, the leading asymptotic term can be privacy-dominated rather than sample-variance-dominated.

A separate 2025 line develops asymptotic inference for DP-SGD under randomized subsampling, rather than only cyclic subsampling. It proves that

$D=\{\xi_i\}_{i=1}^n$6

with asymptotic variance decomposing into statistical, sampling, and privacy-induced components (Xia et al., 28 Jul 2025). This variance decomposition is one of the clearest formal statements of how DP noise enters the asymptotics of averaged SGD.

4. Bias, noise, geometry, and algorithmic variants

A major theme in the recent literature is that DPSGD utility degradation cannot be attributed to Gaussian noise alone. "Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering" formalizes two sources of degradation: DP noise and clipping bias (Xu et al., 11 Nov 2025). For vanilla DPSGD, it gives the expected squared gradient-norm bound

$D=\{\xi_i\}_{i=1}^n$7

where the last term captures sampling variance and clipping bias effects (Xu et al., 11 Nov 2025).

That paper proposes DP-PMLF, combining per-sample momentum

$D=\{\xi_i\}_{i=1}^n$8

with a low-pass filter

$D=\{\xi_i\}_{i=1}^n$9

followed by normalization and update (Xu et al., 11 Nov 2025). The theoretical claim is an improved convergence bound under DP guarantees, and the empirical claim is an improved privacy–utility trade-off relative to several DPSGD variants (Xu et al., 11 Nov 2025).

A different line, "DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning," argues that standard DP-SGD wastes privacy budget on a repeatedly observed common gradient direction, especially early in training (Liu et al., 2024). It decomposes each gradient into a component parallel to a base direction $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$0 and an orthogonal component: $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$1 privatizes the orthogonal component and the scalar coefficient separately, then reconstructs the gradient (Liu et al., 2024). The paper states that this yields the same formal $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$2-DP type as standard DP-SGD but better utility when gradients are coherent.

The loss-landscape literature offers a different diagnosis. "Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent" separates the effects of clipping and Gaussian noise and reports that, in its experiments, noise alone is often tolerable while clipping has the larger impact (Watson et al., 2023). It gives a clipped-gradient alignment bound involving

$$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$3

showing that greater per-example gradient dispersion weakens alignment between the clipped aggregate and the true mean direction (Watson et al., 2023). The same work argues that magnitude pruning is a useful dimension-reduction technique for DPSGD and reports that heavy pruning can improve test accuracy (Watson et al., 2023).

Other variants modify optimization rather than representation. SA-DPSGD adds a simulated-annealing-based accept–reject filter to candidate noisy updates, with acceptance probability

$$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$4

where $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$5 depends on the number of accepted updates (Fu et al., 2022). The paper treats the screening step as post-processing and therefore not a source of extra privacy cost (Fu et al., 2022).

These variants share a premise: the privacy mechanism is fixed by clipping and noise, but the optimization path can be materially improved by reducing variance before clipping, filtering noise after aggregation, exploiting structure in gradient trajectories, or screening bad steps.

5. Systems, preprocessing, and implementation practice

DPSGD is computationally distinctive because it requires per-example gradients rather than only batch-averaged gradients. "Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization" identifies this as the main source of large runtime overheads and shows that vectorization, just-in-time compilation, and static graph optimization can reduce those costs dramatically (Subramani et al., 2020). The paper reports that its JAX and TensorFlow implementations achieve up to 50x speedups over prior alternatives (Subramani et al., 2020). A representative headline comparison at batch size 128 reports per-epoch runtimes such as 0.53 s for a private MNIST CNN versus 6.50 s for the best alternative private implementation, and 8.2 s for a private LSTM versus 407 s for the best alternative private implementation (Subramani et al., 2020).

Several works in the data block emphasize that DPSGD performance is highly sensitive to preprocessing and architectural choices. "On the effect of normalization layers on Differentially Private training of deep Neural networks" argues that normalization layers significantly affect utility under DPSGD and proposes a way to integrate batch normalization without additional privacy loss by using a small public dataset for normalization statistics and avoiding private running averages (Davody et al., 2020). In that work, BN-LeNet-5 on MNIST achieved 97.61% at $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$6, compared to 94.11% for the baseline DPSGD model, and 88.15% at $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$7, compared to 78.96% for baseline (Davody et al., 2020).

Feature preprocessing can also be intrinsic to the optimization problem rather than just the architecture. "The importance of feature preprocessing for differentially private linear optimization" argues, for linear classification, that private feature preprocessing is vital and proposes an algorithm also named DPSGD-F that combines DPSGD with feature preprocessing, replacing dependence on the maximum feature norm by dependence on the feature diameter (Sun et al., 2023). This suggests that, unlike in non-private optimization, preprocessing can change not only optimization conditioning but the privacy-sensitive geometry of the problem.

Hyperparameter selection is itself constrained by privacy. "Efficient Hyperparameter Optimization for Differentially Private Deep Learning" formulates DPSGD tuning as a black-box optimization problem over the noise multiplier $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$8 and learning rate $$x_{t+1} = x_t - \alpha_t g_t,\qquad g_t = \nabla f(x_t;\xi_t),$$9, with reward

$\nabla f(x_t)=\mathbb{E}[g_t]$0

and compares evolutionary, Bayesian, and reinforcement-learning-based search to grid search (Priyanshu et al., 2021). The paper emphasizes that repeated use of private data for tuning is itself a privacy concern and that adaptive search can reduce the number of required DPSGD runs (Priyanshu et al., 2021).

A practical misconception is that DPSGD hyperparameters can be inherited unchanged from non-private SGD. The papers summarized here do not support that view. Both the hyperparameter-optimization literature and the fairness literature argue that DP training has materially different dynamics, so non-DP hyperparameters are often unreliable under privacy constraints (Priyanshu et al., 2021, Demelius et al., 2 Oct 2025).

6. Fairness, individual privacy, and statistical inference

DPSGD has been studied not only for average utility but for how privacy loss and performance are distributed across groups and examples. "Private and Fair Machine Learning: Revisiting the Disparate Impact of Differentially Private SGD" shows that disparate impact is metric-dependent: a disparity on one metric does not imply a disparity on another (Demelius et al., 2 Oct 2025). It further concludes that tuning hyperparameters directly on differentially private models does not reliably mitigate disparate impact, although it can improve utility–fairness trade-offs relative to reusing non-private hyperparameters (Demelius et al., 2 Oct 2025).

An earlier line, "Removing Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracy," formalizes group-specific cost of privacy in terms of noise variance and clipping bias. For a group $\nabla f(x_t)=\mathbb{E}[g_t]$1, it gives the bound

$\nabla f(x_t)=\mathbb{E}[g_t]$2

and proposes a fairness-aware variant with adaptive group-specific clipping bounds (Xu et al., 2020). This work uses “equality of privacy impact” to describe equalized utility loss across groups when moving from non-private to private training (Xu et al., 2020).

Individual privacy heterogeneity has also been formalized. "Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent" introduces output-specific individual $\nabla f(x_t)=\mathbb{E}[g_t]$3-DP and shows that most examples enjoy stronger privacy guarantees than the worst-case bound (Yu et al., 2022). It also finds that training loss and the privacy parameter of an example are well-correlated, and reports, for CIFAR-10, that the average $\nabla f(x_t)=\mathbb{E}[g_t]$4 of the class with the lowest test accuracy is 44.2% higher than that of the class with the highest accuracy (Yu et al., 2022). This indicates that the examples and groups that are hardest to fit can simultaneously receive weaker utility and weaker realized privacy.

Finally, recent work has begun to treat DPSGD as an inferential object rather than only an optimizer. "Statistical Inference for Differentially Private Stochastic Gradient Descent" develops plug-in and random-scaling confidence intervals for averaged DP-SGD iterates under randomized subsampling and proves that they attain nominal asymptotic coverage while maintaining privacy (Xia et al., 28 Jul 2025). The plug-in method estimates Hessian and score covariance privately; the random-scaling method uses a functional central limit theorem to build an asymptotically pivotal statistic from the SGD path itself (Xia et al., 28 Jul 2025). This suggests that DP-SGD can support not only private training but uncertainty quantification, at least in strongly convex finite-dimensional settings.

Taken together, these results imply that DPSGD is no longer viewed solely as a privacy-preserving optimizer. It is also a distribution-sensitive mechanism whose clipping and noise can affect groups differently, a pathwise stochastic process with nontrivial almost-sure behavior, and a basis for asymptotically valid private statistical inference (Mukherjee et al., 20 Nov 2025, Yu et al., 2022, Xia et al., 28 Jul 2025).