D2P2-SGD: Dynamic DP Projected SGD

• The paper introduces D2P2-SGD, which combines dynamic Gaussian noise decay, automatic per-sample gradient clipping, and random projection to balance privacy, utility, and efficiency in high-dimensional settings.
• Experimental results on image classification tasks show that the method improves test accuracy compared to standard DP-SGD, approaching the performance of non-private SGD.
• The approach mitigates the ambient-dimension challenge by reducing the gradient space dimension, offering an effective balance in the dimension–privacy–utility trilemma.

Searching arXiv for the core D2P2-SGD paper and closely related work on dynamic DP-SGD and projected/privacy-aware subspace methods. Dynamically Differentially Private Projected SGD (D2P2-SGD) is a differentially private stochastic optimization method for empirical risk minimization that combines a time-varying Gaussian privacy mechanism, automatic per-sample gradient clipping, and random projection of gradients into a lower-dimensional space before noise injection (Jiang et al., 11 Sep 2025). In the formulation introduced in “Balancing Utility and Privacy: Dynamically Private SGD with Random Projection” (Jiang et al., 11 Sep 2025), the method is designed to address three simultaneous constraints: protecting training data under a global $(\varepsilon,\delta)$-DP guarantee, preserving model utility under repeated noisy updates, and reducing the effective optimization burden in high-dimensional parameter spaces. The optimizer is situated within the broader literature on dynamic DP-SGD schedules (Du et al., 2021), public-data-assisted dimension reduction for DP optimization (Zhou et al., 2020), and dynamic clipping mechanisms (Wei et al., 29 Mar 2025), but its defining combination is dynamic privacy with random gradient projection.

1. Formal setting and conceptual scope

D2P2-SGD is posed in the standard empirical risk minimization setting with private dataset

$$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$

parameter vector $\mathbf{x}\in\mathbb{R}^d$, and objective

$$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$

The analysis in the D2P2-SGD paper considers both generally convex objectives and smooth non-convex objectives. Its standing assumptions are $L$-smoothness,

$$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$

lower boundedness of $f$, and bounded variance of stochastic gradients,

$$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$

The motivating contrast is with conventional DP-SGD. Standard DP-SGD clips per-example gradients and adds Gaussian noise with fixed variance at every step. In the D2P2-SGD account, this static mechanism damages utility in two ways: it distributes privacy cost uniformly across training steps even though optimization sensitivity changes during training, and it injects noise in the full ambient parameter space, which is increasingly problematic as $d$ grows (Jiang et al., 11 Sep 2025).

A terminological distinction is essential. In D2P2-SGD, the adjective “projected” refers to random projection of the gradient into a lower-dimensional space. This is different from the “Projected DP-SGD” of “Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification” (Zhou et al., 2020), where projection is onto a public-data-estimated gradient eigenspace, and it is also different from projected-gradient methods that project parameters onto a constraint set, as discussed in dynamic DP-SGD design analyses (Du et al., 2021). The overlap in terminology has been a recurring source of confusion.

2. Algorithmic mechanism

At iteration $k=1,\dots,K$, D2P2-SGD samples a minibatch $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$0 of size $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$1, computes per-sample gradients, applies automatic clipping by normalization, projects the clipped minibatch gradient into $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$2 with $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$3, adds Gaussian noise in the projected space, maps the result back to $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$4, and performs an SGD step (Jiang et al., 11 Sep 2025).

The clipping rule differs from the conventional hard threshold

$$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$5

D2P2-SGD uses the automatic clipping form

$$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$6

and sets $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$7 in the algorithm. For each sample $$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$8,

$$\mathcal{D} = \{s_1,\dots,s_n\}, \qquad s_i \sim \mathcal{P}\ \text{i.i.d.},$$9

The minibatch gradient is then

$\mathbf{x}\in\mathbb{R}^d$0

This guarantees $\mathbf{x}\in\mathbb{R}^d$1, thereby bounding sensitivity, while avoiding the “lazy region” associated with standard clipping (Jiang et al., 11 Sep 2025).

The projection step uses a Gaussian random matrix $\mathbf{x}\in\mathbb{R}^d$2. The clipped minibatch gradient is projected as

$\mathbf{x}\in\mathbb{R}^d$3

Noise is sampled in the projected space,

$\mathbf{x}\in\mathbb{R}^d$4

and mapped back: $\mathbf{x}\in\mathbb{R}^d$5 The parameter update is

$\mathbf{x}\in\mathbb{R}^d$6

The dynamic privacy component is the time-varying noise schedule

$\mathbf{x}\in\mathbb{R}^d$7

Noise is therefore largest in early iterations and decays as training proceeds. The stated rationale is that early optimization is less sensitive to perturbation because gradients are larger and the iterate is farther from optimum, whereas later optimization benefits more from lower noise (Jiang et al., 11 Sep 2025).

The random projection is motivated by the Johnson–Lindenstrauss lemma: for suitable $\mathbf{x}\in\mathbb{R}^d$8, the compressed representation approximately preserves norms and pairwise distances. In the D2P2-SGD analysis, the Gaussian matrix entries are taken as i.i.d. $\mathbf{x}\in\mathbb{R}^d$9, or more generally $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$0, and the algorithm may use a fresh $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$1 at each step. For large networks, the implementation is described as layer-wise to reduce memory overhead (Jiang et al., 11 Sep 2025).

3. Privacy model and accounting

The paper adopts the standard definition of $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$2-differential privacy: for neighboring datasets $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$3 differing in one record and any measurable event $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$4,

$$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$5

Here $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$6 is the privacy budget and $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$7 bounds catastrophic privacy failure.

The D2P2-SGD privacy proof is formulated through Rényi differential privacy and moments-accountant-style composition. The privacy accountant is written as

$$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$8

with additive composition across iterations. The per-step mechanism is a subsampled Gaussian mechanism, and privacy amplification by subsampling is invoked for sampling rate $$\min_{\mathbf{x}\in\mathbb{R}^d} f(\mathbf{x}) = \frac{1}{n}\sum_{s\in\mathcal{D}} f(\mathbf{x},s).$$9 (Jiang et al., 11 Sep 2025).

The main theorem states that there exist constants $L$0 such that, for any

$L$1

D2P2-SGD is $L$2-DP for any $L$3 if

$L$4

The per-iteration variance is then set to $L$5 (Jiang et al., 11 Sep 2025).

Two points are structurally important. First, the automatic clipping step fixes per-sample sensitivity at a dimension-independent scale, because each normalized gradient has norm at most $L$6. Second, the random projection does not alter privacy accounting. The paper treats the projection and the back-mapping as post-processing relative to the Gaussian mechanism in the projected space, so the DP guarantee is governed by the clipped sensitivity, the Gaussian noise scale, the sampling rate, and the number of iterations, not by the projection dimension $L$7 itself (Jiang et al., 11 Sep 2025).

This dynamic schedule differs from the GDP-based dynamic mechanisms proposed earlier for DP-SGD, where clipping thresholds and per-step privacy parameters are also varied over time under a fixed total budget (Du et al., 2021, Zhu et al., 10 May 2025). D2P2-SGD fixes the automatic clipping form and makes the privacy mechanism dynamic through the explicit decay law $L$8.

4. Utility guarantees and the dimension–privacy–utility trilemma

The theoretical claims for D2P2-SGD are organized around convex and non-convex regimes. In both settings, the paper emphasizes that the optimizer attains sub-linear convergence rates matching the best available rate, despite the addition of dynamic privacy and random projection (Jiang et al., 11 Sep 2025).

For convex, $L$9-smooth objectives, the paper gives an expected suboptimality bound for the averaged iterate $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$0 that contains three qualitatively distinct contributions: an initialization term, a DP noise term, and a projection error term. Under the choice $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$1, the corollary states

$$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$2

The paper further states that, relative to vanilla DP-SGD, random projection effectively replaces the dimension $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$3 by $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$4 in the privacy-dependent term, but introduces projection error (Jiang et al., 11 Sep 2025).

For smooth non-convex objectives, the target quantity is the gradient norm. The corresponding theorem bounds $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$5 by terms that mirror the convex decomposition—initialization, DP noise, and projection error—together with clipping bias. Under $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$6, the corollary gives

$$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$7

The non-vanishing bias is explicitly tied to the stochastic-gradient variance term and the automatic-clipping stability constant $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$8 (Jiang et al., 11 Sep 2025).

The resulting trade-off is described as a dimension–privacy–utility trilemma. Stronger privacy, i.e. smaller $$\|\nabla f(\mathbf{x}_1)-\nabla f(\mathbf{x}_2)\| \le L\|\mathbf{x}_1-\mathbf{x}_2\|,$$9, requires larger base noise $f$0 and therefore worsens utility. Smaller projection dimension $f$1 reduces the dimension of the noise-bearing subspace and can improve efficiency, but excessive reduction increases projection error. Larger $f$2 improves fidelity to the original gradient but erodes the computational and utility gains of projection (Jiang et al., 11 Sep 2025).

This interpretation differs from the subspace-identification argument of Projected DP-SGD. In “Bypassing the Ambient Dimension,” the projection is data-dependent and aligned with a public gradient eigenspace, so the main utility benefit comes from replacing ambient-dimension dependence by dependence on an effective subspace dimension $f$3, up to polylogarithmic factors in $f$4 under structural assumptions on the gradient spectrum (Zhou et al., 2020). D2P2-SGD does not attempt to estimate such a task-specific subspace; it instead uses random projection and a dynamic noise schedule, so its approximation error is of a different kind.

5. Relation to adjacent methods and common misconceptions

A first misconception is to identify D2P2-SGD with public-subspace Projected DP-SGD. The two methods solve related but distinct problems. Projected DP-SGD computes

$f$5

extracts a top-$f$6 eigenspace $f$7 from public data, defines $f$8, and updates with

$f$9

Its purpose is to exploit low-dimensional gradient structure, and its privacy cost is identical to that of DP-SGD because the projection is post-processing of a Gaussian mechanism (Zhou et al., 2020). D2P2-SGD does not estimate a public eigenspace; its projection is random, not learned.

A second misconception is that projection alone explains the reported gains. The empirical study for D2P2-SGD includes four baselines: non-private SGD, DPSGD, D2P-SGD, and DP2-SGD, where DP2-SGD is the static-noise variant with projection only and is described as equivalent to PrivSGD. The paper reports that DP2-SGD underperforms D2P2-SGD, and interprets this as evidence that dynamic DP is crucial when projection is used (Jiang et al., 11 Sep 2025).

A third misconception is that D2P2-SGD subsumes all dynamic-DP optimizers. Earlier work on dynamic DP-SGD adjusts clipping thresholds and noise powers while keeping to a total privacy budget, using GDP-based accounting and schedule families such as

$$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$0

with $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$1 (Du et al., 2021, Zhu et al., 10 May 2025). D2P2-SGD instead combines a fixed automatic-clipping form with a $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$2 noise-decay rule and low-dimensional random projection (Jiang et al., 11 Sep 2025). The mechanisms are therefore related in objective, but not identical in accounting framework or schedule parameterization.

A fourth misconception is that dynamic clipping and dynamic privacy are interchangeable. The literature on DC-SGD shows that clipping-threshold selection itself is a major source of utility loss and hyperparameter overhead, and proposes private histogram-based estimation of gradient norm distributions to adapt $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$3 either by percentile or by minimizing an expected squared error criterion (Wei et al., 29 Mar 2025). D2P2-SGD addresses the same general privacy–utility tension, but its clipping component is automatic normalization rather than histogram-based threshold adaptation.

6. Empirical profile, limitations, and research directions

The empirical evaluation of D2P2-SGD is conducted on image classification tasks using FashionMNIST, SVHN, CIFAR-10, KMNIST, EMNIST, and MNIST, with CNNs and ResNet20. The reported baselines are non-private SGD, DPSGD, D2P-SGD, and DP2-SGD. Across these experiments, D2P2-SGD is reported to improve test accuracy over DPSGD, D2P-SGD, and DP2-SGD on SVHN and FashionMNIST, often approaching the performance of non-private SGD (Jiang et al., 11 Sep 2025).

Several ablation findings are central. Increasing the noise scale $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$4 degrades both DP2-SGD and D2P2-SGD, but D2P2-SGD maintains better accuracy in the low privacy regime $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$5, whereas it may underperform in very high-noise regimes. Larger batch size $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$6 improves utility but increases privacy loss, consistent with the theorem’s $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$7 dependence. The projection dimension exhibits a non-monotone optimum: the paper reports that an intermediate reduction rate, around a $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$8 reduction, often yields the best accuracy, while privacy loss is essentially independent of $$\mathbb{E}\big[\|\nabla f(\mathbf{x},s)-\nabla f(\mathbf{x})\|^2\big]\le \sigma^2.$$9 and $d$0 primarily affects utility and computational cost (Jiang et al., 11 Sep 2025).

The paper also records that, for fixed base $d$1, dynamic schemes incur larger final privacy loss $d$2 than static schemes. On SVHN, dynamic schemes reach $d$3 versus $d$4 for static schemes with the same base $d$5. This is presented as the expected cost of decaying per-step variance: better utility is obtained by consuming privacy budget less uniformly across iterations (Jiang et al., 11 Sep 2025).

The implementation guidance is correspondingly concrete. The codebase is built on PyTorch with Opacus for DP accounting. Typical experimental choices include learning rate $d$6, $d$7, batch sizes $d$8, $d$9 epochs, default reduction rate $k=1,\dots,K$0, and $k=1,\dots,K$1 in many plots (Jiang et al., 11 Sep 2025).

The limitations are explicit. D2P2-SGD still degrades sharply under very strict privacy, practical selection of projection dimension $k=1,\dots,K$2 remains largely empirical, the current empirical scope is restricted to CNNs and mid-scale models, and the simple $k=1,\dots,K$3 decay law leaves open the possibility of more sophisticated dynamic privacy schedules (Jiang et al., 11 Sep 2025). A plausible implication is that future work may hybridize D2P2-SGD with data-dependent subspaces or public-data-informed geometric preconditioning. This possibility is consistent with adjacent work that uses public curvature estimates and transformed-space DP operations to improve optimization efficiency under a fixed privacy budget (Li et al., 7 Jul 2026), though such extensions are not part of the D2P2-SGD formulation itself.

In that sense, D2P2-SGD occupies a specific position in the DP optimization landscape. It is not the public-eigenspace method of Projected DP-SGD, not the GDP-scheduled dynamic clipping framework of dynamic DP-SGD, and not the histogram-adaptive clipping framework of DC-SGD. Its distinctive contribution is the combination of dynamic Gaussian noise, automatic clipping by per-sample normalization, and random low-dimensional projection, together with privacy and convergence analyses showing that this combination preserves a standard global DP guarantee while retaining sub-linear optimization rates (Jiang et al., 11 Sep 2025).