Less Random, More Private: What is the Optimal Subsampling Scheme for DP-SGD?

Abstract: Poisson subsampling is the default sampling scheme in differentially private machine learning, largely because its unstructured randomness yields tractable privacy amplification analyses. Yet this same randomness introduces substantial participation variance: each sample appears in very different numbers of training iterations. In this work, we show that this variance is not merely a practical artifact to be tolerated, but a fundamental source of suboptimal privacy amplification. We prove that Balanced Iteration Subsampling (BIS), a structured scheme in which each sample participates in exactly a fixed number of iterations, achieves stronger privacy amplification than Poisson subsampling and is optimal at both extremes of the noise spectrum ($σ\to 0$ and $σ\to \infty$). Our analysis reveals that the privacy-noise tradeoff is governed not by maximizing randomness, but by eliminating participation variance while preserving uniform marginal participation across iterations. To translate this asymptotic theory into finite-noise guarantees, we introduce a practical near-exact Monte Carlo accountant for BIS, which removes the analytical slack of existing RDP and composition-based PLD analyses. Evaluations across more than 60 practical DP-SGD configurations show that BIS consistently outperforms Poisson subsampling in the low-noise regimes most relevant for high-utility private training, reducing the required noise multiplier by up to $9.6\%$. These results overturn the common intuition that more sampling randomness necessarily yields stronger privacy amplification: in DP-SGD, structured participation can be both more practical and more private. Our implementation is available at https://github.com/dong-xin-ao-andy/bis-mc-accountant.

• The paper introduces Balanced Iteration Subsampling (BIS) to eliminate participation variance in DP-SGD, outperforming Poisson subsampling in privacy amplification.
• It employs a novel Monte Carlo privacy accountant with dynamic programming and ultra-fast screening to achieve near-exact (ε,δ)-DP bounds.
• Empirical results show that BIS reduces Gaussian noise requirements by up to 9.6% in low-noise regimes, ensuring tighter privacy-utility tradeoffs.

Optimal Subsampling in DP-SGD: Theoretical and Practical Superiority of Balanced Iteration Subsampling

Overview

This paper rigorously investigates the impact of the data subsampling scheme on privacy amplification in differentially private stochastic gradient descent (DP-SGD). It challenges the established reliance on Poisson subsampling—where each example participates in each iteration independently—with the alternative of Balanced Iteration Subsampling (BIS), a structured mechanism in which each sample participates in a fixed number of training iterations, chosen uniformly at random. The results demonstrate that minimizing participation variance, rather than maximizing sampling randomness, is the critical factor for optimal privacy-utility tradeoffs. BIS is shown to be optimal, both asymptotically and empirically, in two key noise regimes and across practical DP training configurations.

Theoretical Analysis of Subsampling Schemes

Independent-Example Mechanisms

The analysis is performed in the context of independent-example mechanisms, where each data sample independently draws its participation pattern over $T$ iterations with a fixed expected count $k$. Poisson and BIS both fit into this family, with the crucial distinction that BIS fixes the number of participations per sample, while Poisson introduces notable variance.

Low-Noise Regime ($\sigma \to 0$)

In the low-noise (high-utility) setting, the privacy loss is primarily governed by the variance in the number of times a sample is selected (participation-count variance). The theory establishes that BIS achieves the minimal possible amplification loss by eliminating participation variance completely and maximizing entropy subject to this constraint. Explicitly, while Poisson subsampling maximizes entropy on the hypercube, this comes at the expense of excessive participation outliers, deteriorating the worst-case privacy loss tail.

Key theoretical claim: In the low-noise limit, privacy amplification is maximized by removing participation-count variance among all subsampling schemes with a fixed expectation.

High-Noise Regime ($\sigma \to \infty$)

At the other asymptotic extreme, the dominant factor becomes the uniformity of per-iteration marginal participation probabilities. Both Poisson and BIS achieve the optimal uniformity (all marginals equal to $k/T$); however, BIS further suppresses subdominant higher-order fluctuations due to its symmetric structure.

Key theoretical claim: In the high-noise limit, any scheme with uniform marginals, including BIS and Poisson sampling, achieves optimal first-order privacy amplification; only the second-order tail (vanishingly small in this regime) remains for BIS to further reduce.

Practical Monte Carlo Privacy Accounting for BIS

Efficient Near-Exact Privacy Estimation

Closed-form $(\varepsilon, \delta)$-DP accounting for BIS is infeasible analytically due to the combinatorial structure of $k$-out-of-$T$ sampling. The paper introduces a novel, practical, near-exact Monte Carlo (MC) privacy accountant. The method is based on the following innovations:

• Dynamic Programming Exact Likelihood: By rewriting the computation of the likelihood ratio for output samples as a dynamic program, the accountant reduces per-sample complexity from $O(\binom{T}{k})$ to $O(Tk)$.
• Ultra-Fast Screening Bound: A tight $k$0 upper bound on the log-likelihood is used as a computational filter, discarding the majority of samples before invoking the DP routine.
• EVR Framework: Accounting is embedded within the Estimate-Verify-Release framework, ensuring certified privacy bounds that properly absorb the statistical error of MC estimation into the $k$1 failure probability.

This methodology removes all the loose analytical overapproximations endemic to Rényi DP and composition-based PLD analyses used by prior BIS studies.

Empirical Evaluation and Numerical Results

Comparison Across Realistic DP-SGD Regimes

Comprehensive empirical studies spanning 60+ DP training configurations validate the theoretical predictions. The key findings are:

• Privacy Gain in Low-Noise Regimes: In parameter regimes corresponding to high-utility, low-noise DP-SGD, BIS consistently reduces the required Gaussian noise multiplier $k$2 compared to Poisson subsampling. Verified savings of up to 9.6% in noise are observed, directly translating into improved learning utility or tighter privacy guarantees.
• Convergence at High Noise: In high-noise regimes, the BIS and Poisson requirements converge, aligning with theoretical expectations.
• Necessity of Tight Accounting: The superiority of BIS is obscured by the analytical slack of composition and RDP bounds. The screen-then-exact MC accountant is crucial to revealing its true advantage.

Strong empirical claim: BIS consistently outperforms Poisson in low-noise settings relevant to practical deployment, and it never underperforms compared with Poisson in certified privacy.

Implications and Future Directions

The theoretical results overturn the widely-held intuition that maximizing sampling randomness necessarily improves privacy amplification in subsampled mechanisms. Instead, the work demonstrates that controlling participation variance is paramount. Practically, the adoption of BIS leads to tighter privacy-utility tradeoffs, especially in high-utility DP-SGD. Furthermore, reducing participation variance has secondary benefits for fairness and system design, especially in decentralized or federated learning, by ensuring more even data contribution.

For future research, several directions follow:

• Beyond Independent-Example Mechanisms: Investigating optimality among more general, possibly dependent, batch construction schemes.
• Analytical Accounting: Derivation of closed-form $k$3 bounds for BIS would allow for even more efficient deployment.
• Extensions to Heterogeneous Participation and Adaptive Schemes: Extending tight accounting and optimal design to settings with non-uniform participation or adaptive batch selection.

Conclusion

Balanced Iteration Subsampling achieves provably optimal privacy amplification among independent-example mechanisms for DP-SGD, outperforming established Poisson sampling in the low-noise regime critical for real-world private ML. This optimality is realized both theoretically and practically through a combination of variance elimination and novel, tight MC privacy accounting. The results fundamentally refine practitioners' understanding of privacy amplification, prompting a shift toward structured subsampling for differentially private training.