Subsampled Rényi Differential Privacy and Analytical Moments Accountant (1808.00087v2)

Abstract: We study the problem of subsampling in differential privacy (DP), a question that is the centerpiece behind many successful differentially private machine learning algorithms. Specifically, we provide a tight upper bound on the R\'enyi Differential Privacy (RDP) (Mironov, 2017) parameters for algorithms that: (1) subsample the dataset, and then (2) applies a randomized mechanism M to the subsample, in terms of the RDP parameters of M and the subsampling probability parameter. Our results generalize the moments accounting technique, developed by Abadi et al. (2016) for the Gaussian mechanism, to any subsampled RDP mechanism.

• The paper develops a generalized method for amplifying Rényi differential privacy via subsampling, deriving tight bounds on privacy loss.
• It introduces an innovative analytical moments accountant that efficiently tracks privacy parameters in complex compositions.
• Extensive experiments validate improvements over traditional differential privacy methods, influencing privacy-preserving machine learning design.

Subsampling and Rényi Differential Privacy: A Comprehensive Review

The paper under review explores a nuanced problem within the domain of differential privacy (DP): specifically, subsampling and its effects on Rényi Differential Privacy (RDP) parameters. This avenue of research is critical as differential privacy has become the standard for privacy-preserving computations in both academia and industry. Subsampling is a well-utilized mechanism in differentially private machine learning algorithms and understanding its influence on RDP can lead to more efficient and accurate privacy guarantees.

Overview of Research

The authors provide a theoretical framework to derive upper bounds on the RDP parameters for algorithms that subsample datasets before applying a randomized mechanism. The paper generalizes existing methods, such as the moments accountant technique used for the Gaussian mechanism, to accommodate any RDP mechanism.

The innovation lies primarily in two areas:

1. Generalized RDP Amplification: The authors develop a method to derive amplifications in RDP through subsampling, which goes beyond the existing approaches that primarily focus on the Gaussian mechanism.
2. Analytical Tools and Bounds: A comprehensive set of bounds is provided involving ternary-$|\chi|^\alpha$-divergences and Pearson-Vajda divergences, which pave the way for broader applications of subsampling in differentially private algorithms.

Significant Results and Methodologies

The paper presents several key results:

• A tight bound on the RDP parameter of a subsampled mechanism as a function of the original mechanism's RDP parameters and the subsampling ratio. This generalizes the subsampling lemmas known for classical $(\epsilon, \delta)$-DP.
• The introduction of a new theoretical framework involving ternary-$|\chi|^\alpha$-divergences that is more naturally suited to handle subsampling.
• An innovative analytical moments accountant is proposed, which efficiently tracks privacy parameters under complex compositions — enhancing previous methodologies that were restricted to discrete orders.

Numerical Results and Implications

The authors validate their theoretical findings through extensive numerical experiments on several popular mechanisms: Gaussian, Laplace, and randomized response. The subsampled Gaussian mechanism, in particular, demonstrates marked improvements in privacy parameters compared to traditional differential privacy methods.

These results suggest that the proposed techniques could significantly impact the design of differentially private machine learning models, particularly those employing iterative algorithms like noisy stochastic gradient descent.

Future Directions

The exploration of subsampling in RDP opens up several future research directions:

1. Data-dependent RDP and per-instance privacy loss could provide more refined privacy estimates by taking into account dataset-specific characteristics.
2. Leveraging this work in sublinear algorithms that employ subsampling could yield further privacy advantages in big data contexts.
3. Potential connections with non-DP-driven methodologies like bootstrap and jackknife can be explored, potentially enabling hybrid models with robust privacy guarantees.

In conclusion, this research enriches our understanding of differential privacy techniques, particularly through the lens of subsampling and RDP, with direct implications for both theoretical explorations and practical applications in privacy-preserving computations. The analytical advancements and comprehensive results provide a compelling pathway for future developments in designing more efficient and effective differentially private algorithms.