Analytical Moments Accountant (AMA)

• AMA is a quantitative framework for precise privacy accounting in differentially private machine learning, leveraging moment generating functions and Rényi Differential Privacy.
• It employs moment-based composition and subsampling amplification to compute tight, data-independent upper bounds on cumulative privacy loss.
• Its efficient algorithmic implementation facilitates fast numerical evaluations and reduced privacy loss in practical DP-SGD systems.

The Analytical Moments Accountant (AMA) is a quantitative framework for precisely tracking the privacy loss in differentially private machine learning algorithms, particularly those based on stochastic gradient descent (SGD) with randomized mechanisms such as the Gaussian mechanism and incorporating dataset subsampling. AMA generalizes and extends the original Moments Accountant approach of Abadi et al. (2016) for the Gaussian mechanism to a broad class of mechanisms admitting Rényi Differential Privacy (RDP) guarantees, explicitly accounting for subsampling amplification effects. It enables practitioners to compute tight, data-independent upper bounds on cumulative privacy loss, translating these bounds into precise $(\epsilon,\delta)$-differential privacy guarantees after arbitrary composition (Abadi et al., 2016, Wang et al., 2018).

1. Theoretical Foundations

AMA begins by formalizing the privacy-loss random variable for a randomized mechanism $M$ acting on two neighboring datasets $d$, $d'$: $$L(o;M,d,d') = \ln \frac{\Pr[M(d)=o]}{\Pr[M(d')=o]}, \quad o\sim M(d).$$ The key quantitative tool is the log moment generating function (LMGF) or cumulant generating function (CGF) of the privacy-loss random variable,

$$\alpha_M(\lambda) = \sup_{d\sim d'} \ln \mathbb{E}_{o\sim M(d)}\left[\exp(\lambda L(o;M,d,d'))\right],$$

where the supremum is taken over all auxiliary randomness and all adjacent dataset pairs. Within the RDP framework, the order-$\alpha$ Rényi divergence between output distributions is tightly linked to the CGF: $$D_\alpha(M(d)\|\;M(d')) = \frac{1}{\alpha-1}\,\alpha_M(\alpha-1).$$ These moment (or cumulant) representations allow precise analysis under composition and subsampling, producing sharp upper bounds on privacy loss.

2. Moment-Based Composition and Subsampling

The composability of CGFs enables AMA to handle the adaptive composition of $k$ mechanisms: $$\alpha_M(\lambda) \leq \sum_{i=1}^k \alpha_{M_i}(\lambda).$$ This moment-based analysis is valid even for adaptive composition (where each step may depend on previous outputs). Subsampling—a probabilistic selection of a data subset before mechanism application—provides privacy amplification. For a mechanism $M$ with RDP guarantee $(\alpha, \varepsilon_M(\alpha))$, when subsampling a fraction $\gamma = m/n$ without replacement, the RDP of the composed mechanism is: $$\varepsilon_{\mathrm{sub}}(\alpha) \leq \frac{1}{\alpha-1}\log\Bigg[1+\sum_{j=2}^\alpha \binom{\alpha}{j}\gamma^j e^{(j-1)\varepsilon_M(j)}\Bigg],$$ for Poisson subsampling, with further generalizations for arbitrary mechanisms (Wang et al., 2018). This result provides the main amplification theorem, translating per-step RDP guarantees to overall amplified privacy protection under repeated mechanisms with subsampling.

3. Conversion to $(\epsilon,\delta)$-Differential Privacy

The translation from moment/CDF-based guarantees to $(\epsilon,\delta)$-differential privacy is achieved through Markov’s inequality and standard dual forms:

• Given target $\delta$, the smallest achievable $\epsilon$ satisfies

$$\epsilon(\delta) = \min_{\lambda>0}\; \frac{\alpha_M(\lambda)+\log(1/\delta)}{\lambda}.$$

• Given target $\epsilon$, the minimal achievable $\delta$ is

$$\delta(\epsilon) = \min_{\lambda>0}\; \exp(\alpha_M(\lambda) - \lambda \epsilon).$$

The single-variable convex optimization over $\lambda$ (or equivalently over the RDP order $\alpha = \lambda+1$) can be efficiently solved using bisection; convexity and monotonicity properties guarantee correctness and fast convergence. In practical settings, $\alpha$ is discretized on a computational grid with optional piecewise-linear interpolation to cover fractional orders (Abadi et al., 2016, Wang et al., 2018).

4. Algorithmic Implementation and Data Structures

AMA is implemented as a data structure maintaining a dictionary of mechanisms, each paired with its count of applications and CGF. Each time a mechanism (with or without subsampling) is applied, the CGF is computed (or cached) and incremented. The pseudocode is:

DataStructure AMA: initialize empty dictionary D function add_mechanism(M, count=1): if D contains M: D[M].count += count else: D[M] := (count, K_M) function add_subsampled_mechanism(M, gamma, count=1): compute ε_sub(α) for M∘subsample via amplification theorem define K_sub(λ) = λ·ε_sub(λ+1) add_mechanism(M_sub, count) # anonymous_mech_with_CGF=K_sub function total_CGF(λ): return sum(count * K(λ) for (count, K) in D.values()) function get_epsilon(delta): bisection_solve(lambda λ: (total_CGF(λ)+log(1/delta))/λ) function get_delta(epsilon): bisection_solve(lambda λ: exp(total_CGF(λ)-λ*epsilon))

Each mechanism increment is constant time; each CGF evaluation is linear in the number of distinct mechanisms.

5. Closed-Form and Numerical Evaluation

For certain mechanisms, such as Poisson-subsampled Gaussian, AMA enables closed-form or numerically tractable computation of log-moments: $\alpha(\lambda) = \ln\Bigl(\max\{E_1, E_2\}\Bigr)$ with the $E_j$ involving expectations over Gaussian densities mixed according to the inclusion probability. For small sampling rates and large noise, an asymptotic bound applies: $$\alpha(\lambda) \leq \frac{q^2\lambda(\lambda+1)}{2\sigma^2} + O(q^3\lambda^3/\sigma^3).$$ Numerical integration may be required for intermediate parameter regimes. In practical systems, precomputed tables and efficient log-sum-exp evaluations are used to ensure stability and tractability (Abadi et al., 2016).

6. Practical Performance, Tradeoffs, and Recommendations

Computational cost for maintaining the AMA is modest: per-step overhead in standard DP-SGD implementations (e.g., TensorFlow) is typically 10–30% compared to standard SGD. Best practice calls for:

• Discretizing lambda ($\lambda$) on a grid (e.g., 1 to 32 in (Abadi et al., 2016), $\alpha$ up to several hundred or thousand in (Wang et al., 2018)), with interpolation as required.
• Numerical stability via log-sum-exp and careful floating-point operations, especially for large CGFs.
• Storing a dictionary of distinct mechanisms and their counts for efficient moment accounting.
• Fixing a bisection tolerance for invertible privacy cost queries (e.g., $\tau=10^{-12}$).

Empirical studies illustrate that AMA yields strictly tighter privacy accounting relative to classical composition (naive addition or strong $(\epsilon,\delta)$ composition [Kairouz–Oh–Viswanath]), especially at high composition counts and in the high-privacy regime (small $\epsilon$). This results in order-of-magnitude reductions in overall privacy loss for commonly used mechanisms, as demonstrated for subsampled Gaussian, Laplace, and randomized response mechanisms (Wang et al., 2018).

7. Significance and Impact

AMA provides a rigorous, general, and data-independent tool for privacy accounting in large-scale private machine learning. It is compatible with arbitrary RDP mechanisms and subsampling schemes, enabling precise per-instance privacy cost conversion and optimal parameter selection. AMA represents a major advance over advanced composition approaches, achieving tighter privacy loss and directly supporting the design and auditing of deep learning systems for privacy preservation (Abadi et al., 2016, Wang et al., 2018).