Tight Auditing of Differential Privacy in MST and AIM

Abstract: State-of-the-art Differentially Private (DP) synthetic data generators such as MST and AIM are widely used, yet tightly auditing their privacy guarantees remains challenging. We introduce a Gaussian Differential Privacy (GDP)-based auditing framework that measures privacy via the full false-positive/false-negative tradeoff. Applied to MST and AIM under worst-case settings, our method provides the first tight audits in the strong-privacy regime. For $(ε,δ)=(1,10^{-2})$, we obtain $μ_{emp}\approx0.43$ vs. implied $μ=0.45$, showing a small theory-practice gap. Our code is publicly available: https://github.com/sassoftware/dpmm.

• The paper demonstrates that a GDP-based auditing method tightly aligns empirical privacy leakage (μ_emp ≈ 0.43) with theoretical guarantees.
• It employs a robust hypothesis-testing framework using optimal classifier thresholding and worst-case dataset selection to maximize adversarial advantage.
• The study validates MST and AIM implementations in strong-privacy settings, ensuring practical DP guarantees that closely mirror zCDP/GDP profiles.

Tight Auditing of Differential Privacy in MST and AIM

Introduction and Motivation

Differentially Private (DP) synthetic data generation is central to privacy-preserving data sharing, especially for sensitive tabular datasets. Two frameworks—MST (Maximum Spanning Tree) and AIM (Adaptive and Iterative Mechanism)—are extensively adopted for this purpose due to their competitive privacy-utility tradeoffs and integration in important applications (e.g., NIST competition, UK Census). Despite the formal $(\epsilon, \delta)$-DP guarantees of these models, rigorously auditing their real-world privacy leakage, particularly in the strong-privacy regime ($\epsilon = 1$), represents a persistent challenge. Existing empirical audits have been loose or inconclusive, exposing a risk of undetected privacy violations or over-conservative design.

This paper introduces a Gaussian Differential Privacy (GDP)-based empirical auditing methodology for MST and AIM, enabling tight, systematic evaluation of membership inference risk and direct comparison with theoretical guarantees under multiple DP notions.

GDP-Based Auditing Methodology

The cornerstone of the approach is reframing DP auditing through the lens of the hypothesis testing interpretation—specifically, using the full tradeoff between False Positive Rate (FPR) and False Negative Rate (FNR) in Membership Inference Attacks (MIA). This is formalized by mapping each DP guarantee (conventional $(\epsilon, \delta)$-DP, zero-Concentrated DP (zCDP), and GDP) onto a $\mu$-parameterized Gaussian tradeoff curve that sets a lower bound on the achievable adversarial advantage.

The audit apparatus consists of:

• Worst-case neighboring datasets: $D_{out}$ with identical background records and $D_{in}$ differing by a single target record, maximizing susceptibility to MIA.
• Hybrid threat model: Exploiting both black-box (synthetic data-based) and white-box (model internals) features for MIA feature construction.
• Large-scale randomized mechanism runs: Training thousands of independent MST/AIM models on both $D_{out}$/$D_{in}$ and releasing synthetic data and/or noisy marginals.
• Empirical estimation of $\mu_{emp}$: Adversarial classifiers (XGBoost) are calibrated for maximal empirical advantage, then TPR/FPR tradeoffs are mapped onto the GDP curve to obtain a tight Bayesian lower bound for $\mu$.

A key procedural step is the optimal selection of the classifier decision threshold on validation data to maximize empirical adversarial advantage, ensuring the tightest possible audit bound.

Experimental Evaluation

The central claim is validated through systematic experiments targeting MST/AIM in the independent-marginal regime (identical one-way marginals), under worst-case privacy threat models and parameters $\epsilon = 1$0. Empirical and theoretical tradeoff curves are compared in the $\epsilon = 1$1-GDP framework.

Figure 1: Empirical privacy tradeoff of MIA for MST/AIM compared to theoretical GDP and $\epsilon = 1$2-DP bounds across the FPR–FNR spectrum. At $\epsilon = 1$3, empirical leakage nearly saturates the theoretical GDP bound, with $\epsilon = 1$4 and implied $\epsilon = 1$5.

A salient result is the narrow theory-practice gap in the strong-privacy regime: the empirical bound $\epsilon = 1$6 is almost equal to the implied $\epsilon = 1$7 from zCDP accounting, establishing the first nontrivial audit in this regime for MST/AIM. In contrast, direct $\epsilon = 1$8 conversion without the zCDP intermediate step yields weaker privacy (larger $\epsilon = 1$9), underscoring the implications of accounting paths in DP mechanisms.

Optimal threshold selection (Figure 2) is shown to maximize the adversarial advantage, with smooth, stable empirical advantage curves as a function of the decision threshold.

Figure 2: Threshold selection on validation data, reporting FPR, FNR, and empirical advantage (TPR–FPR) as the threshold varies. The optimum $(\epsilon, \delta)$0 yields maximal advantage.

The auditing framework's robustness is demonstrated through an extensive ablation study, varying audit component choices (threshold selection, estimator, $(\epsilon, \delta)$1 size, classifier, threat model). The default configuration is consistently the tightest; naively chosen thresholds or estimators can produce vacuous ($(\epsilon, \delta)$2) bounds, explaining prior failures to audit the strong-privacy regime.

Use of higher-order marginals (see Appendix) does not degrade, and in some cases slightly tightens, the empirical audit, confirming the generality of the findings.

Practical and Theoretical Implications

This work provides evidence that MST and AIM, under standard implementations and proper accounting, realize privacy guarantees in practice that closely match their formal zCDP/GDP profiles. The small gap between empirical and theoretical guarantees validates the current accounting and highlights the utility-conserving effect of sound zCDP-to-GDP conversion.

From a theoretical perspective, the audit paradigm adopted here bridges the abstract hypothesis-testing DP view with operational privacy leakage in deployed models. Consistent tightness of the audit bounds across variants and threat models elevates empirical auditing from a diagnostic tool to a reliable certificate of privacy in the strong-regime, at least for mechanisms based on Gaussian noise and marginal measurements.

Future research directions include extending GDP-based auditing to more general generative models (e.g., deep generative models under custom noise injection schemes), and formalizing the statistical properties of the Bayesian lower bound estimator for audit reproducibility, especially for mechanisms that diverge from the Gaussian paradigm. Moreover, practitioners should standardize such tight auditing as part of the responsible deployment pipeline for any DP synthetic data toolchain.

Conclusion

This paper demonstrates that GDP-based empirical privacy auditing provides a practical, reproducible, and tight modality for certifying the privacy of MST and AIM synthetic data generators in the strong-privacy regime. The framework exposes the empirical-theoretical closeness of leakage risks, reinforcing confidence in these mechanisms when correctly implemented and accounted. Broader community adoption of this auditing standard, facilitated by the open-source code release, will further enhance the operational verifiability of DP guarantees and drive higher assurance in differential privacy deployments.