The Sample Complexity of Membership Inference and Privacy Auditing

Abstract: A membership-inference attack gets the output of a learning algorithm, and a target individual, and tries to determine whether this individual is a member of the training data or an independent sample from the same distribution. A successful membership-inference attack typically requires the attacker to have some knowledge about the distribution that the training data was sampled from, and this knowledge is often captured through a set of independent reference samples from that distribution. In this work we study how much information the attacker needs for membership inference by investigating the sample complexity-the minimum number of reference samples required-for a successful attack. We study this question in the fundamental setting of Gaussian mean estimation where the learning algorithm is given $n$ samples from a Gaussian distribution $\mathcal{N}(\mu,\Sigma)$ in $d$ dimensions, and tries to estimate $\hat\mu$ up to some error $$\mathbb{E}[|\hat \mu - \mu|^{2_{\Sigma}]\leq} \rho² d$$. Our result shows that for membership inference in this setting, $\Omega(n + n² \rho<sup>2)$ samples can be necessary to carry out any attack that competes with a fully informed attacker. Our result is the first to show that the attacker sometimes needs many more samples than the training algorithm uses to train the model. This result has significant implications for practice, as all attacks used in practice have a restricted form that uses $O(n)$ samples and cannot benefit from $\omega(n)$ samples. Thus, these attacks may be underestimating the possibility of membership inference, and better attacks may be possible when information about the distribution is easy to obtain.

• The paper's main contribution is establishing a lower bound on the sample complexity required for membership-inference attacks when the covariance is unknown.
• It demonstrates that effective attacks need Ω(n²ρ²) auxiliary samples in high dimensions, emphasizing that learning the covariance is the main challenge.
• A matching upper bound via a constructive attack highlights implications for privacy auditing and questions previous empirical methodologies.

The Sample Complexity of Membership Inference and Privacy Auditing

Overview and Motivation

This paper rigorously investigates the sample complexity required for effective membership-inference attacks (MIA) in statistical learning, focusing on the canonical problem of Gaussian mean estimation. The central question is: how many auxiliary samples from the underlying population does an attacker need to reliably distinguish whether a target individual was included in the training set, given only the model output and limited knowledge of the data distribution? The analysis is motivated by both theoretical and practical concerns in privacy auditing, as MIAs are widely used to empirically assess privacy risks in machine learning models.

Problem Formulation

The authors formalize the MIA scenario as follows: a learning algorithm receives $n$ samples from a $d$-dimensional Gaussian $\mathcal{N}(\mu, \Sigma)$ and outputs an estimate $\hat{\mu}$ of the mean, with accuracy measured in Mahalanobis norm, i.e., $$\mathbb{E}[\|\hat{\mu} - \mu\|^2_\Sigma] \leq \rho^2 d$$. The attacker is given $\hat{\mu}$, a target point $X$ (either a member of the training set or an independent sample), and $m$ auxiliary samples from the same distribution. The goal is to distinguish the "IN" (member) from "OUT" (non-member) cases with true positive and false positive rates exceeding random guessing.

The paper introduces a minimal definition of sample-based MIA, requiring only a slight advantage over random guessing, and analyzes the sample complexity as a function of $n$, $d$, and $d$0.

Main Results

Lower Bound: Unknown Covariance

The principal technical contribution is a lower bound on the sample complexity for MIAs when both the mean and covariance of the population are unknown to the attacker. The authors prove that, in the regime where $d$1 (the threshold for successful attacks with full distributional knowledge), any sample-based MIA requires $d$2 auxiliary samples. In particular, when $d$3, the required number of samples is $d$4, which can be much larger than the training set size.

This result is established via a reduction to hypothesis testing over a family of spiked covariance matrices, leveraging information-theoretic arguments and total variation bounds. The analysis shows that the difficulty of MIA is driven by the challenge of estimating the covariance structure, especially in high-dimensional settings with many "spiked" directions.

Upper Bound: Constructive Attack

The paper also provides a matching upper bound: with $d$5 samples, the attacker can estimate the covariance and mean sufficiently well to implement an effective MIA, using a test statistic based on the empirical covariance and mean of the auxiliary samples. This attack generalizes the Neyman-Pearson test for the fully informed case.

Known Covariance Case

When the covariance $d$6 is known to the attacker, the sample complexity drops dramatically. The authors show that only $d$7 or $d$8 samples are required, matching previous results. This highlights that the main bottleneck in MIA sample complexity is learning the covariance, not the mean.

Technical Approach

The lower bound is proved by constructing a family of Gaussian distributions with spiked covariance matrices and showing that, unless the attacker has enough samples to identify all high-variance directions, the IN and OUT cases are statistically indistinguishable. The proof involves:

• Reducing the problem to hypothesis testing over mixtures of Gaussians with unknown subspaces.
• Characterizing sufficient statistics for the attacker and bounding the total variation distance between IN and OUT distributions.
• Using chain rules for total variation and Pinsker's inequality to relate KL divergence to distinguishability.

The upper bound is achieved by constructing an explicit attack that estimates the covariance and mean from the auxiliary samples and applies a generalized Neyman-Pearson test.

Implications for Practice

The results have significant implications for privacy auditing and empirical MIA methodology:

• Existing empirical attacks (e.g., shadow models) typically use $d$9 auxiliary samples and only calibrate thresholds, not test statistics. The paper shows that such attacks may be fundamentally limited and can underestimate privacy risk when more distributional information is available.
• Optimal attacks may require many more samples than the training set size, especially in high-dimensional, low-signal regimes. This challenges the assumption that attackers need only as much data as the training algorithm.
• Auditing methodology should be revisited to consider attacks that exploit richer distributional knowledge, especially in settings where auxiliary data is abundant.

Theoretical and Future Directions

The findings clarify the relationship between sample complexity, dimensionality, and privacy risk in statistical learning. They suggest several avenues for future research:

• Extending the analysis to more complex models (e.g., deep neural networks) and other learning tasks.
• Designing practical attacks that exploit auxiliary data beyond threshold calibration, possibly using quadratic or higher-order approximations to the loss landscape.
• Investigating the trade-offs between sample complexity and computational complexity in privacy auditing.

Conclusion

This paper establishes tight information-theoretic bounds on the sample complexity of membership-inference attacks in Gaussian mean estimation, showing that optimal attacks may require substantially more auxiliary data than the training set size when the covariance is unknown. The results challenge prevailing empirical methodologies and underscore the importance of attacker knowledge in privacy risk assessment. The work provides a rigorous foundation for future research on privacy auditing and adversarial inference in statistical learning.