Revisiting Membership Inference Under Realistic Assumptions (2005.10881v5)

Abstract: We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. First, we consider skewed priors, to cover cases such as when only a small fraction of the candidate pool targeted by the adversary are actually members and develop a PPV-based metric suitable for this setting. This setting is more realistic than the balanced prior setting typically considered by researchers. Second, we consider adversaries that select inference thresholds according to their attack goals and develop a threshold selection procedure that improves inference attacks. Since previous inference attacks fail in imbalanced prior setting, we develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function, and show that an attack that combines this with thresholds on the per-instance loss can achieve high PPV even in settings where other attacks appear to be ineffective. Code for our experiments can be found here: https://github.com/bargavj/EvaluatingDPML.

• The paper introduces a PPV metric for skewed priors that offers a realistic measure of membership inference risk.
• It develops a hypothesis testing framework and an adaptive threshold selection procedure to enhance the precision of inference attacks.
• Experiments on datasets like Purchase-100X and CIFAR-100 show significant privacy leakage even in differentially private models.

Revisiting Membership Inference Under Realistic Assumptions

The paper, "Revisiting Membership Inference Under Realistic Assumptions," addresses the critical issue of membership inference attacks in machine learning systems. Current research in differential privacy (DP) highlights the privacy-utility tradeoffs, yet an empirical understanding of real-world privacy risks, particularly in relation to membership inference, remains rudimentary. This paper takes significant strides in bridging this gap by considering realistic adversarial settings and developing novel tools and metrics to evaluate privacy leakage.

Overview and Key Contributions

The core focus of this research is on membership inference attacks, which exploit a model's overfitting behavior to deduce whether specific data points were part of the training dataset. Traditional studies have mostly assumed balanced prior probabilities, which is often not reflective of real-world scenarios, where attackers may face skewed prior distributions.

1. Theoretical Contributions:
   • Metrics for Skewed Priors: The authors propose a positive predictive value (PPV) based metric that accommodates scenarios with skewed priors, thus offering a more realistic measure of privacy risk. Existing metrics fail to capture the risk when the majority of a candidate pool consists of non-members.
   • Hypothesis Testing Framework: Building upon f-differential privacy, the manuscript reformulates privacy leakage metrics to better reflect the adversary's hypothesis testing capabilities. The derived metrics are claimed to offer a tighter bound on membership advantage and PPV.
2. Empirical Contributions:
   • Threshold Selection Procedure: A systematic methodology for selecting threshold values tailored to the adversary's attack objectives is proposed, enhancing the precision of inference attacks. This is critical in maximizing privacy leakage within constraints set by adversaries regarding false-positive rates.
   • New Inference Attacks - Merlin and Morgan: The Merlin attack, which uses perturbations in the loss function to infer membership, sets a new precedent by being more effective at achieving low false-positive rates. The Morgan attack combines elements of the Merlin and Yeom attacks, demonstrating superior performance by identifying highly vulnerable members with higher confidence.
3. Experiments and Findings:
   • Across multiple data sets, including Purchase-100X and CIFAR-100, the paper illustrates that both non-private and differentially private models are susceptible to these attacks, particularly when the privacy budgets are not appropriately chosen.
   • Notably, non-private models faced significant privacy leakage even when skewed prior distributions were present, underscoring the practicality of the proposed approaches.

Implications and Future Directions

The implications of this research are substantial. From a theoretical perspective, the introduction of metrics that account for skewed priors means that privacy practitioners now have a tool that better reflects real-world adversarial conditions. The empirical strategies of selecting adaptive thresholds and tailoring attack methodologies may inspire future research streams that further refine adversarial models against diverse realistic scenarios.

In practical terms, the work encourages a critical reevaluation of privacy budgets in machine learning, suggesting that models trained under differential privacy might still be vulnerable to refined and well-constructed attacks, especially at large budgets. As a result, it is paramount for system designers to carefully balance utility with privacy protection, potentially exploring new privacy-preserving mechanisms beyond differential privacy.

The paper opens several avenues for future research, including:

• Extending the hypothesis testing framework to other types of privacy attacks, such as attribute or property inference.
• Investigating the applicability of the proposed metrics and attacks on newer, more sophisticated learning models, like LLMs and generative networks.
• Analyzing the combined effectiveness and interplay of multiple privacy-preserving mechanisms in mitigating membership inference attacks.

Overall, this work lays a robust groundwork for both theoreticians and practitioners in developing more resilient privacy-preserving machine learning systems under realistic and practical adversarial settings. Its contributions towards a nuanced understanding of inference risks challenge the community to rethink current privacy guarantees critically.