Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference (1906.11798v2)

Abstract: Membership inference (MI) attacks exploit the fact that machine learning algorithms sometimes leak information about their training data through the learned model. In this work, we study membership inference in the white-box setting in order to exploit the internals of a model, which have not been effectively utilized by previous work. Leveraging new insights about how overfitting occurs in deep neural networks, we show how a model's idiosyncratic use of features can provide evidence for membership to white-box attackers---even when the model's black-box behavior appears to generalize well---and demonstrate that this attack outperforms prior black-box methods. Taking the position that an effective attack should have the ability to provide confident positive inferences, we find that previous attacks do not often provide a meaningful basis for confidently inferring membership, whereas our attack can be effectively calibrated for high precision. Finally, we examine popular defenses against MI attacks, finding that (1) smaller generalization error is not sufficient to prevent attacks on real models, and (2) while small-$\epsilon$-differential privacy reduces the attack's effectiveness, this often comes at a significant cost to the model's accuracy; and for larger $\epsilon$ that are sometimes used in practice (e.g., $\epsilon=16$), the attack can achieve nearly the same accuracy as on the unprotected model.

• The paper demonstrates a novel white-box attack that leverages model internals to accurately infer training data membership.
• It employs local linear approximations and Bayesian derivations to calibrate inference models across diverse data distributions.
• Empirical evaluations expose vulnerabilities even in models using differential privacy, highlighting the need for stronger defense mechanisms.

Analysis of White-Box Membership Inference Using Model Internals

The paper "Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference" provides substantial contributions to the domain of privacy threats posed by machine learning models, focusing specifically on membership inference (MI) attacks. Traditionally, these attacks attempt to determine whether a particular data point was included in the training set of a machine learning model. While previous research predominantly concentrated on black-box settings where attackers only access model outputs, this work innovatively explores the white-box scenario, thus leveraging model internals for inference.

Summary of Contributions

The paper's primary contribution is its demonstration of how to effectively exploit the internals of deep neural networks to enhance the precision and efficacy of membership inference attacks. This approach is grounded on the observation that models often memorize unusual features present in their training data that may not generalize to the overall data distribution, thus betraying membership information.

Key Insights and Methodologies

1. Conceptual Framework for White-Box Attacks:
   • The paper proposes a conceptual framework and a detailed methodological approach that utilizes the model's idiosyncratic feature usage over simple generalization error analysis, positing that internal representations within models hold significant potential for identifying membership. The frameworks include leveraging:
     • Local Linear Approximations: By viewing model components around specific data points as approximations of simpler models, the authors derive attack models.
     • Scalability to Deep Networks: Techniques are adapted to not only utilize final outputs but also intermediate layer activations and gradients, providing a comprehensive inference approach.
2. Bayes-Optimal Attack Derivation:
   • A theoretically driven approach is presented to derive Bayes-optimal attack models that can achieve superior inference accuracy. This relies on understanding the statistical distribution of data in both training and population distributions, quantified through differential use of empirical and general population variance.
3. Empirical Efficacy and Calibration:
   • The empirical evaluation spans a variety of datasets and model configurations that demonstrate their attack model's consistent superiority over prior black-box methods. Importantly, they underscore the paper's ability to meet high precision requirements with calibrated confidence, an essential attribute for practical inference attacks.
4. Evaluation of Privacy Defenses:
   • The paper critically evaluates common defenses like differential privacy (DP) and regularization (dropout), providing insights into their limitations and scenarios where they fail. Specifically, it notes the ineffectiveness of large-ε DP in practice owing to its detrimental impact on utility, while demonstrating that even models protected with minor generalization error are vulnerable to structured MI attacks.

Practical Implications and Future Directions

The implications of these findings are profound as they provoke reconsiderations in how we assess model security and privacy. This research impacts both theoretical and practical dimensions of AI implementations:

• For Model Developers: It advises a need for more robust mechanisms which can secure models beyond surface-level defenses (e.g., DP or dropout) especially in applications dealing with private information.
• Future Research Directions: Encourages exploration into more nuanced defense strategies that can withstand not just query-based but also white-box style threats, and the development of transferable defense mechanisms that can generalize across different model architectures.
• Theoretical Exploration: Future avenues include enhancing theoretical bounds of inference success in relation to varied model architectures and exploring how overfitting dynamics differ across models, potentially leading to newer attack vectors or enhanced defenses.

In conclusion, this work stands as a critical examination of the potential vulnerabilities present in machine learning systems through a shift in focus to white-box indicative behaviors. By rigorously testing the space of feature memorization, it systematically advances the discourse on the safe deployment of these models in sensitive applications.