Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy (2403.01218v3)

Abstract: The high cost of model training makes it increasingly desirable to develop techniques for unlearning. These techniques seek to remove the influence of a training example without having to retrain the model from scratch. Intuitively, once a model has unlearned, an adversary that interacts with the model should no longer be able to tell whether the unlearned example was included in the model's training set or not. In the privacy literature, this is known as membership inference. In this work, we discuss adaptations of Membership Inference Attacks (MIAs) to the setting of unlearning (leading to their "U-MIA" counterparts). We propose a categorization of existing U-MIAs into "population U-MIAs", where the same attacker is instantiated for all examples, and "per-example U-MIAs", where a dedicated attacker is instantiated for each example. We show that the latter category, wherein the attacker tailors its membership prediction to each example under attack, is significantly stronger. Indeed, our results show that the commonly used U-MIAs in the unlearning literature overestimate the privacy protection afforded by existing unlearning techniques on both vision and LLMs. Our investigation reveals a large variance in the vulnerability of different examples to per-example U-MIAs. In fact, several unlearning algorithms lead to a reduced vulnerability for some, but not all, examples that we wish to unlearn, at the expense of increasing it for other examples. Notably, we find that the privacy protection for the remaining training examples may worsen as a consequence of unlearning. We also discuss the fundamental difficulty of equally protecting all examples using existing unlearning schemes, due to the different rates at which examples are unlearned. We demonstrate that naive attempts at tailoring unlearning stopping criteria to different examples fail to alleviate these issues.

• The paper introduces inexact unlearning as a method to efficiently remove data influence without full retraining.
• It demonstrates that per-example U-MIAs significantly outperform population-based attacks in exposing privacy vulnerabilities.
• The results highlight that unlearning may inadvertently heighten privacy risks for remaining data, urging the need for robust adversarial models.

An Analysis of "Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy"

In the domain of machine learning, privacy preservation is an increasingly pivotal concern as models grow in size and complexity, necessitating innovative approaches to data management. The paper "Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy" by Hayes et al. critically examines current methodologies in machine unlearning, highlighting the inadequacies of existing privacy measures and advocates for more rigorous evaluation techniques.

The authors introduce the concept of "inexact unlearning," which aims to remove the influence of specific data samples from a trained model without the need for complete retraining, thereby offering computational efficiency. A key challenge here is ensuring that once unlearning is performed, external entities cannot discern whether a particular data sample was part of the training set. This notion is often evaluated through Membership Inference Attacks (MIAs), particularly adapted to the unlearning framework, known as U-MIAs.

A significant contribution of this paper is the differentiation between "population U-MIAs" and "per-example U-MIAs." Population U-MIAs apply a generalized attack strategy across all data points, potentially underestimating privacy risks due to their non-specific nature. Conversely, per-example U-MIAs tailor the attack to individual data samples, proving to be considerably more potent in the authors' experiments, and revealing the overestimated privacy protections claimed by many unlearning algorithms.

Empirical results from the paper underscore the need for robust adversarial models in evaluating unlearning techniques. Population-based U-MIAs were shown to considerably underestimate the actual privacy risk, with attack success rates significantly higher when per-example U-MIAs were applied. This suggests that unlearning algorithms, if evaluated with insufficiently strong adversaries, may provide a misleading sense of privacy security.

The experiments conducted on vision and LLMs provide a solid foundation for the authors’ findings. They demonstrate that per-example U-MIAs consistently outperform population-based ones across various unlearning algorithms, such as SCRUB and SPARSITY. Additionally, the performance of these algorithms tended to degrade when faced with stronger, tailored attacks, suggesting an inherent vulnerability in their privacy protection when evaluated under more sophisticated adversarial conditions.

The paper also explores the unintended consequences of unlearning, namely that unlearning a subset of data can inadvertently increase the privacy risk of the remaining data in the training set. This poses a significant ethical and practical challenge to the design of unlearning algorithms and calls for a balanced consideration of privacy risks at both the individual and population levels.

In terms of future implications, the paper emphasizes the necessity for the research community to adopt more formal adversarial definitions and threat models, which can provide a clearer understanding of an unlearning method’s efficacy. Furthermore, the need for optimization strategies that consider individual sample vulnerabilities rather than blanket approaches is highlighted as a critical avenue for future research.

In conclusion, the insights presented in this paper accentuate the pitfalls of current unlearning methodologies and the crucial need for more nuanced evaluation mechanisms. By advocating for stronger adversary models and raising awareness of the potential privacy risks that remain inadequately addressed, the authors prompt a reevaluation of how privacy is currently managed in machine learning systems. Such efforts are vital to advancing the field towards truly secure and privacy-preserving machine learning practices.