Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Data Selection for Transfer Unlearning (2405.10425v1)

Published 16 May 2024 in cs.LG

Abstract: As deep learning models are becoming larger and data-hungrier, there are growing ethical, legal and technical concerns over use of data: in practice, agreements on data use may change over time, rendering previously-used training data impermissible for training purposes. These issues have driven increased attention to machine unlearning: removing "the influence of" a subset of training data from a trained model. In this work, we advocate for a relaxed definition of unlearning that does not address privacy applications but targets a scenario where a data owner withdraws permission of use of their data for training purposes. In this context, we consider the important problem of \emph{transfer unlearning} where a pretrained model is transferred to a target dataset that contains some "non-static" data that may need to be unlearned in the future. We propose a new method that uses a mechanism for selecting relevant examples from an auxiliary "static" dataset, and finetunes on the selected data instead of "non-static" target data; addressing all unlearning requests ahead of time. We also adapt a recent relaxed definition of unlearning to our problem setting and demonstrate that our approach is an exact transfer unlearner according to it, while being highly efficient (amortized). We find that our method outperforms the gold standard "exact unlearning" (finetuning on only the "static" portion of the target dataset) on several datasets, especially for small "static" sets, sometimes approaching an upper bound for test accuracy. We also analyze factors influencing the accuracy boost obtained by data selection.

PDF HTML Abstract

Rethinking the Meaning of Machine Unlearning

Introduction

Machine unlearning is emerging as a critical area in the field of AI, especially with the increasing complexity and data dependencies of deep learning models. The crux of the issue revolves around the need to "forget" specific training data when required by ethical, legal, or technical considerations. The paper we're looking at advocates for a relaxed definition of unlearning, particularly outside the privacy protection context. Instead, it focuses on scenarios where data owners revoke their permission to use their data for training purposes. This approach introduces "transfer unlearning," which tackles unlearning in transfer learning settings.

What is Transfer Unlearning?

Transfer learning is a widely used technique that involves adapting a pre-trained model on one dataset to work on another, potentially different, dataset. The efficiency this method brings saves significant computational resources and often boosts performance on tasks with limited training data. However, things get tricky when some of the data in the target dataset is "non-static," meaning permission to use it can be revoked.

Transfer unlearning seeks to handle this problem by allowing us to unlearn non-static data efficiently without compromising the utility of the model. The proposed method involves using an auxiliary static dataset to select relevant examples. These examples replace the non-static data points that may need to be unlearned in the future, effectively preempting any future unlearning requests.

The Proposed Method: Data Selection and Transfer Learning

The paper introduces a novel approach using a selection mechanism that picks relevant examples from an auxiliary "static" dataset. Here’s how it works:

  1. Data Selection: The method computes the similarity between each non-static data point and the candidates in the static dataset, using a pre-trained model's embedding space. This forms an average similarity score, and the top examples are selected to replace the non-static data.
  2. Transfer Learning: The selected examples are then used to fine-tune the pre-trained model. Notably, this means that the non-static data is never directly used for training, bypassing the need for expensive future unlearning.

Performance and Results

The proposed method was evaluated on nine diverse datasets, treating ImageNet as the source dataset. Here are some highlights from the results:

  • When the entirety of the target dataset was non-static, the proposed method significantly outperformed a random selection of examples from the auxiliary dataset. In some cases, it even approached the upper bound of performance that would be achieved if no unlearning were required.
  • When a portion of the target data was static, the method still outperformed the gold standard of exact unlearning (fine-tuning only with the static portion) in several cases, especially when the static set was small.
  • Factors such as "domain affinity" between the auxiliary and the target dataset played a significant role in the success of the method. Higher domain affinity translated into better performance.

Practical and Theoretical Implications

On the practical side, this approach offers an efficient and robust solution for models that need to handle data revocation flexibly. It avoids the computational overhead of repeated retraining or approximate unlearning procedures while ensuring that the model maintains high utility.

Theoretically, the notion of relaxed unlearning broadens the scope of what unlearning can achieve. By focusing on practical applicability, it strikes a balance between rigid theoretical guarantees and the realistic constraints of model training and unlearning in dynamic data environments.

Future Developments

The path ahead in transfer unlearning likely involves refining data selection mechanisms to further improve domain affinity and exploring other domains beyond computer vision. Additionally, newer definitions of unlearning could be developed to bridge the gap between exact and approximate methods, providing more nuanced guarantees.

Conclusion

The paper provides a compelling case for rethinking what unlearning means in practical scenarios. By leveraging data selection and auxiliary static datasets, it offers an efficient, theoretically sound method that significantly boosts model performance while accommodating data revocation requests. These advancements hold promise for more flexible and robust AI models in the future.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Deep learning methods for land cover and land use classification in remote sensing: A review. In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO), pp. 903–908. IEEE, 2020.
  2. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141–159. IEEE, 2021.
  3. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463–480. IEEE, 2015.
  4. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pp.  896–911, 2021a.
  5. Weighted training for cross-task learning. arXiv preprint arXiv:2105.14095, 2021b.
  6. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017.
  7. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3606–3613, 2014.
  8. Imagenet large scale visual recognition challenge. In International Conference on Computer Vision (ICCV), 2009.
  9. Diabetic retinopathy detection, 2015. URL https://kaggle.com/competitions/diabetic-retinopathy-detection.
  10. Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pp.  1–12. Springer, 2006.
  11. From privacy-theoretic to relative notions of unlearning, 2024.
  12. Head2toe: Utilizing intermediate representations for better transfer learning. In International Conference on Machine Learning, pp. 6009–6033. PMLR, 2022.
  13. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611, 2006.
  14. Learn-to-share: A hardware-friendly transfer learning framework exploiting computation and parameter sharing. In Proceedings of the International Conference on Machine Learning, 2021.
  15. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
  16. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201.06640, 2022.
  17. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9304–9312, 2020a.
  18. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pp.  383–398. Springer, 2020b.
  19. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  792–801, 2021.
  20. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  21. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the Conference on Computer Vision and Pattern Recognition, 2019.
  22. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218, 2024.
  23. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, 2016.
  24. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning, 2019.
  25. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  26. Towards unbounded machine unlearning. arXiv preprint arXiv:2302.09880, 2023.
  27. Improved fine-tuning by better leveraging pre-training data. In Advances in Neural Information Processing Systems, 2021.
  28. Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’. Computer Law & Security Review, 29(3):229–235, 2013.
  29. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  30. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022.
  31. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp.  722–729. IEEE, 2008.
  32. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp.  3498–3505. IEEE, 2012.
  33. Pytorch: An imperative style, high-performance deep learning library. https://pytorch.org/, 2019.
  34. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
  35. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.13239, 2020.
  36. Learning multiple visual domains with residual adapters. Advances in Neural Information Processing Systems, 2017.
  37. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021.
  38. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp.  303–319. IEEE, 2022.
  39. Neurips 2023 - machine unlearning, 2023. URL https://kaggle.com/competitions/neurips-2023-machine-unlearning.
  40. Dataset projection: Finding target-aligned subsets of auxiliary data, 2022.
  41. Arcane: An efficient architecture for exact machine unlearning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp.  4006–4013, 2022.
  42. Identifying spurious biases early in training through the lens of simplicity bias. arXiv preprint arXiv:2305.18761, 2023.
  43. Large language model unlearning. arXiv preprint arXiv:2310.10683, 2023.
  44. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
Citations (3)
View on Semantic Scholar