Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems (2311.02240v2)
Abstract: Machine unlearning is a crucial tool for enabling a classification model to forget specific data that are used in the training time. Recently, various studies have presented machine unlearning algorithms and evaluated their methods on several datasets. However, most of the current machine unlearning algorithms have been evaluated solely on traditional computer vision datasets such as CIFAR-10, MNIST, and SVHN. Furthermore, previous studies generally evaluate the unlearning methods in the class-unlearning setup. Most previous work first trains the classification models and then evaluates the machine unlearning performance of machine unlearning algorithms by forgetting selected image classes (categories) in the experiments. Unfortunately, these class-unlearning settings might not generalize to real-world scenarios. In this work, we propose a machine unlearning setting that aims to unlearn specific instance that contains personal privacy (identity) while maintaining the original task of a given model. Specifically, we propose two machine unlearning benchmark datasets, MUFAC and MUCAC, that are greatly useful to evaluate the performance and robustness of a machine unlearning algorithm. In our benchmark datasets, the original model performs facial feature recognition tasks: face age estimation (multi-class classification) and facial attribute classification (binary class classification), where a class does not depend on any single target subject (personal identity), which can be a realistic setting. Moreover, we also report the performance of the state-of-the-art machine unlearning methods on our proposed benchmark datasets. All the datasets, source codes, and trained models are publicly available at https://github.com/ndb796/MachineUnlearning.
- Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), 141–159. IEEE.
- Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901.
- Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2633–2650.
- Can bad teaching induce forgetting? Unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 7210–7217.
- Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security.
- Evaluating inexact unlearning requires revisiting forgetting. arXiv preprint arXiv:2201.06640.
- Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9304–9312.
- Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In ECCV 2020: 16th European Conference, Glasgow, UK, 2020, Proceedings, 383–398. Springer.
- Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34: 16319–16330.
- A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1): 87–110.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, 4171–4186.
- Learning multiple layers of features from tiny images.
- Towards Unbounded Machine Unlearning. arXiv preprint arXiv:2302.09880.
- GPT understands, too. AI Open.
- Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018): 11.
- Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10422–10431.
- Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, 931–962. PMLR.
- Reading digits in natural images with unsupervised feature learning.
- Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34: 18075–18086.
- Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), 3–18. IEEE.
- Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, 6105–6114. PMLR.
- Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems.
- Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), 303–319. IEEE.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Machine unlearning via algorithmic stability. In Conference on Learning Theory, 4126–4142. PMLR.
- Attention is all you need. Advances in neural information processing systems, 30.
- Federated unlearning via class-discriminative pruning. In Proceedings of the ACM Web Conference 2022, 622–632.
- Bag of Tricks for Training Data Extraction from Language Models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23.
- Machine unlearning for image retrieval: A generative scrubbing approach. In Proceedings of the 30th ACM International Conference on Multimedia, 237–245.