Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation (2312.16823v1)
Abstract: Recently, serious concerns have been raised about the privacy issues related to training datasets in machine learning algorithms when including personal data. Various regulations in different countries, including the GDPR grant individuals to have personal data erased, known as 'the right to be forgotten' or 'the right to erasure'. However, there has been less research on effectively and practically deleting the requested personal data from the training set while not jeopardizing the overall machine learning performance. In this work, we propose a fast and novel machine unlearning paradigm at the layer level called layer attack unlearning, which is highly accurate and fast compared to existing machine unlearning algorithms. We introduce the Partial-PGD algorithm to locate the samples to forget efficiently. In addition, we only use the last layer of the model inspired by the Forward-Forward algorithm for unlearning process. Lastly, we use Knowledge Distillation (KD) to reliably learn the decision boundaries from the teacher using soft label information to improve accuracy performance. We conducted extensive experiments with SOTA machine unlearning models and demonstrated the effectiveness of our approach for accuracy and end-to-end unlearning performance.
- Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), 141–159. IEEE.
- Burgess, M. 2023. ChatGPT Has a Big Privacy Problem. https://www.wired.com/story/italy-ban-chatgpt-privacy-gdpr/.
- Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), 67–74. IEEE.
- Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, 463–480. IEEE.
- Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. arXiv preprint arXiv:2301.11578.
- Boundary Unlearning. arXiv preprint arXiv:2303.11570.
- Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9304–9312.
- Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
- Hinton, G. 2022. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345.
- Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
- Kaye, K. 2023. The FTC’s ’profoundly vague’ plan to force companies to destroy algorithms could get very messy. https://www.protocol.com/enterprise/ftc-algorithm-data-model-ai/.
- Learning multiple layers of features from tiny images.
- Kublik, V. 2023. EU/US Copyright Law and Implications on ML Training Data. https://valohai.com/blog/copyright-laws-and-machine-learning/.
- Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
- Mantelero, A. 2013. The EU Proposal for a General Data Protection Regulation and the roots of the ‘right to be forgotten’. Computer Law & Security Review, 29(3): 229–235.
- A survey of machine unlearning. arXiv preprint arXiv:2209.02299.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
- Learning with Selective Forgetting. In IJCAI, volume 3, 4.
- Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
- Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.