Ticketed Learning-Unlearning Schemes (2306.15744v1)
Abstract: We consider the learning--unlearning paradigm defined as follows. First given a dataset, the goal is to learn a good predictor, such as one minimizing a certain loss. Subsequently, given any subset of examples that wish to be unlearnt, the goal is to learn, without the knowledge of the original training dataset, a good predictor that is identical to the predictor that would have been produced when learning from scratch on the surviving examples. We propose a new ticketed model for learning--unlearning wherein the learning algorithm can send back additional information in the form of a small-sized (encrypted) ticket'' to each participating training example, in addition to retaining a small amount ofcentral'' information for later. Subsequently, the examples that wish to be unlearnt present their tickets to the unlearning algorithm, which additionally uses the central information to return a new predictor. We provide space-efficient ticketed learning--unlearning schemes for a broad family of concept classes, including thresholds, parities, intersection-closed classes, among others. En route, we introduce the count-to-zero problem, where during unlearning, the goal is to simply know if there are any examples that survived. We give a ticketed learning--unlearning scheme for this problem that relies on the construction of Sperner families with certain properties, which might be of independent interest.
- A new PAC bound for intersection-closed concept classes. Machine Learning, 66(2):151–163, 2007.
- Machine unlearning. In S & P, 2021.
- Proper learning, Helly number, and an optimal SVM bound. In COLT, pages 582–609, 2020.
- Machine unlearning for random forests. In ICML, pages 1092–1104, 2021.
- Towards making systems forget with machine unlearning. In S & P, pages 463–480, 2015.
- Extracting training data from large language models. In USENIX Security, 2021.
- Membership inference attacks from first principles. In S & P, pages 1897–1914, 2022a.
- The privacy onion effect: Memorization is relative. In NeurIPS, 2022b.
- Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188, 2023a.
- Quantifying memorization across neural language models. In ICLR, 2023b.
- Incremental and decremental support vector machine learning. NIPS, 2000.
- Forget unlearning: Towards true data-deletion in machine learning. In ICML, 2023.
- Control, confidentiality, and the right to be forgotten. In TPDP, 2022.
- Hidden poison: Machine unlearning enables camouflaged poisoning attacks. In NeurIPS ML Safety Workshop, 2022.
- Lifelong anomaly detection through unlearning. In CCS, pages 1283–1297, 2019.
- SAFE: Machine unlearning with shard graphs. arXiv preprint arXiv:2304.13169, 2023.
- Calibrating noise to sensitivity in private data analysis. In TCC, pages 265–284, 2006.
- Verifiable and provably secure machine unlearning. arXiv preprint arXiv:2210.09126, 2022.
- Konrad Engel. Sperner Theory. Cambridge University Press, 1997.
- Formalizing data deletion in the context of the right to be forgotten. In EUROCRYPT, pages 373–402, 2020.
- Making AI forget you: Data deletion in machine learning. In NeurIPS, pages 3518–3531, 2019.
- Eternal sunshine of the spotless net: Selective forgetting in deep networks. In CVPR, 2020.
- Mixed-privacy forgetting in deep networks. In CVPR, pages 792–801, 2021.
- Amnesiac machine learning. In AAAI, pages 11516–11524, 2021.
- Certified data removal from machine learning models. In ICML, pages 3832–3842, 2020.
- Adaptive machine unlearning. In NeurIPS, pages 16319–16330, 2021.
- Characterizing history independent data structures. Algorithmica, 42:57–74, 2005.
- Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022.
- Approximate data deletion from machine learning models: Algorithms and evaluation. In AISTATS, pages 2008–2016, 2021.
- Multiple incremental decremental learning of support vector machines. IEEE Transactions on Neural Networks, 21(7):1048–1059, 2010.
- On randomized one-round communication complexity. Computational Complexity, 8(1):21–49, 1999.
- Towards bridging the gaps between the right to explanation and the right to be forgotten. In ICML, 2023.
- Relating data compression and learnability. Unpublished manuscript, 1986.
- Anti-persistence: History independent data structures. In STOC, pages 492–501, 2001.
- Descent-to-delete: Gradient-based methods for machine unlearning. In ALT, 2021.
- Variational Bayesian unlearning. In NeurIPS, pages 16025–16036, 2020.
- Incremental and decremental learning for linear support vector machines. In ICANN, pages 209–218, 2007.
- Remember what you want to forget: Algorithms for machine unlearning. In NeurIPS, pages 18075–18086, 2021.
- Membership inference attacks against machine learning models. In S & P, pages 3–18, 2017.
- Algorithms that approximate data removal: New results and limitations. In NeurIPS, 2022.
- Eleftherios Tachtsis. On Ramsey’s theorem and the existence of infinite chains or infinite anti-chains in infinite posets. J. Symb. Logic, 81(1):384–394, 2016.
- Unrolling SGD: Understanding factors influencing machine unlearning. In EuroS&P, pages 303–319, 2022.
- Incremental and decremental proximal support vector classification using decay coefficients. In DaWak, pages 422–429, 2003.
- Machine unlearning via algorithmic stability. In COLT, pages 4126–4142, 2021.
- Analyzing information leakage of updates to natural language models. In CCS, pages 363–375, 2020.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.