Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Communication Efficient and Provable Federated Unlearning (2401.11018v1)

Published 19 Jan 2024 in cs.LG and cs.DC

Abstract: We study federated unlearning, a novel problem to eliminate the impact of specific clients or data points on the global model learned via federated learning (FL). This problem is driven by the right to be forgotten and the privacy challenges in FL. We introduce a new framework for exact federated unlearning that meets two essential criteria: \textit{communication efficiency} and \textit{exact unlearning provability}. To our knowledge, this is the first work to tackle both aspects coherently. We start by giving a rigorous definition of \textit{exact} federated unlearning, which guarantees that the unlearned model is statistically indistinguishable from the one trained without the deleted data. We then pinpoint the key property that enables fast exact federated unlearning: total variation (TV) stability, which measures the sensitivity of the model parameters to slight changes in the dataset. Leveraging this insight, we develop a TV-stable FL algorithm called \texttt{FATS}, which modifies the classical \texttt{\underline{F}ed\underline{A}vg} algorithm for \underline{T}V \underline{S}tability and employs local SGD with periodic averaging to lower the communication round. We also design efficient unlearning algorithms for \texttt{FATS} under two settings: client-level and sample-level unlearning. We provide theoretical guarantees for our learning and unlearning algorithms, proving that they achieve exact federated unlearning with reasonable convergence rates for both the original and unlearned models. We empirically validate our framework on 6 benchmark datasets, and show its superiority over state-of-the-art methods in terms of accuracy, communication cost, computation cost, and unlearning efficacy.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390 (2020).
  2. Machine unlearning. In 42nd IEEE Symposium on Security and Privacy (SP 2021). IEEE, 141–159.
  3. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 (2018).
  4. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 36th IEEE Symposium on Security and Privacy (SP 2015). IEEE, 463–480.
  5. Graph unlearning. In 29th ACM SIGSAC Conference on Computer and Communications Security (CCS 2022). 499–513.
  6. Calibrating noise to sensitivity in private data analysis. In 3rd Theory of Cryptography Conference (TCC 2006). Springer, 265–284.
  7. Sequential Informed Federated Unlearning: Efficient and Provable Client Unlearning in Federated Optimization. arXiv preprint arXiv:2211.11656 (2022).
  8. Making ai forget you: Data deletion in machine learning. , 3513–3526 pages.
  9. Certified data removal from machine learning models. In 37th International Conference on Machine Learning (ICML 2020). 3832–3842.
  10. Federated Unlearning: How to Efficiently Erase a Client in FL? arXiv preprint arXiv:2207.05521 (2022).
  11. Understanding the scope and impact of the california consumer privacy act of 2018. Journal of Data Protection & Privacy 2, 3 (2019), 234–253.
  12. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019).
  13. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14, 1–2 (2021), 1–210.
  14. Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto (2009).
  15. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
  16. FedEraser: Enabling efficient client-level data removal from federated learning models. In 29th IEEE/ACM International Symposium on Quality of Service (IWQOS 2021). IEEE, 1–10.
  17. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In 41st IEEE International Conference on Computer Communications (INFOCOM 2022). IEEE, 1749–1758.
  18. Communication-efficient learning of deep networks from decentralized data. In 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017). PMLR, 1273–1282.
  19. Descent-to-delete: Gradient-based methods for machine unlearning. In 32nd International Conference on Algorithmic Learning Theory (ALT 2021). PMLR, 931–962.
  20. A survey of machine unlearning. arXiv preprint arXiv:2209.02299 (2022).
  21. Remember what you want to forget: Algorithms for machine unlearning. In 35th Annual Conference on Neural Information Processing Systems (NeurIPS 2021). 18075–18086.
  22. Membership Inference Attacks Against Machine Learning Models. In 38th IEEE Symposium on Security and Privacy (SP 2017). IEEE Computer Society, 3–18.
  23. Analyzing user-level privacy attack against federated learning. IEEE Journal on Selected Areas in Communications 38, 10 (2020), 2430–2444.
  24. Unrolling sgd: Understanding factors influencing machine unlearning. In 7th IEEE European Symposium on Security and Privacy (EuroS&P 2022). IEEE, 303–319.
  25. Machine unlearning via algorithmic stability. In 32nd International Conference on Algorithmic Learning Theory (ALT 2021). PMLR, 4126–4142.
  26. Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10–5555.
  27. Inductive Graph Unlearning. In 32nd USENIX Security Symposium (USENIX Security 2023). 3205–3222.
  28. Federated unlearning via class-discriminative pruning. In 31st ACM Web Conference (WWW 2022. 622–632.
  29. Beyond inferring class representatives: User-level privacy leakage from federated learning. In 38th IEEE International Conference on Computer Communications (INFOCOM 2019). IEEE, 2512–2520.
  30. Federated unlearning with knowledge distillation. arXiv preprint arXiv:2201.09441 (2022).
  31. Federated Unlearning: Guarantee the Right of Clients to Forget. IEEE Network 36, 5 (2022), 129–135.
  32. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).
  33. Deep Leakage from Gradients. 33rd Annual Conference on Neural Information Processing Systems (NeurIPS 2019) 32 (2019), 14774–14784.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Youming Tao (7 papers)
  2. Cheng-Long Wang (10 papers)
  3. Miao Pan (42 papers)
  4. Dongxiao Yu (60 papers)
  5. Xiuzhen Cheng (72 papers)
  6. Di Wang (407 papers)
Citations (10)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets