Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning (2401.10371v5)

Published 18 Jan 2024 in cs.LG

Abstract: Machine unlearning has raised significant interest with the adoption of laws ensuring the ``right to be forgotten''. Researchers have provided a probabilistic notion of approximate unlearning under a similar definition of Differential Privacy (DP), where privacy is defined as statistical indistinguishability to retraining from scratch. We propose Langevin unlearning, an unlearning framework based on noisy gradient descent with privacy guarantees for approximate unlearning problems. Langevin unlearning unifies the DP learning process and the privacy-certified unlearning process with many algorithmic benefits. These include approximate certified unlearning for non-convex problems, complexity saving compared to retraining, sequential and batch unlearning for multiple unlearning requests.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.  308–318, 2016.
  2. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems, 35:3788–3800, 2022a.
  3. Resolving the mixing time of the langevin algorithm to its stationary distribution for log-concave sampling. arXiv preprint arXiv:2210.08448, 2022b.
  4. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp.  141–159. IEEE, 2021.
  5. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp.  463–480. IEEE, 2015.
  6. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp.  267–284, 2019.
  7. Dimension-free log-sobolev inequalities for mixture distributions. Journal of Functional Analysis, 281(11):109236, 2021.
  8. Efficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations, 2022.
  9. Forget unlearning: Towards true data-deletion in machine learning. In International Conference on Machine Learning, pp.  6028–6073. PMLR, 2023.
  10. Differential privacy dynamics of langevin diffusion and noisy gradient descent. Advances in Neural Information Processing Systems, 34:14771–14781, 2021.
  11. Sparse regression learning by aggregation and langevin monte-carlo. Journal of Computer and System Sciences, 78(5):1423–1443, 2012.
  12. Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  13. Nonasymptotic convergence analysis for the unadjusted langevin algorithm. 2017.
  14. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.  265–284. Springer, 2006.
  15. Convergence of langevin monte carlo in chi-squared and rényi divergence. In International Conference on Artificial Intelligence and Statistics, pp.  8151–8175. PMLR, 2022.
  16. Privacy amplification by iteration. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pp.  521–532. IEEE, 2018.
  17. Bayesian inference forgetting. arXiv preprint arXiv:2101.06417, 2021.
  18. Faster differentially private samplers via rényi divergence analysis of discretized langevin mcmc. Advances in Neural Information Processing Systems, 33:7222–7233, 2020.
  19. Gronwall, T. H. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics, pp.  292–296, 1919.
  20. Gross, L. Logarithmic sobolev inequalities. American Journal of Mathematics, 97(4):1061–1083, 1975.
  21. Certified data removal from machine learning models. In International Conference on Machine Learning, pp.  3832–3842. PMLR, 2020.
  22. Adaptive machine unlearning. Advances in Neural Information Processing Systems, 34:16319–16330, 2021.
  23. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp.  1225–1234. PMLR, 2016.
  24. Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. 1970.
  25. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  26. The composition theorem for differential privacy. In International conference on machine learning, pp.  1376–1385. PMLR, 2015.
  27. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning, pp.  5213–5225. PMLR, 2021.
  28. Krizhevsky, A. et al. Learning multiple layers of features from tiny images. 2009.
  29. Lamperski, A. Projected stochastic gradient langevin algorithms for constrained sampling and non-convex learning. In Conference on Learning Theory, pp.  2891–2937. PMLR, 2021.
  30. Sampling can be faster than optimization. Proceedings of the National Academy of Sciences, 116(42):20881–20885, 2019.
  31. maintainers, T. and contributors. Torchvision: Pytorch’s computer vision library. https://github.com/pytorch/vision, 2016.
  32. Markov chains and stochastic stability. Springer Science & Business Media, 2012.
  33. Mironov, I. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp.  263–275. IEEE, 2017.
  34. Towards a complete analysis of langevin monte carlo: Beyond poincar\\\backslash\’e inequality. arXiv preprint arXiv:2303.03589, 2023.
  35. Neal, R. M. et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
  36. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pp.  931–962. PMLR, 2021.
  37. Variational bayesian unlearning. Advances in Neural Information Processing Systems, 33:16025–16036, 2020.
  38. Markov chain monte carlo-based machine unlearning: Unlearning what needs to be forgotten. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, pp.  351–363, 2022.
  39. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp.  8024–8035. Curran Associates, Inc., 2019.
  40. Challenges and pitfalls of bayesian unlearning. arXiv preprint arXiv:2207.03227, 2022.
  41. Differential privacy guarantees for stochastic gradient langevin dynamics. arXiv preprint arXiv:2201.11980, 2022.
  42. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075–18086, 2021.
  43. Sinho Chewi. Log-Concave Sampling. https://chewisinho.github.io/main.pdf, 2023. Online; accessed September 29, 2023.
  44. From adaptive query release to machine unlearning. In International Conference on Machine Learning, pp.  34642–34667. PMLR, 2023.
  45. Machine unlearning via algorithmic stability. In Conference on Learning Theory, pp.  4126–4142. PMLR, 2021.
  46. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems, 32, 2019.
  47. Differentially private learning needs hidden state (or much faster convergence). Advances in Neural Information Processing Systems, 35:703–715, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now