Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates (2403.11687v3)

Published 18 Mar 2024 in stat.ML, cs.LG, and math.OC

Abstract: We study the problem of efficiently computing the derivative of the fixed-point of a parametric nondifferentiable contraction map. This problem has wide applications in machine learning, including hyperparameter optimization, meta-learning and data poisoning attacks. We analyze two popular approaches: iterative differentiation (ITD) and approximate implicit differentiation (AID). A key challenge behind the nonsmooth setting is that the chain rule does not hold anymore. We build upon the work by Bolte et al. (2022), who prove linear convergence of nonsmooth ITD under a piecewise Lipschitz smooth assumption. In the deterministic case, we provide a linear rate for AID and an improved linear rate for ITD which closely match the ones for the smooth setting. We further introduce NSID, a new stochastic method to compute the implicit derivative when the contraction map is defined as the composition of an outer map and an inner map which is accessible only through a stochastic unbiased estimator. We establish rates for the convergence of NSID, encompassing the best available rates in the smooth setting. We also present illustrative experiments confirming our analysis.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 136–145. PMLR, 2017.
  2. Amortized implicit differentiation for stochastic bilevel optimization. In International Conference on Learning Representations, 2021.
  3. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
  4. Beer, G. Topologies on Closed and Closed Convex Sets, volume 268. Springer Science & Business Media, 1993.
  5. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019, 2019.
  6. Implicit differentiation of lasso-type models for hyperparameter optimization. In International Conference on Machine Learning, pp. 810–821. PMLR, 2020.
  7. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. The Journal of Machine Learning Research, 23(1):6680–6722, 2022.
  8. Efficient and modular implicit differentiation. Advances in Neural Information Processing Systems, 35:5230–5242, 2022.
  9. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188:19–51, 2021.
  10. Automatic differentiation of nonsmooth iterative algorithms. Advances in Neural Information Processing Systems, 35:26404–26417, 2022.
  11. Differentiating nonsmooth solutions to parametric monotone inclusion problems. SIAM Journal of Optimization, 34:71–97, 2024.
  12. Optimization methods for large-scale machine learning. SIAM review, 60(2):223–311, 2018.
  13. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34:25294–25307, 2021.
  14. Signal recovery by proximal forward-backward splitting. Multiscale Modeling & Simulation, 4(4):1168–1200, 2005.
  15. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning, pp. 1165–1173. PMLR, 2017.
  16. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp. 1568–1577. PMLR, 2018.
  17. Bilevel learning of the group lasso structure. Advances in Neural Information Processing Systems, 31, 2018.
  18. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018.
  19. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning, pp. 3748–3758. PMLR, 2020.
  20. Convergence properties of stochastic hypergradients. In International Conference on Artificial Intelligence and Statistics, pp.  3826–3834. PMLR, 2021.
  21. Bilevel optimization with a lower-level contraction: Optimal sample complexity without warm-start. Journal of Machine Learning Research, 24(167):1–37, 2023.
  22. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, volume 105. SIAM, 2008.
  23. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning, pp. 4882–4892. PMLR, 2021.
  24. Linearly constrained bilevel optimization: A smoothed implicit gradient approach. In International Conference on Machine Learning, pp. 16291–16325. PMLR, 2023.
  25. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10657–10665, 2019.
  26. Boml: A modularized bilevel optimization library in python for meta learning. In 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp.  1–2. IEEE, 2021.
  27. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp.  1540–1552. PMLR, 2020.
  28. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 2113–2122. PMLR, 2015.
  29. Towards poisoning of deep learning algorithms with back-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, pp. 27–38, 2017.
  30. Bilevel optimization with nonsmooth lower level problems. In Scale Space and Variational Methods in Computer Vision: 5th International Conference, pp.  654–665. Springer, 2015.
  31. Pedregosa, F. Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning, pp. 737–746, 2016.
  32. Meta-learning with implicit gradients. Advances in Neural Information Processing Systems, 32, 2019.
  33. A stochastic forward-backward splitting method for solving monotone inclusions in hilbert spaces. arXiv preprint arXiv:1403.7999, 2014.
  34. Convergence of stochastic proximal gradient algorithm. Applied Mathematics & Optimization, 82:891–917, 2020.
  35. Scholtes, S. Introduction to Piecewise Differentiable Equations. Springer Science & Business Media, 2012.
  36. Is feature selection secure against training data poisoning? In International Conference on Machine Learning, pp. 1689–1698. PMLR, 2015.
  37. Alternating projected sgd for equality-constrained bilevel optimization. In International Conference on Artificial Intelligence and Statistics, pp.  987–1023. PMLR, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube