Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Proximal Oracles for Optimization and Sampling (2404.02239v1)

Published 2 Apr 2024 in math.OC and cs.LG

Abstract: We consider convex optimization with non-smooth objective function and log-concave sampling with non-smooth potential (negative log density). In particular, we study two specific settings where the convex objective/potential function is either semi-smooth or in composite form as the finite sum of semi-smooth components. To overcome the challenges caused by non-smoothness, our algorithms employ two powerful proximal frameworks in optimization and sampling: the proximal point framework for optimization and the alternating sampling framework (ASF) that uses Gibbs sampling on an augmented distribution. A key component of both optimization and sampling algorithms is the efficient implementation of the proximal map by the regularized cutting-plane method. We establish the iteration-complexity of the proximal map in both semi-smooth and composite settings. We further propose an adaptive proximal bundle method for non-smooth optimization. The proposed method is universal since it does not need any problem parameters as input. Additionally, we develop a proximal sampling oracle that resembles the proximal map in optimization and establish its complexity using a novel technique (a modified Gaussian integral). Finally, we combine this proximal sampling oracle and ASF to obtain a Markov chain Monte Carlo method with non-asymptotic complexity bounds for sampling in semi-smooth and composite settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Solving convex programs by random walks. Journal of the ACM (JACM), 51(4):540–556, 2004.
  2. Nonasymptotic mixing of the MALA algorithm. IMA Journal of Numerical Analysis, 33(1):80–110, 2013.
  3. Near-optimal method for highly smooth convex optimization. In Conference on Learning Theory, pages 492–507. PMLR, 2019.
  4. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40(1):120–145, 2011.
  5. Improved analysis for a proximal algorithm for sampling. In Conference on Learning Theory, pages 2984–3014. PMLR, 2022.
  6. Arnak S Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(3):651–676, 2017.
  7. Analysis of Langevin Monte Carlo via convex optimization. The Journal of Machine Learning Research, 20(1):2666–2711, 2019.
  8. Efficient Bayesian computation by proximal Markov Chain Monte Carlo: when Langevin meets Moreau. SIAM Journal on Imaging Sciences, 11(1):473–506, 2018.
  9. A random polynomial-time algorithm for approximating the volume of convex bodies. Journal of the ACM (JACM), 38(1):1–17, 1991.
  10. Improved dimension dependence of a proximal algorithm for sampling. In The Thirty Sixth Annual Conference on Learning Theory, pages 1473–1521. PMLR, 2023.
  11. Optimal tensor methods in smooth convex and uniformly convexoptimization. In Conference on Learning Theory, pages 1374–1391. PMLR, 2019.
  12. Bayesian Data Analysis. CRC press, 2013.
  13. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, (6):721–741, 1984.
  14. Private convex optimization via exponential mechanism. In Conference on Learning Theory, pages 1948–1989. PMLR, 2022.
  15. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549–581, 1994.
  16. An optimal high-order tensor method for convex optimization. Mathematics of Operations Research, 46(4):1390–1412, 2021.
  17. Simulated annealing for convex optimization. Mathematics of Operations Research, 31(2):253–266, 2006.
  18. Random walks and an O*⁢(n5)superscript𝑂superscript𝑛5{O}^{*}(n^{5})italic_O start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ( italic_n start_POSTSUPERSCRIPT 5 end_POSTSUPERSCRIPT ) volume algorithm for convex bodies. Random Structures & Algorithms, 11(1):1–50, 1997.
  19. Complexity of a quadratic penalty accelerated inexact proximal point method for solving linearly constrained nonconvex composite programs. SIAM Journal on Optimization, 29(4):2566–2593, 2019.
  20. An accelerated inexact proximal point method for solving nonconvex-concave min-max problems. SIAM Journal on Optimization, 31(4):2558–2585, 2021.
  21. Galina M. Korpelevič. An extragradient method for finding saddle points and for other problems. Èkonom. i Mat. Metody, 12(4):747–756, 1976.
  22. Werner Krauth. Statistical mechanics: algorithms and computations, volume 13. OUP Oxford, 2006.
  23. Structured logconcave sampling with a restricted Gaussian oracle. In Conference on Learning Theory, pages 2993–3050. PMLR, 2021.
  24. A proximal algorithm for sampling from non-smooth potentials. In 2022 Winter Simulation Conference (WSC), pages 3229–3240. IEEE, 2022.
  25. A proximal algorithm for sampling. Transactions on Machine Learning Research, 2023.
  26. Jiaming Liang and Renato D. C. Monteiro. A doubly accelerated inexact proximal point method for nonconvex composite optimization problems. Available on arXiv:1811.11378, 2018.
  27. Jiaming Liang and Renato D. C. Monteiro. A proximal bundle variant with optimal iteration-complexity for a large range of prox stepsizes. SIAM Journal on Optimization, 31(4):2955–2986, 2021.
  28. Jiaming Liang and Renato D. C. Monteiro. A unified analysis of a class of proximal bundle methods for solving hybrid convex composite optimization problems. Mathematics of Operations Research, 2023.
  29. Proximal bundle methods for hybrid weakly convex composite optimization problems. arXiv preprint arXiv:2303.14896, 2023.
  30. Bernard Martinet. Regularisation d’inequations variationelles par approximations successives. Revue Francaise d’informatique et de Recherche operationelle, 4:154–159, 1970.
  31. An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM Journal on Optimization, 23(2):1092–1125, 2013.
  32. Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim., 23(1):475–507, 2013.
  33. An efficient sampling algorithm for non-smooth composite potentials. Available on arXiv:1910.00551, 2019.
  34. Radford M Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2(11):2, 2011.
  35. Problem complexity and method efficiency in optimization. Wiley, 1983.
  36. Optimal methods of smooth convex minimization. USSR Computational Mathematics and Mathematical Physics, 25(2):21–30, 1985.
  37. Yu Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programming, 152(1-2):381–404, 2015.
  38. Proximal algorithms. Foundations and Trends in optimization, 1(3):127–239, 2014.
  39. Giorgio Parisi. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378–384, 1981.
  40. Langevin diffusions and Metropolis-Hastings algorithms. Methodology and Computing in Applied Probability, 4(4):337–357, 2002.
  41. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, pages 341–363, 1996.
  42. R. T. Rockafellar. Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res., 1(2):97–116, 1976.
  43. R Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM journal on control and optimization, 14(5):877–898, 1976.
  44. Primal dual interpretation of the proximal stochastic gradient Langevin algorithm. Advances in Neural Information Processing Systems, 33:3786–3796, 2020.
  45. Composite logconcave sampling with a restricted Gaussian oracle. Available on arXiv:2006.05976, 2020.
  46. Delimiting species: a renaissance issue in systematic biology. Trends in Ecology & Evolution, 18(9):462–470, 2003.
  47. JG Wendel. Note on the gamma function. The American Mathematical Monthly, 55(9):563–564, 1948.
  48. Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. In Conference on Learning Theory, pages 2093–3027. PMLR, 2018.
  49. Variational transport: A convergent particle-based algorithm for distributional optimization. Available on arXiv:2012.11554, 2020.
  50. On a class of gibbs sampling over networks. In The Thirty Sixth Annual Conference on Learning Theory, pages 5754–5780. PMLR, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now