Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimization by linear kinetic equations and mean-field Langevin dynamics (2401.05553v1)

Published 10 Jan 2024 in math.NA, cond-mat.stat-mech, and cs.NA

Abstract: Probably one of the most striking examples of the close connections between global optimization processes and statistical physics is the simulated annealing method, inspired by the famous Monte Carlo algorithm devised by Metropolis et al. in the middle of the last century. In this paper we show how the tools of linear kinetic theory allow to describe this gradient-free algorithm from the perspective of statistical physics and how convergence to the global minimum can be related to classical entropy inequalities. This analysis highlight the strong link between linear Boltzmann equations and stochastic optimization methods governed by Markov processes. Thanks to this formalism we can establish the connections between the simulated annealing process and the corresponding mean-field Langevin dynamics characterized by a stochastic gradient descent approach. Generalizations to other selection strategies in simulated annealing that avoid the acceptance-rejection dynamic are also provided.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. E. Aarts and J. Korst. Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing. John Wiley & Sons, Inc., 1989.
  2. Kinetic-based optimization enhanced by genetic dynamics. Mathematical Models and Methods in Applied Sciences, 33(14):2905–2933, 2023.
  3. G. Albi and L. Pareschi. Binary interaction algorithms for the simulation of flocking and swarming dynamics. Multiscale Modeling & Simulation, 11(1):1–29, 2013.
  4. Handbook of Evolutionary Computation. IOP Publishing Ltd., 1997.
  5. N. Bellomo and S.-Y. Ha. A quest toward a mathematical theory of the dynamics of swarms. Mathematical Models and Methods in Applied Sciences, 27(4):745–770, 2017.
  6. 3D face recognition using simulated annealing and the surface interpenetration measure. IEEE Transactions on Pattern Analysis & Machine Intelligence, 32(02):206–219, 2010.
  7. Binary interaction methods for high dimensional global optimization and machine learning. Appl. Math. Optim., 86(1):Paper No. 9, 41, 2022.
  8. C. L. Bihan and R. Winter. The grazing collisions limit from the linearized Boltzmann equation to the Landau equation for short-range potentials. Kinetic and Related Models, 16(5):654–675, 2023.
  9. C. M. Bishop. Pattern recognition and machine learning. Springer, 2006.
  10. Entropy dissipation estimates for the linear Boltzmann operator. Journal of Functional Analysis, 269(4):1028–1069, 2015.
  11. C. Blum and A. Roli. Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM computing surveys (CSUR), 35(3):268–308, 2003.
  12. S. Bobkov and P. Tetali. Modified logarithmic Sobolev inequalities in discrete settings. J. Theor. Probab., 19:289—336, 2006.
  13. A consensus-based algorithm for multi-objective optimization and its mean-field description. Proceedings of the IEEE Conference on Decision and Control, page 4131–4136, 2022.
  14. Constrained Consensus-Based Optimization. SIAM J. Optim., 33(1):211–236, 2023.
  15. G. Borghi and L. Pareschi. Kinetic description and convergence analysis of genetic algorithms for global optimization. preprint arXiv:2310.08562, 2023.
  16. R. Caflisch. The Boltzmann equation with a soft potential. i. Linear, spatially-homogeneous. Comm. Math. Phys., 74:71–95, 1980.
  17. On the rate of convergence to equilibrium for the linear Boltzmann equation with soft potentials. Journal of Mathematical Analysis and Applications, 462(1):801–839, 2018.
  18. An analytical framework for consensus-based global optimization method. Mathematical Models and Methods in Applied Sciences, 28(6):1037–1066, 2018.
  19. Consensus-based sampling. Studies in Applied Mathematics, 148(3):1069–1140, 2022.
  20. A consensus-based global optimization method for high dimensional machine learning problems. ESAIM: Control, Optimisation and Calculus of Variations, 27:S5, 2021.
  21. On the Generalized Langevin Equation for Simulated Annealing. SIAM/ASA J. Uncertain. Quantif., 11(1):139–167, 2023.
  22. L. Chizat. Mean-field Langevin dynamics : Exponential convergence and annealing. Transactions on Machine Learning Research, 8, 2022.
  23. B. Claude J. P. Convergence theorems for a class of simulated annealing algorithms on 𝐑dsuperscript𝐑𝑑{\bf R}^{d}bold_R start_POSTSUPERSCRIPT italic_d end_POSTSUPERSCRIPT. J. Appl. Probab., 29(4):885–895, 1992.
  24. L. Desvillettes. On asymptotics of the boltzmann equation when the collisions become grazing. Transport Theory and Statistical Physics, 21(3):259–276, 1992.
  25. L. Desvillettes and C. Villani. Entropic methods for the study of the long time behavior of kinetic equations. Transport Theory and Statistical Physics, 30(2-3):155–168, 2001.
  26. Consensus-based optimization on hypersurfaces: well-posedness and mean-field limit. Math. Models Methods Appl. Sci., 30(14):2725–2751, 2020.
  27. Consensus-based optimization on the sphere: convergence to global minimizers and machine learning. J. Mach. Learn. Res., 22:Paper No. 237, 55, 2021.
  28. Anisotropic diffusion in consensus-based optimization on the sphere. SIAM J. Optim., 32(3):1984–2012, 2022.
  29. Convergence of anisotropic consensus-based optimization in mean-field law. In J. L. Jiménez Laredo, J. I. Hidalgo, and K. O. Babaagba, editors, Applications of Evolutionary Computation, pages 738–754, Cham, 2022. Springer International Publishing.
  30. Non-Maxwellian kinetic equations modeling the dynamics of wealth distribution. Mathematical Models and Methods in Applied Sciences, 30(04):685–725, 2020.
  31. S. Geman and D. Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6(6):721–741, 1984.
  32. S. Geman and C.-R. Hwang. Diffusions for global optimization. SIAM J. Control and Optimization, 24(5):1031–1043, 1986.
  33. Handbook of metaheuristics, volume 2. Springer, 2010.
  34. S. Grassi and L. Pareschi. From particle swarm optimization to consensus based optimization: stochastic modeling and mean-field limit. Math. Models Methods Appl. Sci., 31(8):1625–1657, 2021.
  35. Convergence and error estimates for time-discrete consensus-based optimization algorithms. Numer. Math., 147(2):255–282, 2021.
  36. B. Hajek. Cooling schedules for optimal annealing. Mathematics of Operations Research, 13(2):311–329, 1988.
  37. W. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97–109, 1970.
  38. R. Holley and D. Stroock. Simulated annealing via Sobolev inequalities. Communications in Mathematical Physics, 115(4):553–569, 1988.
  39. H. Hoos and T. Stützle. Stochastic Local Search: Foundations & Applications. Morgan Kaufmann, 2005.
  40. Mean-field langevin dynamics and energy landscape of neural networks. Annales de l’Institut Henri Poincaré, Probabilités et Statistiques, 57(4):2043–2065, 2021.
  41. On the global convergence of particle swarm optimization methods. Appl. Math. Optim., 88:30, 2023.
  42. C.-R. Hwang. Laplace’s method revisited: weak convergence of probability measures. Ann. Probab., 8(6):1177–1182, 1980.
  43. Consensus-based optimization via jump-diffusion stochastic differential equations. Math. Models Methods Appl. Sci., 33(2):289–339, 2023.
  44. Optimization by simulated annealing. Science, 220(4598):671–680, 1983.
  45. V. N. Kolokoltsov. Nonlinear Markov Processes and Kinetic Equations. Cambridge University Press, 2010.
  46. Derivative-free optimization methods. Acta Numer., 28:287–404, 2019.
  47. M. Locatelli. Simulated annealing algorithms for continuous global optimization: convergence conditions. J. Optim. Theory Appl., 104(1):121–133, 2000.
  48. E. Marinari and G. Parisi. Simulated tempering: A new Monte Carlo scheme. Europhysics Letters, 19(6):451–458, 1992.
  49. Equation of state calculations by very fast computing machines. J. Chem. Phys., 21:1087, 1953.
  50. General relative entropy inequality: an illustration on growth models. Journal de Mathématiques Pures et Appliquées, 84(9):1235–1260, 2005.
  51. P. Monmarché. Hypocoercivity in metastable settings and kinetic simulated annealing. Probab. Theory Related Fields, 172(3-4):1215–1248, 2018.
  52. C. Mouhot and R. Strain. Spectral gap and coercivity estimates for linearized Boltzmann collision operators without angular cutoff. J. Math. Pures Appl., 87:515–535, 2007.
  53. L. Pareschi and G. Toscani. Interacting multiagent systems: kinetic equations and Monte Carlo methods. OUP Oxford, 2013.
  54. L. Pareschi and M. Zanella. Structure-preserving schemes for nonlinear fokker-planck equations and applications. Journal of Scientific Computing, 74(3):1575–1600, 2018.
  55. A consensus-based model for global optimization and its mean-field limit. Mathematical Models and Methods in Applied Sciences, 27(1):183–204, 2017.
  56. C. Totzeck. Trends in consensus-based optimization. In Active particles. Vol. 3. Advances in theory, models, and applications, Model. Simul. Sci. Eng. Technol., pages 201–226. Birkhäuser/Springer, Cham, 2022.
  57. C. Tsallis and D. A. Stariolo. Generalized simulated annealing. Physica A: Statistical Mechanics and its Applications, 233(1):395–406, 1996.
  58. C. Villani. A review of mathematical topics in collisional kinetic theory. In Handbook of Mathematical Fluid Dynamics, Vol. I, pages 71–305. North-Holland, Amsterdam, 2002.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now