Swarm-based gradient descent meets simulated annealing (2404.18015v2)
Abstract: We introduce a novel method for non-convex optimization, called Swarm-based Simulated Annealing (SSA), which is at the interface between the swarm-based gradient-descent (SBGD) [J. Lu et. al., ArXiv:2211.17157; E.Tadmor and A. Zenginoglu, Acta Applicandae Math., 190, 2024] and Simulated Annealing (SA) [V. Cerny, J. optimization theory and appl., 45:41-51, 1985; S.Kirkpatrick et. al., Science, 220(4598):671-680, 1983; S. Geman and C.-R. Hwang, SIAM J. Control and Optimization, 24(5):1031-1043, 1986]. Similar to SBGD, we introduce a swarm of agents, each identified with a position, ${\mathbf x}$ and mass $m$, to explore the ambient space. Similar to SA, the agents proceed in the gradient descent direction, and are subject to Brownian motion. The annealing rate, however, is dictated by a decreasing function of their mass. As a consequence, instead of the SA protocol for time-decreasing temperature, we let the swarm decide how to cool down' agents, depending on their accumulated mass over time. The dynamics of masses is coupled with the dynamics of positions: agents at higher ground transfer (part of) their mass to those at lower ground. Consequently, resulting SSA optimizer is dynamically divided between heavier, cooler agents viewed asleaders' and lighter, warmer agents viewed as `explorers'. Mean-field convergence analysis and benchmark optimizations demonstrate the effectiveness of the swarm-based method as a multi-dimensional global optimizer.
- The wind driven optimization technique and its application in electromagnetics. IEEE transactions on antennas and propagation, 61(5):2745–2757, 2013.
- An analytical framework for consensus-based global optimization method. Mathematical Models and Methods in Applied Sciences, 28(06):1037–1066, 2018.
- A consensus-based global optimization method for high dimensional machine learning problems. ESAIM: Control, Optimisation and Calculus of Variations, 27, 2021.
- Consensus-based optimization and ensemble Kalman inversion for global optimization problems with constraints. Modeling and Simulation for Collective Dynamics, pages 195–230, 2023.
- Vladimír Černỳ. Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of optimization theory and applications, 45:41–51, 1985.
- Accelerating nonconvex learning via replica exchange Langevin diffusion. ICLR 2019, 2019.
- Diffusion for global optimization in ℝnsuperscriptℝ𝑛\mathbb{R}^{n}blackboard_R start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT. SIAM Journal on Control and Optimization, 25(3):737–753, 1987.
- Consensus-based optimization methods converge globally. ArXiv:2103.15130v4, 2021.
- On the rate of convergence in Wasserstein distance of the empirical measure. Probability Theory and Related Fields, 162(3):707–738, 2015.
- Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on pattern analysis and machine intelligence, (6):721–741, 1984.
- Diffusions for global optimization. SIAM Journal on Control and Optimization, 24(5):1031–1043, 1986.
- Basilis Gidas. Global optimization via the Langevin equation. In 1985 24th IEEE Conference on Decision and Control, pages 774–778. IEEE, 1985.
- Basilis Gidas. Nonstationary Markov chains and convergence of the annealing algorithm. Journal of Statistical Physics, 39:73–131, 1985.
- Mean-field particle swarm optimization. Modeling and Simulation for Collective Dynamics (W. Bao, P. Markovich, B. Perthame and E. Tadmor, eds.), 27:127–193, 2023.
- Johnson AW Henderson D, Jacobson SH. The theory and practice of simulated annealing, volume 57. Springer Science & Business Media, 2003.
- A comprehensive survey: artificial bee colony (abc) algorithm and applications. Artificial intelligence review, 42:21–57, 2014.
- Particle swarm optimization. In Proceedings of ICNN’95-international conference on neural networks, volume 4, pages 1942–1948. IEEE, 1995.
- Optimization by simulated annealing. Science, 220(4598):671–680, 1983.
- Peter J. M. Laarhoven and Emile H. L. Aarts. Simulated Annealing: Theory and Applications, chapter Simulated Annealing, pages 7–15. Springer Dordrecht, Oxford, 1987.
- Paul Langevin’s 1908 paper" on the theory of brownian motion"[" sur la thiorie du mouvement brownien," cr acad. sci.(paris) 146, 530-533 (1908)]. American Journal of Physics, 65:1079–1081, 1997.
- Swarm-based gradient descent method for non-convex optimization. ArXiv:2211.17157, 2022.
- Sylvie Méléard. Asymptotic behaviour of some interacting particle systems; McKean-Vlasov and Boltzmann models, pages 42–95. Springer Berlin Heidelberg, Berlin, Heidelberg, 1996.
- B Chandra Mohan and R Baskaran. A survey: Ant colony optimization based recent research and implementation on several engineering domain. Expert Systems with Applications, 39(4):4618–4627, 2012.
- Pierre Monmarché. Hypocoercivity in metastable settings and kinetic simulated annealing. Probability Theory and Related Fields, 172(3-4):1215–1248, 2018.
- A consensus-based model for global optimization and its mean-field limit. Mathematical Models and Methods in Applied Sciences, 27(01):183–204, 2017.
- Particle swarm optimization: An overview. Swarm intelligence, 1:33–57, 2007.
- Alain-Sol Sznitman. Topics in propagation of chaos. In Paul-Louis Hennequin, editor, Ecole d’Eté de Probabilités de Saint-Flour XIX — 1989, pages 165–251, Berlin, Heidelberg, 1991. Springer Berlin Heidelberg.
- Swarm-based optimization with random descent. Acta Applicandae Mathematicae, 190, 2024.
- Claudia Totzeck. Trends in consensus-based optimization. In Active Particles, Volume 3: Advances in Theory, Models, and Applications (Bellomo, N., Carrillo, JA and Tadmor, E., eds), pages 201–226. Springer, 2021.
- Xin-She Yang. Firefly algorithms for multimodal optimization. In International symposium on stochastic algorithms, pages 169–178. Springer, 2009.