Papers
Topics
Authors
Recent
AI Research Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 60 tok/s
Gemini 2.5 Pro 46 tok/s Pro
GPT-5 Medium 14 tok/s Pro
GPT-5 High 15 tok/s Pro
GPT-4o 93 tok/s Pro
Kimi K2 156 tok/s Pro
GPT OSS 120B 441 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Mean-field limits for Consensus-Based Optimization and Sampling (2312.07373v4)

Published 12 Dec 2023 in math.PR, math.AP, and math.OC

Abstract: For algorithms based on interacting particle systems that admit a mean-field description, convergence analysis is often more accessible at the mean-field level. In order to transfer convergence results obtained at the mean-field level to the finite ensemble size setting, it is desirable to show that the particle dynamics converge in an appropriate sense to the corresponding mean-field dynamics. In this paper, we prove quantitative mean-field limit results for two related interacting particle systems: Consensus-Based Optimization and Consensus-Based Sampling. Our approach requires a generalization of Sznitman's classical argument: in order to circumvent issues related to the lack of global Lipschitz continuity of the coefficients, we discard an event of small probability, the contribution of which is controlled using moment estimates for the particle systems. In addition, we present new results on the well-posedness of the particle systems and their mean-field limit, and provide novel stability estimates for the weighted mean and the weighted covariance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. “Importance sampling: intrinsic dimension and computational cost” In Statist. Sci. 32.3, 2017, pp. 405–431 DOI: 10.1214/17-STS611
  2. Konstantin Althaus, Iason Papaioannou and Elisabeth Ullmann “Consensus-based rare event estimation” In arXiv preprint 2304.09077, 2023
  3. “An inequality for Hilbert-Schmidt norm” In Comm. Math. Phys. 81.1, 1981, pp. 89–96 URL: http://projecteuclid.org/euclid.cmp/1103920160
  4. “A constrained consensus based optimization algorithm and its application to finance” In Appl. Math. Comput. 416, 2022, pp. Paper No. 126726\bibrangessep10 DOI: 10.1016/j.amc.2021.126726
  5. Rajendra Bhatia “Matrix analysis” 169, Graduate Texts in Mathematics Springer-Verlag, New York, 1997 DOI: 10.1007/978-1-4612-0653-8
  6. Rajendra Bhatia “Matrix factorizations and their perturbations” Second Conference of the International Linear Algebra Society (Lisbon, 1992) In Linear Algebra Appl. 197/198, 1994, pp. 245–276 DOI: 10.1016/0024-3795(94)90490-1
  7. Giacomo Borghi, Michael Herty and Lorenzo Pareschi “An Adaptive Consensus Based Method for Multi-objective Optimization with Uniform Pareto Front Approximation” In Appl. Math. Optim. 88.2, 2023, pp. Paper No. 58 DOI: 10.1007/s00245-023-10036-y
  8. Giacomo Borghi, Michael Herty and Lorenzo Pareschi “Constrained consensus-based optimization” In SIAM J. Optim. 33.1, 2023, pp. 211–236 DOI: 10.1137/22M1471304
  9. Leon Bungert, Tim Roith and Philipp Wacker “Polarized consensus-based dynamics for optimization and sampling” In arXiv preprint 2211.05238, 2022 DOI: 10.48550/ARXIV.2211.05238
  10. “An analytical framework for consensus-based global optimization method” In Math. Models Methods Appl. Sci. 28.6, 2018, pp. 1037–1066 DOI: 10.1142/S0218202518500276
  11. “Consensus-based sampling” In Stud. Appl. Math. 148.3, 2022, pp. 1069–1140 DOI: 10.1111/sapm.12470
  12. J.A. Carrillo, C. Totzeck and U. Vaes “Consensus-based optimization and ensemble Kalman inversion for global optimization problems with constraints” In arXiv preprint 2111.02970, 2021 URL: https://arxiv.org/abs/2111.02970
  13. “A consensus-based global optimization method for high dimensional machine learning problems” In ESAIM Control Optim. Calc. Var. 27, 2021, pp. Paper No. S5\bibrangessep22 DOI: 10.1051/cocv/2020046
  14. “Propagation of chaos: a review of models, methods and applications. I. Models and methods” In Kinet. Relat. Models 15.6, 2022, pp. 895–1015 DOI: 10.3934/krm.2022017
  15. “Propagation of chaos: a review of models, methods and applications. II. Applications” In Kinet. Relat. Models 15.6, 2022, pp. 1017–1173 DOI: 10.3934/krm.2022018
  16. Yuan Shih Chow and Henry Teicher “Probability theory” Independence, interchangeability, martingales, Springer Texts in Statistics Springer-Verlag, New York, 1997 DOI: 10.1007/978-1-4612-1950-7
  17. Cristina Cipriani, Hui Huang and Jinniao Qiu “Zero-inertia limit: from particle swarm optimization to consensus-based optimization” In SIAM J. Math. Anal. 54.3, 2022, pp. 3091–3121 DOI: 10.1137/21M1412323
  18. “Evaluation for moments of a ratio with application to regression estimation” In Bernoulli 15.4, 2009, pp. 1259–1286 DOI: 10.3150/09-BEJ190
  19. Manh Hong Duong and Hung D. Nguyen “Asymptotic analysis for the generalized Langevin equation with singular potentials” In arXiv preprint 2305.03637, 2023
  20. Stewart N. Ethier and Thomas G. Kurtz “Markov processes” Characterization and convergence, Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics John Wiley & Sons, Inc., New York, 1986 DOI: 10.1002/9780470316658
  21. “Consensus-based optimization on hypersurfaces: well-posedness and mean-field limit” In Math. Models Methods Appl. Sci. 30.14, 2020, pp. 2725–2751 DOI: 10.1142/S0218202520500530
  22. “Consensus-based optimization on the sphere: convergence to global minimizers and machine learning” In J. Mach. Learn. Res. 22, 2021, pp. Paper No. 237\bibrangessep55 URL: https://dl.acm.org/doi/abs/10.5555/3546258.3546495
  23. Massimo Fornasier, Timo Klock and Konstantin Riedl “Consensus-based optimization methods converge globally” In arXiv preprint 2103.15130, 2021 URL: https://arxiv.org/abs/2103.15130
  24. Massimo Fornasier, Timo Klock and Konstantin Riedl “Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law” In Applications of Evolutionary Computation Cham: Springer International Publishing, 2022, pp. 738–754
  25. Avner Friedman “Stochastic Differential Equations and Applications” In Stochastic differential equations 77, C.I.M.E. Summer Sch. Springer, Heidelberg, 2010, pp. 75–148 DOI: 10.1007/978-3-642-11079-5\_2
  26. “Interacting Langevin diffusions: gradient structure and ensemble Kalman sampler” In SIAM J. Appl. Dyn. Syst. 19.1, 2020, pp. 412–441 DOI: 10.1137/19M1251655
  27. David Gilbarg and Neil S. Trudinger “Elliptic Partial Differential Equations of Second Order” Reprint of the 1998 edition, Classics in Mathematics Springer-Verlag, Berlin, 2001
  28. “Stochastic Simulation and Monte Carlo Methods” Mathematical foundations of stochastic simulation 68, Stochastic Modelling and Applied Probability Springer, Heidelberg, 2013 DOI: 10.1007/978-3-642-39363-1
  29. “From particle swarm optimization to consensus based optimization: stochastic modeling and mean-field limit” In Math. Models Methods Appl. Sci. 31.8, 2021, pp. 1625–1657 DOI: 10.1142/S0218202521500342
  30. Seung-Yeal Ha, Shi Jin and Doheon Kim “Convergence of a first-order consensus-based global optimization algorithm” In Math. Models Methods Appl. Sci. 30.12, 2020, pp. 2417–2444 DOI: 10.1142/S0218202520500463
  31. “Convergence and error estimates for time-discrete consensus-based optimization algorithms” In Numerische Mathematik, 2020, pp. 1–28
  32. “Stochastic consensus dynamics for nonconvex optimization on the Stiefel manifold: mean-field limit and convergence” In Math. Models Methods Appl. Sci. 32.3, 2022, pp. 533–617 DOI: 10.1142/S0218202522500130
  33. Desmond J. Higham, Xuerong Mao and Andrew M. Stuart “Strong convergence of Euler-type methods for nonlinear stochastic differential equations” In SIAM J. Numer. Anal. 40.3, 2002, pp. 1041–1063 DOI: 10.1137/S0036142901389530
  34. “On the mean-field limit for the consensus-based optimization” In Math. Methods Appl. Sci. 45.12, 2022, pp. 7814–7831
  35. Shi Jin, Lei Li and Jian-Guo Liu “Random batch methods (RBM) for interacting particle systems” In J. Comput. Phys. 400, 2020, pp. 108877\bibrangessep30 DOI: 10.1016/j.jcp.2019.108877
  36. Dante Kalise, Akash Sharma and Michael V. Tretyakov “Consensus-based optimization via jump-diffusion stochastic differential equations” In Math. Models Methods Appl. Sci. 33.2, 2023, pp. 289–339 DOI: 10.1142/S0218202523500082
  37. “Particle swarm optimization” In Proceedings of ICNN’95-international conference on neural networks 4 IEEE, 1995, pp. 1942–1948 URL: https://doi.org/10.1109/ICNN.1995.488968
  38. Rafail Khasminskii “Stochastic Stability of Differential Equations” With contributions by G. N. Milstein and M. B. Nevelson 66, Stochastic Modelling and Applied Probability Springer, Heidelberg, 2012 DOI: 10.1007/978-3-642-23280-0
  39. Fuad Kittaneh “On Lipschitz functions of normal operators” In Proc. Amer. Math. Soc. 94.3, 1985, pp. 416–418 DOI: 10.2307/2045225
  40. Kathrin Klamroth, Michael Stiglmayr and Claudia Totzeck “Consensus-Based Optimization for Multi-Objective Problems: A Multi-Swarm Approach” In arXiv preprint 2211.15737, 2022 URL: https://arxiv.org/abs/2211.15737
  41. “Convergence analysis of the discrete consensus-based optimization algorithm with random batch interactions and heterogeneous noises” In Math. Models Methods Appl. Sci 32.06, 2022, pp. 1071–1107
  42. Michel Ledoux “Isoperimetry and Gaussian analysis” In Lectures on probability theory and statistics (Saint-Flour, 1994) 1648, Lecture Notes in Math. Springer, Berlin, 1996, pp. 165–294 DOI: 10.1007/BFb0095676
  43. Michel Ledoux “The Concentration of Measure Phenomenon” 89, Mathematical Surveys and Monographs American Mathematical Society, Providence, RI, 2001 DOI: 10.1090/surv/089
  44. X. Mao “Stochastic Differential Equations and Applications” Horwood Publishing Limited, Chichester, 2008
  45. Bernt Øksendal “Stochastic Differential Equations” An introduction with applications, Universitext Springer-Verlag, Berlin, 2003 DOI: 10.1007/978-3-642-14394-6
  46. “A consensus-based model for global optimization and its mean-field limit” In Math. Models Methods Appl. Sci. 27.1, 2017, pp. 183–204 DOI: 10.1142/S0218202517400061
  47. Alain-Sol Sznitman “Topics in propagation of chaos” In École d’Été de Probabilités de Saint-Flour XIX—1989 1464, Lecture Notes in Math. Springer, Berlin, 1991, pp. 165–251 DOI: 10.1007/BFb0085169
  48. Claudia Totzeck “Trends in consensus-based optimization” In Active particles. Vol. 3. Advances in theory, models, and applications, Model. Simul. Sci. Eng. Technol. Birkhäuser/Springer, Cham, 2022, pp. 201–226 DOI: 10.1007/978-3-030-93302-9\_6
  49. “Consensus-based global optimization with personal best” In Math. Biosci. Eng. 17.5, 2020, pp. 6026–6044 DOI: 10.3934/mbe.2020320
  50. “An inequality for trace ideals” In Comm. Math. Phys. 76.2, 1980, pp. 143–148 URL: http://projecteuclid.org/euclid.cmp/1103908255
  51. Cédric Villani “Optimal Transport” 338, Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences] Springer-Verlag, Berlin, 2009 DOI: 10.1007/978-3-540-71050-9
  52. Cédric Villani “Topics in Optimal Transportation” 58, Graduate Studies in Mathematics American Mathematical Society, Providence, RI, 2003 DOI: 10.1090/gsm/058
Citations (12)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 4 posts and received 4 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View