Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games (2405.17017v3)

Published 27 May 2024 in math.OC, cs.LG, and cs.MA

Abstract: Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agents. Our analysis uses a Q-table for finite state and action spaces updated at each discrete time-step over an infinite horizon. In [Angiuli et al., 2023], we proved convergence of two-timescale algorithms for MFG and MFC separately highlighting the need to follow multiple population distributions in the MFC case. Here, we integrate this feature for MFCG as well as three rates of update decreasing to zero in the proper ratios. Our technique of proof uses a generalization to three timescales of the two-timescale analysis in [Borkar, 1997]. We give a simple example satisfying the various hypothesis made in the proof of convergence and illustrating the performance of the algorithm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (29)
  1. Reinforcement learning algorithm for mixed mean field control games. arXiv:2205.02330.
  2. Unified reinforcement q-learning for mean field game and control problems. Mathematics of Control, Signals, and Systems, 34:217–271.
  3. Convergence of multiscale reinforcement q-learning algorithms for mean field game and control problems. arXiv:2312.06659.
  4. Mean field games and mean field type control theory. Springer Briefs in Mathematics. Springer, New York.
  5. An analog scheme for fixed point computation. i. theory. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 44(4):351–355.
  6. Borkar, V. S. (1997). Stochastic approximation with two time scales. Systems & Control Letters, 29(5):291–294.
  7. Borkar, V. S. (1998). Asynchronous stochastic approximations. SIAM Journal on Control and Optimization, 36(3):840–851.
  8. The o.d.e. method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38(2):447–469.
  9. Probabilistic Theory of Mean Field Games with Applications I-II. Springer.
  10. Linear-quadratic mean-field reinforcement learning: Convergence of policy gradient methods. Preprint.
  11. Model-free mean-field reinforcement learning: mean-field mdp and mean-field q-learning. The Annals of Applied Probability, 33(6B):5334–5381.
  12. Approximately solving mean field games via entropy-regularized deep reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 1909–1917. PMLR.
  13. On the convergence of model free learning in mean field games. In in proc. of AAAI.
  14. Actor-critic learning for mean-field control in continuous time. arXiv preprint arXiv:2303.06993.
  15. Mean-field controls with q-learning for cooperative marl: convergence and complexity analysis. SIAM Journal on Mathematics of Data Science, 3(4):1168–1196.
  16. Learning mean-field games. In Advances in Neural Information Processing Systems, pages 4966–4976.
  17. Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. Commun. Inf. Syst., 6(3):221–251.
  18. Actor-critic–type learning algorithms for markov decision processes. SIAM Journal on Control and Optimization, 38(1):94–123.
  19. Mean field games. Jpn. J. Math., 2(1):229–260.
  20. Learning mean field games: A survey. arXiv preprint arXiv:2205.12944.
  21. Mean-field markov decision processes with common noise and open-loop controls. arXiv preprint arXiv:1912.07883.
  22. Neveu, J. (1975). Discrete-parameter Martingales. North-Holland mathematical library. North-Holland.
  23. Sell, G. R. (1973). Differential equations without uniqueness and classical topological dynamics. Journal of Differential Equations, 14(1):42–56.
  24. Reinforcement learning in stationary mean-field games. In Proceedings. 18th International Conference on Autonomous Agents and Multiagent Systems.
  25. Reinforcement learning: An introduction. MIT press.
  26. Tembine, H. (2017). Mean-field-type games. AIMS Math, 2(4):706–735.
  27. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1–34.
  28. Continuous-time mean–variance portfolio selection: A reinforcement learning framework. Mathematical Finance, 30(4):1273–1308.
  29. Watkins, C. J. C. H. (1989). Learning from delayed rewards. PhD thesis, King’s College, Cambridge.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets