Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Efficient Multi-agent Reinforcement Learning by Planning (2405.11778v1)

Published 20 May 2024 in cs.LG, cs.AI, and cs.MA

Abstract: Multi-agent reinforcement learning (MARL) algorithms have accomplished remarkable breakthroughs in solving large-scale decision-making tasks. Nonetheless, most existing MARL algorithms are model-free, limiting sample efficiency and hindering their applicability in more challenging scenarios. In contrast, model-based reinforcement learning (MBRL), particularly algorithms integrating planning, such as MuZero, has demonstrated superhuman performance with limited data in many tasks. Hence, we aim to boost the sample efficiency of MARL by adopting model-based approaches. However, incorporating planning and search methods into multi-agent systems poses significant challenges. The expansive action space of multi-agent systems often necessitates leveraging the nearly-independent property of agents to accelerate learning. To tackle this issue, we propose the MAZero algorithm, which combines a centralized model with Monte Carlo Tree Search (MCTS) for policy search. We design a novel network structure to facilitate distributed execution and parameter sharing. To enhance search efficiency in deterministic environments with sizable action spaces, we introduce two novel techniques: Optimistic Search Lambda (OS($\lambda$)) and Advantage-Weighted Policy Optimization (AWPO). Extensive experiments on the SMAC benchmark demonstrate that MAZero outperforms model-free approaches in terms of sample efficiency and provides comparable or better performance than existing model-based methods in terms of both sample and computational efficiency. Our code is available at https://github.com/liuqh16/MAZero.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.
  2. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
  3. Planning in stochastic environments with a learned model. In International Conference on Learning Representations, 2021.
  4. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pp.  314–315, 2019.
  5. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
  6. Cooperative prioritized sweeping. In AAMAS, pp.  160–168, 2021.
  7. Combining deep reinforcement learning and search for imperfect-information games. Advances in Neural Information Processing Systems, 33:17057–17069, 2020.
  8. Efficient model-based deep reinforcement learning with variational state tabulation. In International Conference on Machine Learning, pp. 1049–1058. PMLR, 2018.
  9. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2021.
  10. Scalable multi-agent model-based reinforcement learning. arXiv preprint arXiv:2205.15023, 2022.
  11. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, pp.  1–49, 2022.
  12. World models. arXiv preprint arXiv:1803.10122, 2018.
  13. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
  14. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
  15. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
  16. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955, 2022.
  17. What model does muzero learn? arXiv preprint arXiv:2306.00840, 2023.
  18. A very condensed survey and critique of multiagent deep reinforcement learning. In Proceedings of the 19th international conference on autonomous agents and multiagent systems, pp.  2146–2148, 2020.
  19. Muesli: Combining improvements in policy optimization. In International conference on machine learning, pp. 4214–4226. PMLR, 2021.
  20. Learning-based model predictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269–296, 2020.
  21. Learning and planning in complex action spaces. In International Conference on Machine Learning, pp. 4476–4486. PMLR, 2021.
  22. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
  23. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.
  24. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  25. Bandit based monte-carlo planning. In European conference on machine learning, pp.  282–293. Springer, 2006.
  26. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.  4501–4510, 2020.
  27. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018.
  28. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991–4002, 2021.
  29. Multi-agent game abstraction via graph attention neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  7211–7218, 2020.
  30. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In 2018 IEEE international conference on robotics and automation (ICRA), pp.  6252–6259. IEEE, 2018.
  31. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017.
  32. Tesseract: Tensorised actors for multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 7301–7312. PMLR, 2021.
  33. Speedyzero: Mastering atari with limited data and time. In The Eleventh International Conference on Learning Representations, 2022.
  34. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA), pp.  7559–7566. IEEE, 2018.
  35. Awac: Accelerating online reinforcement learning with offline datasets, 2021.
  36. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9):3826–3839, 2020.
  37. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
  38. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:12208–12221, 2021.
  39. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234–7284, 2020.
  40. Multi-agent actor-critic with hierarchical graph attention network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  7236–7243, 2020.
  41. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
  42. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
  43. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  44. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  45. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  46. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887–5896. PMLR, 2019.
  47. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference on learning theory, pp.  2898–2933. PMLR, 2019.
  48. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.
  49. Reinforcement learning: An introduction. MIT press, 2018.
  50. Gerald Tesauro. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215–219, 1994.
  51. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  52. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020a.
  53. Exploring model-based planning with policy networks. arXiv preprint arXiv:1906.08649, 2019.
  54. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057, 2019.
  55. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020b.
  56. Mambpo: Sample-efficient multi-robot reinforcement learning using learned world models. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  5635–5640. IEEE, 2021.
  57. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939, 2020.
  58. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  6672–6679, 2020.
  59. Towards global optimality in cooperative marl with the transformation and distillation framework, 2023.
  60. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476–25488, 2021.
  61. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624, 2022.
  62. Model-based multi-agent rl in zero-sum markov games with near-optimal sample complexity. Advances in Neural Information Processing Systems, 33:1166–1178, 2020.
  63. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Qihan Liu (6 papers)
  2. Jianing Ye (7 papers)
  3. Xiaoteng Ma (24 papers)
  4. Jun Yang (357 papers)
  5. Bin Liang (115 papers)
  6. Chongjie Zhang (68 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets