Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning (2404.10976v3)

Published 17 Apr 2024 in cs.LG, cs.AI, and cs.MA

Abstract: Cooperative Multi-Agent Reinforcement Learning (MARL) necessitates seamless collaboration among agents, often represented by an underlying relation graph. Existing methods for learning this graph primarily focus on agent-pair relations, neglecting higher-order relationships. While several approaches attempt to extend cooperation modelling to encompass behaviour similarities within groups, they commonly fall short in concurrently learning the latent graph, thereby constraining the information exchange among partially observed agents. To overcome these limitations, we present a novel approach to infer the Group-Aware Coordination Graph (GACG), which is designed to capture both the cooperation between agent pairs based on current observations and group-level dependencies from behaviour patterns observed across trajectories. This graph is further used in graph convolution for information exchange between agents during decision-making. To further ensure behavioural consistency among agents within the same group, we introduce a group distance loss, which promotes group cohesion and encourages specialization between groups. Our evaluations, conducted on StarCraft II micromanagement tasks, demonstrate GACG's superior performance. An ablation study further provides experimental evidence of the effectiveness of each component of our method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Deep coordination graphs. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 980–991. PMLR, 2020.
  2. Multi-agent reinforcement learning-based resource allocation for UAV networks. IEEE Trans. Wirel. Commun., 19(2):729–743, 2020.
  3. Learning from the dark: Boosting graph convolutional neural networks with diverse negative samples. In Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI 2022), Virtual Event, pages 6550–6558. AAAI Press, 2022.
  4. Layer-diverse negative sampling for graph neural networks. Transactions on Machine Learning Research, 2024.
  5. Inferring latent temporal sparse coordination graph for multi-agent reinforcement learning. CoRR, abs/2403.19253, 2024.
  6. Randomized entity-wise factorization for multi-agent reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning (ICML 2021), 18-24 July, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4596–4606. PMLR, 2021.
  7. Graph convolutional reinforcement learning. In 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia, 2020.
  8. Deep implicit coordination graphs for multi-agent reinforcement learning. In AAMAS ’21: 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2021), Virtual Event, United Kingdom, pages 764–772. ACM, 2021.
  9. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL 2019), Osaka, Japan, pages 590–602, 2020.
  10. Multi-agent game abstraction via graph attention neural network. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, NY, USA,, pages 7211–7218. AAAI Press, 2020.
  11. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
  12. Multi-agent deep reinforcement learning for multi-robot applications: A survey. Sensors, 23(7):3625, 2023.
  13. Learning to score behaviors for guided policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, (ICML 2020), volume 119 of Proceedings of Machine Learning Research, pages 7445–7454, 13–18 Jul 2020.
  14. VAST: value function factorization with variable agent sub-teams. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NIPS 2021), December 6-14, virtual, pages 24018–24032, 2021.
  15. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), Stockholmsmässan, Stockholm, Sweden, volume 80, pages 4292–4301, 2018.
  16. Cooperative heterogeneous multi-robot systems: A survey. ACM Comput. Surv., 52(2):29:1–29:31, 2019.
  17. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2019), Montreal, QC, Canada,, pages 2186–2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
  18. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.
  19. Self-organized group for cooperative multi-agent reinforcement learning. In NeurIPS, 2022.
  20. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2018), Stockholm, Sweden, pages 2085–2087, 2018.
  21. Relational forward models for multi-agent learning. In 7th International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA, 2019.
  22. ROMA: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9876–9886, 2020.
  23. Learning nearly decomposable value functions via communication minimization. In 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia, 2020.
  24. Traffic signal control with reinforcement learning based on region-aware cooperative strategy. IEEE Trans. Intell. Transp. Syst., 23(7):6774–6785, 2022.
  25. Context-aware sparse deep coordination graphs. In The Tenth International Conference on Learning Representations (ICLR 2022), Virtual Event. OpenReview.net, 2022.
  26. A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst., 32(1):4–24, 2021.
  27. Self-organized polynomial-time coordination graphs. In International Conference on Machine Learning (ICML 2022), Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24963–24979. PMLR, 2022.
  28. Automatic grouping for efficient cooperative multi-agent reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, (NIPS 2023), 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Wei Duan (18 papers)
  2. Jie Lu (127 papers)
  3. Junyu Xuan (21 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets