Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (2405.15054v1)

Published 23 May 2024 in cs.MA, cs.AI, cs.LG, and cs.RO

Abstract: The study of behavioral diversity in Multi-Agent Reinforcement Learning (MARL) is a nascent yet promising field. In this context, the present work deals with the question of how to control the diversity of a multi-agent system. With no existing approaches to control diversity to a set value, current solutions focus on blindly promoting it via intrinsic rewards or additional loss functions, effectively changing the learning objective and lacking a principled measure for it. To address this, we introduce Diversity Control (DiCo), a method able to control diversity to an exact value of a given metric by representing policies as the sum of a parameter-shared component and dynamically scaled per-agent components. By applying constraints directly to the policy architecture, DiCo leaves the learning objective unchanged, enabling its applicability to any actor-critic MARL algorithm. We theoretically prove that DiCo achieves the desired diversity, and we provide several experiments, both in cooperative and competitive tasks, that show how DiCo can be employed as a novel paradigm to increase performance and sample efficiency in MARL. Multimedia results are available on the paper's website: https://sites.google.com/view/dico-marl.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Pid control system analysis, design, and technology. IEEE transactions on control systems technology, 13(4):559–576, 2005.
  2. Balch, T. Hierarchic social entropy: An information theoretic measure of robot group diversity. Autonomous robots, 8(3):209–238, 2000.
  3. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819–840, 2002.
  4. Vmas: A vectorized multi-agent simulator for collective robot learning. In Proceedings of the 16th International Symposium on Distributed Autonomous Robotic Systems, DARS ’22. Springer, 2022.
  5. Benchmarl: Benchmarking multi-agent reinforcement learning. arXiv preprint arXiv:2312.01472, 2023a.
  6. Heterogeneous multi-robot reinforcement learning. In Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’23. International Foundation for Autonomous Agents and Multiagent Systems, 2023b.
  7. System neural diversity: Measuring behavioral heterogeneity in multi-agent learning. arXiv preprint arXiv:2305.02128, 2023c.
  8. Torchrl: A data-driven decision-making library for pytorch. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=QxItoEAVMb.
  9. Beyond species: functional diversity and the maintenance of ecological processes and services. Journal of applied ecology, 48(5):1079–1087, 2011.
  10. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=UkU05GOH7_6.
  11. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
  12. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pp.  1989–1998. PMLR, 2021.
  13. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.
  14. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1–5, 2016.
  15. Revisiting some common practices in cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp.  6863–6877. PMLR, 2022.
  16. Policy diagnosis via measuring role diversity in cooperative multi-agent rl. In International Conference on Machine Learning, pp.  9041–9071. PMLR, 2022.
  17. Mix-me: Quality-diversity for multi-agent learning. arXiv e-prints, pp.  arXiv–2311, 2023.
  18. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859–865, 2019.
  19. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp.  3040–3049. PMLR, 2019.
  20. The emergence of individuality. In International Conference on Machine Learning, pp.  4992–5001. PMLR, 2021.
  21. Kellert, S. R. The value of life: Biological diversity and human society. Island press, 1997.
  22. Kortvelesy, R. Fixed integral neural networks. arXiv preprint arXiv:2307.14439, 2023.
  23. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34:941–952, 2021.
  24. A unified diversity measure for multiagent reinforcement learning. Advances in Neural Information Processing Systems, 35:10339–10352, 2022.
  25. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017.
  26. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
  27. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):1–16, 2022.
  28. Statistical metrics. Selecta Mathematica: Volume 2, pp.  433–435, 2003.
  29. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050–18062, 2020.
  30. Modelling behavioural diversity for learning in open-ended games. In International conference on machine learning, pp.  8514–8524. PMLR, 2021.
  31. Shapley, L. S. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095–1100, 1953.
  32. Vaserstein, L. N. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64–72, 1969.
  33. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  34. Influence-based multi-agent exploration. In International Conference on Learning Representations, 2019.
  35. Roma: Multi-agent reinforcement learning with emergent roles. In International Conference on Machine Learning, pp.  9876–9886. PMLR, 2020.
  36. Collective intelligence and group performance. Current Directions in Psychological Science, 24(6):420–424, 2015.
  37. Quality-similar diversity via population based reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=bLmSMXbqXr.
  38. Yadan, O. Hydra - a framework for elegantly configuring complex applications. Github, 2019. URL https://github.com/facebookresearch/hydra.
  39. Informative policy representations in multi-agent reinforcement learning via joint-action distributions. arXiv e-prints, pp.  arXiv–2106, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Matteo Bettini (11 papers)
  2. Ryan Kortvelesy (14 papers)
  3. Amanda Prorok (66 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets