Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning (2402.17978v2)

Published 28 Feb 2024 in cs.LG, cs.AI, and cs.MA

Abstract: Effective exploration is crucial to discovering optimal strategies for multi-agent reinforcement learning (MARL) in complex coordination tasks. Existing methods mainly utilize intrinsic rewards to enable committed exploration or use role-based learning for decomposing joint action spaces instead of directly conducting a collective search in the entire action-observation space. However, they often face challenges obtaining specific joint action sequences to reach successful states in long-horizon tasks. To address this limitation, we propose Imagine, Initialize, and Explore (IIE), a novel method that offers a promising solution for efficient multi-agent exploration in complex scenarios. IIE employs a transformer model to imagine how the agents reach a critical state that can influence each other's transition functions. Then, we initialize the environment at this state using a simulator before the exploration phase. We formulate the imagination as a sequence modeling problem, where the states, observations, prompts, actions, and rewards are predicted autoregressively. The prompt consists of timestep-to-go, return-to-go, influence value, and one-shot demonstration, specifying the desired state and trajectory as well as guiding the action generation. By initializing agents at the critical states, IIE significantly increases the likelihood of discovering potentially important under-explored regions. Despite its simplicity, empirical results demonstrate that our method outperforms multi-agent exploration baselines on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments. Particularly, IIE shows improved performance in the sparse-reward SMAC tasks and produces more effective curricula over the initialized states than other generative methods, such as CVAE-GAN and diffusion models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Is Conditional Generative Modeling all you need for Decision Making? In The Eleventh International Conference on Learning Representations.
  2. CVAE-GAN: fine-grained image generation through asymmetric training. In Proceedings of the IEEE international conference on computer vision, 2745–2754.
  3. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084–15097.
  4. Liir: Learning individual intrinsic reward in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 32.
  5. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995.
  6. First return, then explore. Nature, 590(7847): 580–586.
  7. SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2212.07489.
  8. Memory based trajectory-conditioned policies for learning from sparse rewards. Advances in Neural Information Processing Systems, 33: 4333–4345.
  9. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series.
  10. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, 3040–3049. PMLR.
  11. MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvari, C.; Niu, G.; and Sabato, S., eds., Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, 10041–10052. PMLR.
  12. Qmdp-net: Deep learning for planning under partial observability. Advances in neural information processing systems, 30.
  13. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190: 82–94.
  14. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 27921–27936.
  15. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35: 4328–4343.
  16. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32.
  17. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896.
  18. Sources of Hallucination by Large Language Models on Inference Tasks. arXiv preprint arXiv:2305.14552.
  19. A concise introduction to decentralized POMDPs. Springer.
  20. Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation. In International Conference on Learning Representations.
  21. Stabilizing transformers for reinforcement learning. In International conference on machine learning, 7487–7498. PMLR.
  22. Improving language understanding by generative pre-training. In OpenAI. OpenAI.
  23. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33: 10199–10210.
  24. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, 4295–4304. PMLR.
  25. A generalist agent. arXiv preprint arXiv:2205.06175.
  26. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043.
  27. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, 5887–5896. PMLR.
  28. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In International Conference on Learning Representations.
  29. Multi-agent reinforcement learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems, 34: 3271–3284.
  30. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523.
  31. Influence-Based Multi-Agent Exploration. In International Conference on Learning Representations.
  32. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 24611–24624.
  33. Episodic Multi-agent Reinforcement Learning with Curiosity-driven Exploration. Advances in Neural Information Processing Systems, 34.
  34. SMARTS: An Open-Source Scalable Multi-Agent RL Training School for Autonomous Driving. In Conference on Robot Learning, 264–285. PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Zeyang Liu (13 papers)
  2. Lipeng Wan (27 papers)
  3. Xinrui Yang (14 papers)
  4. Zhuoran Chen (3 papers)
  5. Xingyu Chen (98 papers)
  6. Xuguang Lan (34 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now