Papers
Topics
Authors
Recent
Search
2000 character limit reached

MFC-EQ: Mean-Field Control with Envelope Q-Learning for Moving Decentralized Agents in Formation

Published 15 Oct 2024 in cs.RO, cs.AI, cs.LG, and cs.MA | (2410.12062v1)

Abstract: We study a decentralized version of Moving Agents in Formation (MAiF), a variant of Multi-Agent Path Finding aiming to plan collision-free paths for multiple agents with the dual objectives of reaching their goals quickly while maintaining a desired formation. The agents must balance these objectives under conditions of partial observation and limited communication. The formation maintenance depends on the joint state of all agents, whose dimensionality increases exponentially with the number of agents, rendering the learning process intractable. Additionally, learning a single policy that can accommodate different linear preferences for these two objectives presents a significant challenge. In this paper, we propose Mean-Field Control with Envelop $Q$-learning (MFC-EQ), a scalable and adaptable learning framework for this bi-objective multi-agent problem. We approximate the dynamics of all agents using mean-field theory while learning a universal preference-agnostic policy through envelop $Q$-learning. Our empirical evaluation of MFC-EQ across numerous instances shows that it outperforms state-of-the-art centralized MAiF baselines. Furthermore, MFC-EQ effectively handles more complex scenarios where the desired formation changes dynamically -- a challenge that existing MAiF planners cannot address.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. R. Stern, N. Sturtevant, A. Felner, S. Koenig, H. Ma, T. Walker, J. Li, D. Atzmon, L. Cohen, T. Kumar, et al., “Multi-agent pathfinding: Definitions, variants, and benchmarks,” in SoCS, 2019, pp. 151–158.
  2. H. Ma and S. Koenig, “Ai buzzwords explained: multi-agent path finding (mapf),” AI Matters, vol. 3, no. 3, pp. 15–19, 2017.
  3. P. R. Wurman, R. D’Andrea, and M. Mountz, “Coordinating hundreds of cooperative, autonomous vehicles in warehouses,” AI magazine, vol. 29, no. 1, pp. 9–9, 2008.
  4. R. Morris, C. S. Pasareanu, K. S. Luckow, W. Malik, H. Ma, T. S. Kumar, and S. Koenig, “Planning, scheduling and monitoring for airport surface operations.” in AAAI Workshop: Planning for Hybrid Systems, 2016, pp. 608–614.
  5. H. Ma, J. Yang, L. Cohen, T. Kumar, and S. Koenig, “Feasibility study: Moving non-homogeneous teams in congested video game environments,” in AIIDE, 2017, pp. 270–272.
  6. A. Gautam and S. Mohan, “A review of research in multi-robot systems,” in ICIIS, 2012, pp. 1–5.
  7. J. Li, K. Sun, H. Ma, A. Felner, T. Kumar, and S. Koenig, “Moving agents in formation in congested environments,” in SoCS, 2020, pp. 131–132.
  8. G. Sartoretti, J. Kerr, Y. Shi, G. Wagner, T. S. Kumar, S. Koenig, and H. Choset, “Primal: Pathfinding via reinforcement and imitation multi-agent learning,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2378–2385, 2019.
  9. Z. Liu, B. Chen, H. Zhou, G. Koushik, M. Hebert, and D. Zhao, “Mapper: Multi-agent path planning with evolutionary reinforcement learning in mixed dynamic environments,” in IROS, 2020, pp. 11 748–11 754.
  10. Z. Ma, Y. Luo, and H. Ma, “Distributed heuristic multi-agent path finding with communication,” in ICRA, 2021, pp. 8699–8705.
  11. M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in Machine learning proceedings.   Elsevier, 1994, pp. 157–163.
  12. Q. Lin and H. Ma, “Sacha: Soft actor-critic with heuristic-based attention for partially observable multi-agent path finding,” IEEE Robotics and Automation Letters, vol. 8, no. 8, pp. 2377–3766, 2023.
  13. S. Liu, L. Wen, J. Cui, X. Yang, J. Cao, and Y. Liu, “Moving forward in formation: a decentralized hierarchical learning approach to multi-agent moving together,” in IROS, 2021, pp. 4777–4784.
  14. Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and J. Wang, “Mean field multi-agent reinforcement learning,” in ICML, 2018, pp. 5571–5580.
  15. S. G. Subramanian, M. E. Taylor, M. Crowley, and P. Poupart, “Partially observable mean field reinforcement learning,” in AAMAS.   IFAAMAS, 2021.
  16. Z. Gábor, Z. Kalmár, and C. Szepesvári, “Multi-criteria reinforcement learning.” in ICML, 1998, pp. 197–205.
  17. S. Mannor and N. Shimkin, “The steering approach for multi-criteria reinforcement learning,” NeurIPS, 2001.
  18. S. Natarajan and P. Tadepalli, “Dynamic preferences in multi-criteria reinforcement learning,” in ICML, 2005, pp. 601–608.
  19. S. Parisi, M. Pirotta, N. Smacchia, L. Bascetta, and M. Restelli, “Policy gradient approaches for multi-objective sequential decision making,” in IJCNN, 2014, pp. 2323–2330.
  20. K. Van Moffaert and A. Nowé, “Multi-objective reinforcement learning using sets of pareto dominating policies,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 3483–3512, 2014.
  21. X. Chen, A. Ghadirzadeh, M. Björkman, and P. Jensfelt, “Meta-learning for multi-objective reinforcement learning,” in IROS, 2019, pp. 977–983.
  22. A. Castelletti, F. Pianosi, and M. Restelli, “Multi-objective fitted q-iteration: Pareto frontier approximation in one single run,” in ICNSC, 2011, pp. 260–265.
  23. A. Abels, D. Roijers, T. Lenaerts, A. Nowé, and D. Steckelmacher, “Dynamic weights in multi-objective deep reinforcement learning,” in ICML, 2019, pp. 11–20.
  24. R. Yang, X. Sun, and K. Narasimhan, “A generalized algorithm for multi-objective reinforcement learning and policy adaptation,” NeurIPS, 2019.
  25. C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992.
  26. H. Hasselt, “Double q-learning,” NeurIPS, 2010.
  27. L. T. Watson and R. T. Haftka, “Modern homotopy methods in optimization,” Computer Methods in Applied Mechanics and Engineering, vol. 74, no. 3, pp. 289–305, 1989.
  28. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in ICML, 2009, pp. 41–48.
  29. D. Silver, “Cooperative pathfinding,” in AIIDE, 2005, pp. 117–122.
  30. G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant, “Conflict-based search for optimal multi-agent pathfinding,” Artificial Intelligence, vol. 219, pp. 40–66, 2015.
  31. J. Pearl and J. H. Kim, “Studies in semi-admissible heuristics,” IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 4, pp. 392–399, 1982.
  32. Y. Haimes, “On a bicriterion formulation of the problems of integrated system identification and system optimization,” IEEE Transactions on Systems, Man, and Cybernetics, no. 3, pp. 296–297, 1971.
  33. T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” Journal of Machine Learning Research, vol. 21, no. 1, pp. 7234–7284, 2020.
  34. T. Hu, B. Luo, C. Yang, and T. Huang, “Mo-mix: Multi-objective multi-agent cooperative decision-making with deep reinforcement learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  35. K. Van Moffaert, M. M. Drugan, and A. Nowé, “Scalarized multi-objective reinforcement learning: Novel design techniques,” in ADPRL.   IEEE, 2013, pp. 191–199.
  36. S. Mannor and N. Shimkin, “A geometric approach to multi-criterion reinforcement learning,” JMLR, vol. 5, pp. 325–360, 2004.
  37. C. He, T. Yang, T. Duhan, Y. Wang, and G. Sartoretti, “Alpha: Attention-based long-horizon pathfinding in highly-structured areas,” in ICRA.   IEEE, 2024, pp. 14 576–14 582.
  38. Z. Ma, Y. Luo, and J. Pan, “Learning selective communication for multi-agent path finding,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 1455–1462, 2021.
  39. W. Li, H. Chen, B. Jin, W. Tan, H. Zha, and X. Wang, “Multi-agent path finding with prioritized communication learning,” in ICRA.   IEEE, 2022, pp. 10 695–10 701.
  40. Y. Wang, B. Xiang, S. Huang, and G. Sartoretti, “Scrimp: Scalable communication for reinforcement-and imitation-learning-based multi-agent pathfinding,” in IROS, 2023, pp. 9301–9308.
  41. B. Wang, Z. Liu, Q. Li, and A. Prorok, “Mobile robot path planning in dynamic environments through globally guided reinforcement learning,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 6932–6939, 2020.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up