Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning (2310.08746v1)

Published 12 Oct 2023 in cs.LG

Abstract: Multi-agent reinforcement learning (MARL) plays a pivotal role in tackling real-world challenges. However, the seamless transition of trained policies from simulations to real-world requires it to be robust to various environmental uncertainties. Existing works focus on finding Nash Equilibrium or the optimal policy under uncertainty in one environment variable (i.e. action, state or reward). This is because a multi-agent system itself is highly complex and unstationary. However, in real-world situation uncertainty can occur in multiple environment variables simultaneously. This work is the first to formulate the generalised problem of robustness to multi-modal environment uncertainty in MARL. To this end, we propose a general robust training approach for multi-modal uncertainty based on curriculum learning techniques. We handle two distinct environmental uncertainty simultaneously and present extensive results across both cooperative and competitive MARL environments, demonstrating that our approach achieves state-of-the-art levels of robustness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Dc-mrta: Decentralized multi-robot task allocation and navigation in complex environments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  11711–11718. IEEE, 2022.
  2. Rtaw: An attention inspired reinforcement learning method for multi-robot task allocation in warehouse environments. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.  1393–1399. IEEE, 2023.
  3. A curriculum learning method for improved noise robustness in automatic speech recognition. In 2017 25th European Signal Processing Conference (EUSIPCO), pp.  548–552. IEEE, 2017.
  4. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008.
  5. What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705, 2022.
  6. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
  7. Discounted robust stochastic games and an application to queueing control. Operations research, 59(2):365–382, 2011a.
  8. Discounted robust stochastic games and an application to queueing control. Operations research, 59(2):365–382, 2011b.
  9. Multi-agent reinforcement learning for traffic signal control: A cooperative approach. Sustainability, 15(4):3479, 2023.
  10. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.  4213–4220, 2019a.
  11. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence, 2019b.
  12. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. Advances in Neural Information Processing Systems, 35:22547–22561, 2022.
  13. Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp.  157–163. Elsevier, 1994.
  14. A review of uncertainty for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 18, pp. 155–162, 2022.
  15. Siddharth Mysore. Reward-guided curriculum for robust reinforcement learning. 2019.
  16. Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research, 21(1):7382–7431, 2020.
  17. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.
  18. Robust multi-agent q-learning in cooperative games with adversaries. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  19. Game theory and multi-agent reinforcement learning. Reinforcement Learning: State-of-the-Art, pp.  441–470, 2012.
  20. Learning and testing resilience in cooperative multi-agent systems. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp.  1055–1063, 2020.
  21. Resilient multi-agent reinforcement learning with adversarial value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  11308–11316, 2021.
  22. Sat: Improving adversarial training via curriculum-based loss smoothing. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security, pp.  25–36, 2021.
  23. Romax: Certifiably robust deep multiagent reinforcement learning via convex relaxation. In 2022 International Conference on Robotics and Automation (ICRA), pp.  5503–5510. IEEE, 2022.
  24. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.
  25. Improved multi-agent reinforcement learning for path planning-based crowd simulation. IEEE Access, 7:73841–73855, 2019.
  26. A survey on curriculum learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, 44(09):4555–4576, sep 2022. ISSN 1939-3539.
  27. Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013.
  28. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In International Conference on Machine Learning, 2022.
  29. Robust policy learning over multiple uncertainty sets. In International Conference on Machine Learning, pp. 24414–24429. PMLR, 2022.
  30. Robust multi-agent reinforcement learning with model uncertainty. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020.
  31. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321–384, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Aakriti Agrawal (13 papers)
  2. Rohith Aralikatti (8 papers)
  3. Yanchao Sun (32 papers)
  4. Furong Huang (150 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now