Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient (2312.15667v3)

Published 25 Dec 2023 in cs.MA and cs.AI

Abstract: Multi-Agent Policy Gradient (MAPG) has made significant progress in recent years. However, centralized critics in state-of-the-art MAPG methods still face the centralized-decentralized mismatch (CDM) issue, which means sub-optimal actions by some agents will affect other agent's policy learning. While using individual critics for policy updates can avoid this issue, they severely limit cooperation among agents. To address this issue, we propose an agent topology framework, which decides whether other agents should be considered in policy gradient and achieves compromise between facilitating cooperation and alleviating the CDM issue. The agent topology allows agents to use coalition utility as learning objective instead of global utility by centralized critics or local utility by individual critics. To constitute the agent topology, various models are studied. We propose Topology-based multi-Agent Policy gradiEnt (TAPE) for both stochastic and deterministic MAPG methods. We prove the policy improvement theorem for stochastic TAPE and give a theoretical explanation for the improved cooperation among agents. Experiment results on several benchmarks show the agent topology is able to facilitate agent cooperation and alleviate CDM issue respectively to improve performance of TAPE. Finally, multiple ablation studies and a heuristic graph search algorithm are devised to show the efficacy of the agent topology.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Leveraging Communication Topologies Between Learning Agents in Deep Reinforcement Learning. arXiv preprint arXiv:1902.06740.
  2. Statistical mechanics of complex networks. Reviews of modern physics, 74(1): 47.
  3. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410.
  4. Emergence of scaling in random networks. science, 286(5439): 509–512.
  5. Deep coordination graphs. In International Conference on Machine Learning, 980–991. PMLR.
  6. Learning Credit Assignment for Cooperative Reinforcement Learning. arXiv preprint arXiv:2210.05367.
  7. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, 1538–1546. PMLR.
  8. Off-policy actor-critic. arXiv preprint arXiv:1205.4839.
  9. Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33: 22069–22079.
  10. LIIR: learning individual intrinsic reward in multi-agent reinforcement learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS), 4403–4414.
  11. A Review of Cooperation in Multi-agent Learning. arXiv preprint arXiv:2312.05162.
  12. Learning Correlated Communication Topology in Multi-Agent Reinforcement learning. In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 456–464.
  13. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1): 17–60.
  14. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101.
  15. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29.
  16. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  17. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.
  18. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  19. Iterated reasoning with mutual information in cooperative and byzantine decentralized teaming. arXiv preprint arXiv:2201.08484.
  20. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190: 82–94.
  21. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 3991–4002.
  22. Deep implicit coordination graphs for multi-agent reinforcement learning. arXiv preprint arXiv:2006.11438.
  23. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
  24. Reinforcement learning applications in unmanned vehicle control: A comprehensive overview. Unmanned Systems, 1–10.
  25. PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination. arXiv preprint arXiv:2301.06387.
  26. Leveraging Joint-action Embedding in Multi-agent Reinforcement Learning for Cooperative Games. IEEE Transactions on Games.
  27. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30.
  28. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, 1928–1937. PMLR.
  29. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
  30. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29.
  31. Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Systems with Applications, 116830.
  32. A concise introduction to decentralized POMDPs. Springer.
  33. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32: 289–353.
  34. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS).
  35. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 12208–12221.
  36. Precup, D. 2000. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, 80.
  37. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33: 10199–10210.
  38. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1): 7234–7284.
  39. GCS: graph-based coordination strategy for multi-agent reinforcement learning. arXiv preprint arXiv:2201.06257.
  40. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043.
  41. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440.
  42. Deterministic policy gradient algorithms. In International conference on machine learning, 387–395. Pmlr.
  43. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502–14515.
  44. Reinforcement learning: An introduction. MIT press.
  45. The information bottleneck method. arXiv preprint physics/0004057.
  46. An experimental study of the small world problem. In Social networks, 179–197. Elsevier.
  47. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062.
  48. Cross-Region Courier Displacement for On-Demand Delivery With Multi-Agent Reinforcement Learning. IEEE Transactions on Big Data.
  49. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523.
  50. Learning nearly decomposable value functions via communication minimization. arXiv preprint arXiv:1910.05366.
  51. Off-policy multi-agent decomposed policy gradients. arXiv preprint arXiv:2007.12322.
  52. Collective dynamics of ‘small-world’networks. nature, 393(6684): 440–442.
  53. Optimal payoff functions for members of collectives. Advances in Complex Systems, 4(02n03): 265–279.
  54. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning, 5872–5881. PMLR.
  55. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, 12491–12500. PMLR.
  56. PAC: Assisted Value Factorisation with Counterfactual Predictions in Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2206.11420.
  57. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 33: 11853–11864.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Xingzhou Lou (7 papers)
  2. Junge Zhang (47 papers)
  3. Timothy J. Norman (23 papers)
  4. Kaiqi Huang (60 papers)
  5. Yali Du (63 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now