Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning (2312.15555v1)

Published 24 Dec 2023 in cs.MA

Abstract: Value function factorization has achieved great success in multi-agent reinforcement learning by optimizing joint action-value functions through the maximization of factorized per-agent utilities. To ensure Individual-Global-Maximum property, existing works often focus on value factorization using monotonic functions, which are known to result in restricted representation expressiveness. In this paper, we analyze the limitations of monotonic factorization and present ConcaveQ, a novel non-monotonic value function factorization approach that goes beyond monotonic mixing functions and employs neural network representations of concave mixing functions. Leveraging the concave property in factorization, an iterative action selection scheme is developed to obtain optimal joint actions during training. It is used to update agents' local policy networks, enabling fully decentralized execution. The effectiveness of the proposed ConcaveQ is validated across scenarios involving multi-agent predator-prey environment and StarCraft II micromanagement tasks. Empirical results exhibit significant improvement of ConcaveQ over state-of-the-art multi-agent reinforcement learning approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Deep coordination graphs. In PMLR.
  2. Convex Analysis and Nonlinear Optimization: Theory and Examples. Springer.
  3. Convex Optimization. Cambridge university press.
  4. Multi-agent covering option discovery based on kronecker product of factor graphs. IEEE Transactions on Artificial Intelligence.
  5. Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs. In Advances in Neural Information Processing Systems, volume 35, 30406–30418. Curran Associates, Inc.
  6. Minimizing return gaps with discrete communications in decentralized pomdp. arXiv preprint arXiv:2308.03358.
  7. Czumaj, A. 2004. Lecture notes in Approximation and Randomized Algorithms. https://www.ic.unicamp.br/~celio/peer2peer/math/czumaj-balls-into-bins.pdf.
  8. Counterfactual Multi-Agent Policy Gradients. In AAAI.
  9. Accmer: Accelerating multi-agent experience replay with cache locality-aware prioritization. In 2023 IEEE 34th International Conference on Application-specific Systems, Architectures and Processors (ASAP), 205–212. IEEE.
  10. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. In ICLR.
  11. Actor-Attention-Critic for Multi-Agent Reinforcement Learning. In ICML.
  12. Karush, W. 1939. Minima of Functions of Several Variables with Inequalities as Side Conditions. Master’s thesis, Department of Mathematics, University of Chicago.
  13. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In NeuralIPS.
  14. ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning. arXiv preprint arXiv:2302.05593.
  15. MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’23, 466–475. International Foundation for Autonomous Agents and Multiagent Systems.
  16. Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In NeuralIPS.
  17. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In ICML.
  18. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043.
  19. ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization. In NeurIPS.
  20. QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning. In ICML.
  21. Value-decomposition multi-agent actor-critics. In AAAI.
  22. Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In AAMAS.
  23. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In ICLR.
  24. DOP: Off-Policy Multi-Agent Decomposed Policy Gradients. In ICLR.
  25. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03939.
  26. FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning. In ICML.
  27. PAC: Assisted Value Factorization with Counterfactual Predictions in Multi-Agent Reinforcement Learning. In NeurlIPS.
  28. Value Functions Factorization With Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(5): 1351–1361.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Huiqun Li (1 paper)
  2. Hanhan Zhou (14 papers)
  3. Yifei Zou (18 papers)
  4. Dongxiao Yu (60 papers)
  5. Tian Lan (162 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now