Value-Decomposition Networks For Cooperative Multi-Agent Learning (1706.05296v1)

Abstract: We study the problem of cooperative multi-agent reinforcement learning with a single joint reward signal. This class of learning problems is difficult because of the often large combined action and observation spaces. In the fully centralized and decentralized approaches, we find the problem of spurious rewards and a phenomenon we call the "lazy agent" problem, which arises due to partial observability. We address these problems by training individual agents with a novel value decomposition network architecture, which learns to decompose the team value function into agent-wise value functions. We perform an experimental evaluation across a range of partially-observable multi-agent domains and show that learning such value-decompositions leads to superior results, in particular when combined with weight sharing, role information and information channels.

Value-Decomposition Networks For Cooperative Multi-Agent Learning

Authors: Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, Thore Graepel

The paper investigates the problem of cooperative multi-agent reinforcement learning (MARL) where a system of agents must jointly optimize a single reward signal. The challenges in this domain stem from the large combined action and observation spaces, which lead to issues like spurious rewards and the "lazy agent" problem when using fully centralized or decentralized approaches. To address these, the authors propose a novel value-decomposition network architecture that decomposes the team value function into agent-wise value functions.

Problem and Motivation

Cooperative MARL problems are prevalent in real-world applications such as self-driving cars, traffic signal coordination, and optimizing factory productivity. Traditional centralized approaches, which treat the system as a single agent operating over a combined state and action space, often fail due to inefficient policies and the lazy agent problem, where one agent's actions inadvertently discourage another from learning effectively. Decentralized approaches, where each agent learns independently, suffer from non-stationarity and partial observability, leading to spurious reward signals that complicate the learning process.

Approach and Methods

The proposed solution involves a learned additive value-decomposition approach where the joint action-value function is decomposed into agent-specific value functions:

$$Q(h^1, h^2, \ldots, h^d, a^1, a^2, \ldots, a^d) \approx \sum_{i=1}^d \tilde{Q}_i(h^i, a^i)$$

Here, $\tilde{Q}_i$ depends only on individual agent’s local observations and actions. The decomposition is learned autonomously from the team reward signal by backpropagating the total $Q$ gradient through neural networks representing the individual value functions. This allows for a centralized training phase and decentralized deployment, as each agent's policy is derived from its local value function.

Experimentation and Evaluation

The authors perform extensive experimental evaluations across several partially observable multi-agent domains. They introduce environments such as Fetch, Switch, and Checkers, which require significant coordination among agents:

1. Switch: Agents navigate maps with narrow corridors, requiring one agent to yield to another to prevent collisions.
2. Fetch: Agents pick up and return items, necessitating synchronized actions.
3. Checkers: Agents navigate a grid with apples (rewarding) and lemons (penalizing) where one agent is more sensitive to these rewards than the other.

Nine different agent architectures were evaluated, including independent learners, fully centralized learners, and value-decomposition networks with various enhancements such as weight sharing, role information, and information channels.

Results

The value-decomposition architectures outperformed both fully centralized and independent learning approaches significantly. Key findings include:

• Increased Performance: Value-decomposition networks showed superior performance and quicker adaptation across all environments, as indicated by both normalized area under the curve (AUC) and final reward performance.
• Addressing Lazy Agent Problem: Weight sharing and role information were particularly effective in environments requiring specialized coordination roles, mitigating the lazy agent problem.
• Efficacy of Value Decomposition: The learned value functions effectively disambiguated contributions from individual agents, as demonstrated by the Fetch experiment, where the value functions for each agent correctly anticipated rewards contingent on their actions.

Practical and Theoretical Implications

The practical implications of this research are profound, suggesting that value-decomposition networks can significantly enhance the effectiveness of multi-agent systems in real-world scenarios requiring intricate coordination. Theoretically, this approach advances the understanding of how complex team tasks can be autonomously decomposed into simpler, more manageable subproblems, a crucial step towards scalable multi-agent learning.

Future Directions

Future research may focus on:

• Scaling: Investigating the scalability of value-decomposition with increasing numbers of agents and the associated combinatorial explosion of action spaces.
• Non-linear Aggregation: Exploring non-linear methods for value aggregation to capture more complex interdependencies between agents.
• Policy Gradient Methods: Extending the value-decomposition approach to policy gradient methods such as A3C to benefit from the hybrid advantages of both value-based and policy-based techniques.

In summary, the introduction of value-decomposition networks represents a significant advancement in cooperative MARL, addressing fundamental challenges and paving the way for more sophisticated and scalable multi-agent systems.