Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (1703.06182v4)

Abstract: Many real-world tasks involve multiple agents with partial observability and limited communication. Learning is challenging in these settings due to local viewpoints of agents, which perceive the world as non-stationary due to concurrently-exploring teammates. Approaches that learn specialized policies for individual tasks face problems when applied to the real world: not only do agents have to learn and store distinct policies for each task, but in practice identities of tasks are often non-observable, making these approaches inapplicable. This paper formalizes and addresses the problem of multi-task multi-agent reinforcement learning under partial observability. We introduce a decentralized single-task learning approach that is robust to concurrent interactions of teammates, and present an approach for distilling single-task policies into a unified policy that performs well across multiple related tasks, without explicit provision of task identity.

• The paper formalizes the MT-MARL problem using Dec-POMDPs to enable cooperative learning under partial observability.
• It introduces a two-phase decentralized approach with DRQNs and synchronized CERTs for stable multi-agent policy training.
• The study distills specialized policies into a generalized framework that achieves comparable performance in grid-based experiments.

Overview of Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability

The paper "Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability" investigates a novel approach to reinforcement learning (RL) in complex environments where multiple agents interact with each other under partial observability and limited communication. This work, led by researchers from MIT, Northeastern University, and Boeing Research & Technology, addresses the growing demand for multi-task learning in decentralized multi-agent settings, such as in autonomous vehicles and robotics.

Key Contributions

1. Formalization of MT-MARL: The paper formalizes the problem of multi-task multi-agent reinforcement learning (MT-MARL) under partial observability, leveraging decentralized partially observable Markov decision processes (Dec-POMDPs) as the foundation. This formulation provides a structured way to approach cooperative learning tasks where agents perceive the world from limited viewpoints and do not have explicit knowledge of the task identities.
2. Decentralized Learning Approach: The researchers introduce a two-phase decentralized learning framework. The first phase involves specialization, where agents use decentrally-learned single-task policies through deep recurrent Q-networks (DRQNs). This approach allows agents to coordinate and learn individual tasks efficiently despite environmental non-stationarity.
3. Concurrent Experience Replay Trajectories (CERTs): To stabilize learning, especially with non-stationary dynamics from partial observability, the paper proposes CERTs. This mechanism extends traditional experience replay by synchronizing experience trajectories across agents, thus addressing non-concurrent sampling issues and improving coordination.
4. Policy Distillation: The second phase converts these specialized policies into a single generalized policy capable of performing across all tasks without explicit task identity during execution. This distillation process uses supervised learning to regress from specialized policy Q-values across multiple tasks, ensuring robust multi-task performance.
5. Empirical Evaluation: The researchers validate their approach using multi-agent grid domains, demonstrating that Dec-HDRQNs, reliant on cautious optimism through hysteretic learning, outperform non-hysteretic models in stability and efficiency. Furthermore, the distilled policies achieved performance similar to specialized policies without needing task identification, reflecting effective generalization.

Implications and Future Work

The implications of this research are substantial for both theoretical understanding and practical applications of multi-agent systems. From a theoretical standpoint, the paper presents a structured methodology for addressing MT-MARL in partially observable and realistic settings, which were previously challenging due to non-stationarity and incomplete state information.

Practically, the insights derived from this research can be applied to many fields, including autonomous systems, robotic coordination, distributed sensor networks, and resource allocation tasks, highlighting the potential of decentralized decision-making models.

Moving forward, the proposed frameworks could be extended or integrated with advances in other areas of AI, such as transfer learning or meta-learning, to enhance adaptability across a broader range of contexts. Further exploration could also assess the scalability of this approach in high-dimensional spaces and with a larger number of agents, reflecting the complexities of real-world scenarios.

In conclusion, this paper makes significant contributions to the understanding and implementation of MT-MARL under challenging conditions, paving the way for more sophisticated autonomous systems capable of operability in diverse and dynamic environments.