Learning Multiagent Communication with Backpropagation (1605.07736v2)

Abstract: Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNet, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand.

• The paper presents the CommNet model that learns continuous communication protocols via backpropagation to enhance multiagent coordination.
• The methodology uses a broadcast mechanism where agents share continuous signals, enabling dynamic adaptation in tasks like traffic navigation and combat.
• Empirical results show near-perfect lever task success and significant reductions in collision rates, demonstrating the model’s practical efficacy.

Learning Multiagent Communication with Backpropagation

The paper "Learning Multiagent Communication with Backpropagation," authored by Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus, introduces a novel approach to multiagent reinforcement learning (MARL) that autonomously learns communication protocols between agents to enhance cooperative performance. The proposed model, CommNet, features a simple yet effective architecture that utilizes continuous communication channels between agents, which are optimized using backpropagation alongside typical reinforcement learning (RL) algorithms or supervised learning methods. This paper explores diverse tasks to establish the model's effectiveness and provides insightful analyses that shed light on the learned communication strategies.

Model Architecture

CommNet is built upon a foundation of multiple agents controlled by independent deep feed-forward networks with shared communication channels. The communication is formulated as continuous vectors which are learned during the training phase, not predefined. Each agent's network receives the combined inputs from both its state and the aggregated signals from other agents. This structure allows the backpropagation algorithm to train the entire system end-to-end, facilitating the learning of communication strategies that are seamlessly integrated with the task-specific policy.

The agents' communication follows a broadcast mechanism where each agent transmits a continuous vector that is averaged and then shared among all agents. This setup, detailed in the paper, enables the model to dynamically vary the number and type of agents during runtime, making it adaptable to real-world applications with fluctuating agent populations, such as autonomous vehicles or distributed sensor networks.

Experiments and Results

The efficacy of CommNet is demonstrated through a variety of tasks. Key experiments involve a lever-pulling task, traffic junction simulations, a combat scenario, and the bAbI dataset for question answering.

1. Lever Pulling Task: This introductory demonstration involves agents learning to pull distinct levers simultaneously. The results show that agents using CommNet achieve a nearly perfect success rate (99%) compared to the independent controller baseline, highlighting the effective coordination through learned communication.
2. Traffic Junction: In this task, agents are car controllers navigating through a junction while minimizing collisions. CommNet-equipped agents significantly reduce failure rates compared to baseline models (e.g., 1.6% vs. 9.4% for LSTM-based controllers). Further analysis reveals that communication is especially beneficial as visibility decreases, with performance maintained even when agents have no direct visibility of other cars.
3. Combat Task: Here, agents in a team combat setting must strategize to defeat an opponent team. CommNet agents exhibit a higher win rate (up to 49.5%) than those using independent controllers or fully-connected models. The model's flexibility is validated by adjusting the number of agents and their visibility range, consistently showing communication's positive impact.
4. bAbI Tasks: The application of CommNet to the bAbI QA dataset outperforms traditional LSTM baselines, though it lags behind the specialized MemN2N model. Nevertheless, the results underscore CommNet's potential for tasks requiring information sharing and reasoning across multiple agents.

Analysis of Communication

The paper explores an interpretative analysis of the learned communication strategies, particularly within the traffic junction task. Detailed principal component analysis visualizations show that agents often communicate only when necessary, avoiding redundant signaling. Such sparse communication clusters convey critical information relevant to collision avoidance, demonstrating that CommNet not only learns to act but also to communicate efficiently.

Implications and Future Directions

Practically, the research underscores the feasibility of scalable MARL systems capable of dynamic agent communication. The ability to learn communication protocols endogenously means that systems like autonomous vehicle fleets, multi-robot systems, and distributed sensor networks can operate more robustly without predefined communication schemas.

Theoretically, this work offers a significant contribution to the MARL literature by proving that backpropagation is a viable method for training continuous communication channels in multiagent settings. While the simple broadcast mechanism is effective, future work could explore more sophisticated communication structures to handle larger and more complex environments. Additionally, extending the model to heterogeneous agents could further consolidate its applicability in varied real-world scenarios.

In conclusion, "Learning Multiagent Communication with Backpropagation" is a comprehensive paper showcasing the potential of CommNet for autonomous multiagent systems. The paper provides robust empirical evidence and thoughtful analysis, paving the way for future advancements in multiagent communication and coordination.