Learning to Communicate with Deep Multi-Agent Reinforcement Learning (1605.06676v2)

Abstract: We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

The paper entitled "Learning to Communicate with Deep Multi-Agent Reinforcement Learning" authored by Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson, thoroughly investigates the possibility of autonomous learning of communication protocols among multiple agents using deep reinforcement learning (RL). This paradigm is essential for addressing coordination tasks in partially observable environments, where effective communication is a prerequisite for achieving optimal performance.

The paper introduces two principal methodologies: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). Distinctively, RIAL leverages deep Q-learning (DQN) integrated with recurrent neural networks, catering to the partial observability via individual state maintenance and separate communication actions. Meanwhile, DIAL extends the concept of differentiable communication by allowing backpropagation through the communication channel during centralized learning, facilitating end-to-end training across agents.

Methodologies

Reinforced Inter-Agent Learning (RIAL)

RIAL employs deep Q-learning where agents learn Q-values corresponding to environment and communication actions via recurrent networks tailored for partial observability. Each agent independently approximates $Q(o^a_t, m^{a'}_{t-1}, h^a_{t-1}, u^a_t)$, adapting to observations $o$ and previous messages $m$. Recognizing non-stationarity in concurrent agent learning, experience replay is disabled. Crucially, parameter sharing among agents speeds up learning by collapsing the multi-agent problem into an agent-independent network, maintaining distinct hidden states and behaviors based on individual observations and agent indices.

Differentiable Inter-Agent Learning (DIAL)

DIAL capitalizes on the backpropagation of gradients across agents via a continuous communication channel during centralized learning. This methodology enhances the training signal by passing gradients from recipient agents to senders, facilitating more rapid and precise protocol learning. During decentralized execution, continuous messages are discretized, maintaining fidelity to the constraints of limited bandwidth communication. This approach is particularly potent due to its ability to directly optimize the message content to minimize the overall Q-network loss, even when rewards materialize several timesteps later.

Experimental Validation

The efficacy of RIAL and DIAL is thoroughly validated using two experimental domains: the Switch Riddle and multi-agent MNIST-based tasks. These scenarios present progressively challenging environments necessitating efficient inter-agent communication for optimal task completion.

Switch Riddle

In this classic multi-agent coordination problem, agents need to develop a shared protocol to ensure that all have visited a central interrogation room. Using three and four agents configurations, experiments show optimal policy learning by RIAL and DIAL, with DIAL demonstrating faster convergence and efficacy, especially when parameter sharing is implemented.

MNIST-based Tasks

Two tasks, Colour-Digit MNIST and Multi-Step MNIST, challenge agents with high-dimensional observation spaces and complex inter-agent dependency in their rewards. Here, DIAL significantly outperforms RIAL and other baselines. Particularly notable is DIAL's capacity to integrate information across timesteps, learning a binary encoding schema for digit identification, underscoring the importance of differentiable communication in sophisticated coordination tasks.

Implications and Future Directions

This research underscores several pivotal contributions to the field of multi-agent RL:

• Engineering Innovations: The structured methodologies of RIAL and DIAL, along with parameter sharing and DRU, streamline the learning of protocols, marking tangible advancements in deep learning architectures for multi-agent systems.
• Scalability and Complexity: The empirical results denote DIAL's paramount efficiency, providing empirical evidence that differentiable communication can solve high-dimension communication protocol learning problems more effectively than traditional RL approaches.
• Noise Regularization: Analysis on the impact of channel noise reveals insights into why language evolved to use discrete structures, showcasing how regularization through noise facilitates robust communication protocol learning.

Future investigations can extend these methodologies to more varied and complex scenarios, including those involving competitive settings, hierarchical communication, and richer observational spaces. Additionally, scaling the number of agents and refining architectures to better manage non-stationarity and partial observability will push the boundaries of autonomous communication learning.

In conclusion, this paper builds significant groundwork for the autonomous learning of communication protocols in multi-agent reinforcement learning, expediting advances in both theoretical and practical applications. The methodologies and insights derived here hold considerable potential for various real-world applications, from cooperative robotics to intelligent distributed systems.