Distral: Robust Multitask Reinforcement Learning (1707.04175v1)

Abstract: Most deep reinforcement learning algorithms are data inefficient in complex and rich environments, limiting their applicability to many scenarios. One direction for improving data efficiency is multitask learning with shared neural network parameters, where efficiency may be improved through transfer across related tasks. In practice, however, this is not usually observed, because gradients from different tasks can interfere negatively, making learning unstable and sometimes even less data efficient. Another issue is the different reward schemes between tasks, which can easily lead to one task dominating the learning of a shared model. We propose a new approach for joint training of multiple tasks, which we refer to as Distral (Distill & transfer learning). Instead of sharing parameters between the different workers, we propose to share a "distilled" policy that captures common behaviour across tasks. Each worker is trained to solve its own task while constrained to stay close to the shared policy, while the shared policy is trained by distillation to be the centroid of all task policies. Both aspects of the learning process are derived by optimizing a joint objective function. We show that our approach supports efficient transfer on complex 3D environments, outperforming several related methods. Moreover, the proposed learning process is more robust and more stable---attributes that are critical in deep reinforcement learning.

• The paper presents a novel multitask RL method that jointly optimizes a distilled policy with task-specific policies.
• It employs a joint optimization objective that minimizes KL divergence and utilizes entropy regularization to balance exploration and exploitation.
• Numerical results show improved learning speed and stability in complex 3D tasks compared to baseline methods like A3C.

Overview of "Distral: Robust Multitask Reinforcement Learning"

The paper discusses a novel approach to multitask reinforcement learning (RL), named Distral (Distill and Transfer Learning), aimed at improving data efficiency and stability in complex 3D environments. Traditional deep RL algorithms often exhibit data inefficiency and instability, particularly when dealing with multiple tasks. Distral addresses these challenges by proposing a method for joint training across tasks through shared policy distillation.

Methodology

In Distral, the emphasis is on leveraging a 'distilled' policy that captures common behaviors across tasks rather than directly sharing network parameters among task-specific models. Each task-specific 'worker' policy is trained to perform its unique task while remaining close to the shared distilled policy. Conversely, the distilled policy is refined to be a centroid of the task policies. This design is formalized through a joint optimization objective, which simultaneously maximizes expected returns and minimizes the KL divergence between task-specific and distilled policies.

The paper introduces a mathematical framework that regularizes task policies towards the distilled policy and encourages exploration through discounted entropy regularization. This dual regularization assists in maintaining task diversity while promoting effective learning transfer.

Numerical Results

Distral demonstrates significant performance improvements over existing methods, such as A3C, when evaluated in visually rich, complex environments. Specifically, the framework proves effective in enhancing learning speed, asymptotic performance, and robustness to hyperparameter variations. For instance, the paper reports superior stability and efficiency in 3D maze and navigation tasks, outperforming baseline multitask learning approaches.

Implications and Future Directions

The implications of Distral's approach are substantial, suggesting avenues for more robust RL algorithms that can efficiently handle multitask scenarios while maintaining stable learning dynamics. By distilling common behaviors and regulating exploration, Distral provides a pathway for improved transfer learning in RL.

Future research could explore the integration of auxiliary tasks to further bolster data efficiency, the development of policies that account for greater task diversity, or application to sequential task scenarios indicative of continual learning environments. Furthermore, adaptive regularization techniques might refine the balance between exploration and exploitation, enhancing task-specific optimization without compromising transferability.

Thus, Distral not only contributes to the theoretical understanding of multitask learning in RL but also extends practical capabilities for AI systems operating in dynamically complex environments.