Reinforcement Learning with Deep Energy-Based Policies (1702.08165v2)

Abstract: We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows transferring skills between tasks, which we confirm in simulated experiments with swimming and walking robots. We also draw a connection to actor-critic methods, which can be viewed performing approximate inference on the corresponding energy-based model.

• The paper presents soft Q-learning, a novel algorithm extending maximum entropy reinforcement learning to support arbitrary policy distributions.
• It leverages amortized Stein variational gradient descent for efficient sampling from a Boltzmann policy distribution, thereby improving exploration.
• Experimental results demonstrate superior exploration and enhanced transfer learning in continuous control tasks compared to state-of-the-art methods.

Reinforcement Learning with Deep Energy-Based Policies

This paper, written by Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine, addresses the challenge of learning expressive energy-based policies for reinforcement learning (RL) in continuous state and action spaces. This task has traditionally been feasible only in tabular domains. The authors propose a novel method that combines deep reinforcement learning (deep RL) with energy-based models (EBMs), leading to the introduction of a new algorithm called soft Q-learning.

Core Concept and Methodology

The central contribution of this paper lies in the extension of maximum entropy policy search to support arbitrary policy distributions, leveraging the rich representational power of EBMs. The proposed method aims to address two main challenges:

1. Improved Exploration: By learning stochastic policies, the approach enables better exploration in environments with multimodal reward distributions.
2. Compositionality for Transfer Learning: The learned policies serve as a robust foundation for transferring skills across different tasks.

The authors frame the control problem as an inference problem by using a maximum entropy Reinforcement Learning (MaxEnt RL) objective. They directly optimize the policy to maximize both the expected reward and the entropy, promoting stochasticity in the learned policies. Typical deep RL methods use deterministic optimization; however, the MaxEnt framework allows policies to capture a variety of potential behaviors, enhancing exploration and robustness.

Soft Q-Learning Algorithm

The novel soft Q-learning algorithm modifies traditional Q-learning to fit the MaxEnt objective. The Q-function is related to the policy via a Boltzmann distribution, where the Q-values serve as energies in the EBM. However, sampling from these distributions in high-dimensional spaces can be computationally intensive. Hence, the authors employ amortized Stein variational gradient descent (SVGD) to learn a sampling network. This network generates samples that approximate the true distribution of actions, thus reducing the intractability of sampling.

Theoretical Contributions

The paper provides rigorous theoretical underpinnings, demonstrating that:

• The soft Q-function satisfies a soft BeLLMan equation, generalizing the conventional BeLLMan equation to accommodate the entropy term.
• The solution to this equation optimizes the entropy-augmented objective and naturally results in policies represented by energy-based models.

Experimental Results

The authors validate their method through extensive experiments involving both simple and complex continuous control tasks. Key results include:

• Multi-goal Environment: Demonstrates the algorithm's ability to learn multimodal policies effectively. The learned policies can select optimal actions leading to all goals rather than committing prematurely to a single goal.
• Improved Exploration: In tasks such as a snake-like swimming robot and a quadrupedal walking robot navigating a maze, the soft Q-learning algorithm significantly outperforms state-of-the-art methods like DDPG in terms of exploration efficiency.
• Compositionality and Transfer Learning: The method provides a superior initialization for learning new tasks, facilitating faster adaptation compared to randomly initialized policies or those initialized with conventional deterministic objectives.

Practical and Theoretical Implications

The practical implications of this research are manifold:

• Applications in robotics where continuous control and exploration are crucial, such as autonomous navigation and manipulation.
• Potential to improve the robustness of RL algorithms in adversarial environments, enhancing the reliability of learned behaviors in real-world scenarios.

From a theoretical perspective, the soft Q-learning algorithm bridges the gap between RL and probabilistic inference, offering a new avenue for developing RL algorithms with enhanced exploration capabilities and robust policy learning.

Future Directions

Future research can explore further composability of energy-based policies in more complex, high-dimensional tasks, potentially leading to modular skill-building and transfer. Additionally, investigating the scalability of the proposed method to even larger action spaces and its application to real-world robotic systems could yield valuable insights.

In summary, Haarnoja et al. have introduced a significant advancement in the field of RL by integrating deep learning with energy-based models to formulate soft Q-learning. This approach not only improves exploration but also facilitates the transfer of learned policies, presenting a substantial step forward in the development of versatile and robust autonomous systems.