Sample Efficient Policy Gradient Methods with Recursive Variance Reduction (1909.08610v3)

Abstract: Improving the sample efficiency in reinforcement learning has been a long-standing research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires $O(1/\epsilon^{3/2})$ episodes to find an $\epsilon$-approximate stationary point of the nonconcave performance function $J(\boldsymbol{\theta})$ (i.e., $\boldsymbol{\theta}$ such that $|\nabla J(\boldsymbol{\theta})|_2^2\leq\epsilon$). This sample complexity improves the existing result $O(1/\epsilon^{5/3})$ for stochastic variance reduced policy gradient algorithms by a factor of $O(1/\epsilon^{1/6})$. In addition, we also propose a variant of SRVR-PG with parameter exploration, which explores the initial policy parameter from a prior probability distribution. We conduct numerical experiments on classic control problems in reinforcement learning to validate the performance of our proposed algorithms.

• The paper presents SRVR-PG, a policy gradient algorithm that reduces sample complexity from O(1/ε^(5/3)) to O(1/ε^(3/2)).
• It introduces a variant with parameter exploration, initializing policies from a prior distribution to enhance exploration.
• Experimental results on classic control tasks validate the efficiency improvements and practical potential of SRVR-PG in reinforcement learning.

Sample Efficient Policy Gradient Methods with Recursive Variance Reduction

This paper presents a novel approach to improving sample efficiency in reinforcement learning through the development of a policy gradient algorithm named SRVR-PG (Stochastic Recursive Variance Reduced Policy Gradient). The focus of the research is to enhance the convergence rate of policy gradient methods by reducing the required number of trajectory samples, thereby decreasing overall sample complexity. The authors propose that SRVR-PG achieves this by leveraging recursive variance reduction techniques.

Main Contributions

1. Reduced Sample Complexity: The proposed SRVR-PG algorithm requires $O(1/\epsilon^{3/2})$ episodes to find an $\epsilon$-approximate stationary point of the non-concave performance function $J(\boldsymbol{\theta})$. This is a significant improvement over the existing $O(1/\epsilon^{5/3})$ for stochastic variance reduced policy gradient algorithms, equating to a reduction factor of $O(1/\epsilon^{1/6})$.
2. Variant with Parameter Exploration: The authors introduce a variant of SRVR-PG that includes parameter exploration. This variant involves initializing the policy parameter from a prior probability distribution, which offers potential benefits in exploring the policy space more effectively.
3. Experimental Validation: The paper presents numerical results validating the theoretical improvements of the SRVR-PG and its variant over standard policy gradient methods on classic control problems within reinforcement learning.

Implications and Future Directions

This work underscores the potential of recursive variance reduction techniques in achieving sample-efficient policy optimization. The reduction in sample complexity not only signifies computational savings but also implies improvements in the applicability of reinforcement learning in environments where data collection is costly or time-consuming.

Theoretical Implications: The results provide a theoretical groundwork for developing more sophisticated variance reduction methods in policy gradient algorithms. Future work could expand this paper to integrate additional gradient estimation techniques or consider adaptive step-size strategies based on variance estimates.

Practical Implications: The proposed algorithm could be influential in industries relying on reinforcement learning for decision-making processes, such as autonomous driving or robotic controls, where reducing the interaction with the environment while still learning effective policies is crucial.

Future Research Directions:

• Scalability: Investigating the scalability of SRVR-PG in high-dimensional action spaces or in scenarios with complex, multi-agent environments.
• Robustness: Evaluating robustness concerning non-stationary environments and how the recursive variance reduction could be adapted or enhanced to handle dynamic changes more efficiently.
• Integration with Other RL Paradigms: Exploring the application of recursive variance reduction within other paradigms of reinforcement learning, such as actor-critic methods, to evaluate potential improvements in sample efficiency across a broader spectrum of algorithms.

The paper contributes to the ongoing evolution of reinforcement learning methods by articulating both theoretical advancements and empirical validations that bolster policy gradient methods' efficiency. With continued research and refinement, such algorithms could bolster reinforcement learning applications across a diverse set of complex real-world problems.