Deep Gaussian Covariance Network with Trajectory Sampling for Data-Efficient Policy Search

Abstract: Probabilistic world models increase data efficiency of model-based reinforcement learning (MBRL) by guiding the policy with their epistemic uncertainty to improve exploration and acquire new samples. Moreover, the uncertainty-aware learning procedures in probabilistic approaches lead to robust policies that are less sensitive to noisy observations compared to uncertainty unaware solutions. We propose to combine trajectory sampling and deep Gaussian covariance network (DGCN) for a data-efficient solution to MBRL problems in an optimal control setting. We compare trajectory sampling with density-based approximation for uncertainty propagation using three different probabilistic world models; Gaussian processes, Bayesian neural networks, and DGCNs. We provide empirical evidence using four different well-known test environments, that our method improves the sample-efficiency over other combinations of uncertainty propagation methods and probabilistic models. During our tests, we place particular emphasis on the robustness of the learned policies with respect to noisy initial states.

• The paper presents a novel integration of deep Gaussian covariance networks and trajectory sampling that significantly enhances sample efficiency in model-based reinforcement learning.
• It demonstrates that the proposed method outperforms traditional Gaussian process and ensemble approaches, especially under noisy initial conditions.
• Experimental results on benchmarks like IPSU and CMC confirm the method's competitiveness and potential for robust, data-efficient policy search.

Exploring Sample Efficiency and Robust Policy Learning in MBRL via Deep Gaussian Covariance Networks and Trajectory Sampling

Introduction

The quest for data-efficient solutions in Model-Based Reinforcement Learning (MBRL) has led to the exploration of probabilistic world models. These models leverage epistemic uncertainty to guide policy improvement and exploration, aiming at developing robust policies with fewer samples. This paper introduces a methodology combining Deep Gaussian Covariance Networks (DGCN) with trajectory sampling, aiming to achieve optimal policy search efficiency in MBRL settings. The approach is evaluated against other probabilistic models under various uncertainty propagation methods, with a focus on sample efficiency and policy robustness in noisy environments.

Probabilistic Models in Focus

• Gaussian Process Regression: Serves as a foundational probabilistic model, emphasizing the role of GP in incorporating both aleatoric and epistemic uncertainties into model predictions. The implementation utilizes widely recognized kernels, such as the squared exponential and Matérn kernels, facilitated by an extensive overview of GP’s theoretical underpinnings.
• Deep Gaussian Covariance Network (DGCN): Builds on the GP framework by employing neural networks to predict kernel parameters dynamically, thus accommodating non-stationary data characteristics. The model efficiently handles heteroscedastic uncertainty, marking a significant advancement over traditional GP models.
• Probabilistic Neural Networks (PNN): Represented in this study by ensemble-based PNN models (E-PNN), these capture predictive distributions and emphasize the amalgamation of aleatoric and epistemic uncertainties, showcasing a competitive alternative to DGCN in MBRL applications.

Uncertainty Propagation Methods

Two principal methods for propagating uncertainty were assessed:

• Density-based Uncertainty Propagation: Utilizes approximations (e.g., moment matching) to propagate uncertainty through models, a method traditionally associated with GP models and requiring Gaussian distribution assumptions.
• Trajectory Sampling: Offers a more flexible and potentially more accurate alternative for non-linear and complex dynamics, eschewing the need for Gaussian approximations by directly sampling trajectories according to the probabilistic model's output.

Deep Gaussian Covariance Network with Trajectory Sampling (DGCNTS)

A novel framework termed DGCNTS was proposed, integrating DGCN with trajectory sampling within an MBRL setup. Unlike traditional methods that might rely on ensemble models or restrictive uncertainty propagation techniques, DGCNTS leverages the best of both worlds: the robust uncertainty modeling capabilities of DGCN and the flexibility of trajectory sampling. This combination seeks to enhance sample efficiency and policy robustness, particularly under noisy initial states.

Experimental Insights

Empirical tests were conducted across four benchmarks: Inverted Pendulum Swing Up (IPSU), Continuous Mountain Car (CMC), Inverted Double Pendulum (IDP), and Pendulum (P). Key findings include:

• Sample Efficiency: DGCNTS exhibited pronounced improvements in sample efficiency across most tested environments when compared to ensemble-based PNNs and GP models utilizing density-based uncertainty propagation.
• Robustness to Initial State Variability: Policies derived using DGCNTS demonstrated superior robustness against noisy initial conditions, underscoring the method's practicality in real-world applications where such conditions are prevalent.
• Comparison with Existing Methods: The performance of DGCNTS was consistently competitive or superior to existing probabilistic models combined with alternative uncertainty propagation methods, highlighting its effectiveness in navigating MBRL problems.

Future Directions

Given the promising results of DGCNTS, several avenues for future research emerge:

• Scalability and Complexity: Extending the DGCNTS framework to handle more complex tasks and higher-dimension spaces remains an open challenge.
• Exploration Mechanisms: Integrating exploration strategies that leverage epistemic uncertainty could further improve the data efficiency of DGCNTS.
• Model Predictive Control (MPC): The applicability of DGCNTS in MPC settings warrants exploration, potentially broadening its utility in MBRL.

Conclusion

The introduction of DGCNTS represents a significant stride toward resolving the sample efficiency challenges in MBRL. By marrying DGCN's robust uncertainty handling with the flexible trajectory sampling approach, this method sets a new precedent for developing efficient and robust policies under conditions of uncertainty. As MBRL continues to evolve, techniques like DGCNTS will undoubtedly play a pivotal role in enhancing the practicality and applicability of MBRL solutions across a wide array of tasks and environments.