Generalized Bayesian deep reinforcement learning (2412.11743v2)

Abstract: Bayesian reinforcement learning (BRL) is a method that merges principles from Bayesian statistics and reinforcement learning to make optimal decisions in uncertain environments. As a model-based RL method, it has two key components: (1) inferring the posterior distribution of the model for the data-generating process (DGP) and (2) policy learning using the learned posterior. We propose to model the dynamics of the unknown environment through deep generative models, assuming Markov dependence. In the absence of likelihood functions for these models, we train them by learning a generalized predictive-sequential (or prequential) scoring rule (SR) posterior. We used sequential Monte Carlo (SMC) samplers to draw samples from this generalized Bayesian posterior distribution. In conjunction, to achieve scalability in the high-dimensional parameter space of the neural networks, we use the gradient-based Markov kernels within SMC. To justify the use of the prequential scoring rule posterior, we prove a Bernstein-von Mises-type theorem. For policy learning, we propose expected Thompson sampling (ETS) to learn the optimal policy by maximising the expected value function with respect to the posterior distribution. This improves upon traditional Thompson sampling (TS) and its extensions, which utilize only one sample drawn from the posterior distribution. This improvement is studied both theoretically and using simulation studies, assuming a discrete action space. Finally, we successfully extended our setup for a challenging problem with a continuous action space without theoretical guarantees.

• The paper introduces a model-based reinforcement learning framework that integrates prequential scoring rules to infer parameters without tractable likelihoods.
• It develops Expected Thompson Sampling as a multi-sample strategy that outperforms standard methods, achieving faster convergence and reduced regret.
• Empirical results on tasks like the inverted pendulum and Hopper demonstrate improved learning efficiency across both finite and continuous action spaces.

Generalized Bayesian Deep Reinforcement Learning

The paper "Generalized Bayesian Deep Reinforcement Learning" introduces a rigorous framework for model-based reinforcement learning (RL), integrating Bayesian inference into the domain of deep RL to improve decision-making under uncertainty. The authors propose using generalized Bayesian methods, particularly prequential scoring rules, to infer model parameters in scenarios where likelihood functions are intractable, such as when dynamics are modeled with deep generative networks.

Prequential Scoring Rule and Bayesian Inference

The paper begins by tackling the challenge of learning complex models where the traditional likelihood functions are unavailable. To address this, the authors leverage prequential scoring rules, which are inherently proper scoring rules used to assess the fit of predictive distributions in Bayesian settings. This approach is used to define a generalized Bayesian posterior for reinforcement learning tasks, notably under Markov decision processes (MDPs). The prequential scoring rule-based Bayesian framework allows for effective learning and updating of model parameters without requiring explicit likelihood formulations.

The generalized posterior is shown to converge to the true model parameters under a Bernstein-von Mises type theorem, specifically for finite action spaces. This result extends classical consistency guarantees of Bayesian methods, ensuring the robustness of the learned models in representing true environmental dynamics.

Expected Thompson Sampling for Policy Learning

A significant contribution of this research is the development of Expected Thompson Sampling (ETS), an enhancement over traditional Thompson Sampling (TS) methods. In standard TS, a single sample from the model's posterior is used to inform policy decisions, which can lead to suboptimal exploration. ETS addresses this limitation by using multiple samples from the posterior, allowing for a more comprehensive exploration-exploitation balance. This multi-sample approach is theoretically justified and empirically shown to result in faster convergence to optimal policies, as demonstrated by reduced regret in the experiments.

Empirical Validation and Application

The empirical results extend across scenarios with both finite and continuous action spaces. For finite action spaces, the inverted pendulum task showcases the advantages of ETS in both well-specified and misspecified models, outperforming model-free RL strategies like least squares policy iteration (LSPI). The paper further applies their framework to continuous action spaces, exemplified by the Hopper task, using policy gradient methods enhanced by ETS. In all cases, the integration of ETS with Bayesian model inference leads to more efficient learning processes.

Theoretical and Practical Implications

Theoretically, this work advances the understanding of Bayesian methods in reinforcement learning by formalizing the use of scoring rules and demonstrating their compatibility with complex models like deep networks. Practically, the proposed method offers a scalable solution to deal with model uncertainty and sample inefficiency, a common challenge in real-world RL applications. This approach could lead to significant improvements in domains such as robotics, autonomous systems, and any application requiring quick adaptability to changing environments.

Future Directions

The research suggests several avenues for future exploration. While the paper establishes strong theoretical foundations and practical improvements, further applications to diverse types of generative models and exploration of shrinkage priors for high-dimensional parameter spaces could enhance model robustness. Developing unified objectives that integrate both model learning and policy adaptation could optimize the entire learning process, effectively bridging the gap between Bayesian modeling and reinforcement learning.

In conclusion, this paper presents a comprehensive framework that utilises generalized Bayesian inference to overcome the limitations posed by intractable likelihoods in reinforcement learning. The incorporation of the ETS approach promises more effective and efficient policy learning, setting the stage for future advancements in adaptive AI systems.