Posterior Sampling in Reinforcement Learning

• Posterior Sampling for Reinforcement Learning (PSRL) is a Bayesian approach that samples plausible environment models to manage the exploration–exploitation tradeoff.
• The algorithm involves sampling a model from the posterior, computing an optimal policy, and updating beliefs with new data, ensuring statistical efficiency.
• PSRL achieves near-optimal Bayesian regret bounds and extends to settings like causal priors, constrained MDPs, and deep reinforcement learning.

Posterior Sampling for Reinforcement Learning (PSRL) is a randomized, Bayesian approach to the exploration–exploitation tradeoff in reinforcement learning (RL). At its core, PSRL operates by sampling plausible models of the environment from the current posterior, executing the optimal policy under the sampled model for a designated duration, and repeating this procedure as new data are collected. This method enables both statistical efficiency—matching or exceeding the minimax rates of optimism-based approaches—and practical flexibility, as the structure and inductive biases of the prior can be tuned to the domain.

1. Principles and Algorithmic Structure

The canonical PSRL algorithm is defined for finite-horizon Markov Decision Processes (MDPs) $M = (S, A, p, r, \mu, H)$, where $p(s'|s,a)$ is the transition kernel, $r(s,a)$ is the mean reward, $\mu$ is the initial-state distribution, and $H$ is the horizon. A Bayesian agent maintains a prior $P_0$ over MDPs. At the beginning of each episode $k$, given data $\mathcal{D}_{k-1}$, the agent forms the posterior $P_k = P(M | \mathcal{D}_{k-1})$, then:

1. Sample Model: Draw $M_k \sim P_k$.
2. Optimal Policy Planning: Compute $p(s'|s,a)$0, e.g., via value iteration.
3. Policy Execution: Execute $p(s'|s,a)$1 in the true environment $p(s'|s,a)$2, collect $p(s'|s,a)$3 transitions and rewards, and update $p(s'|s,a)$4 and thus $p(s'|s,a)$5.
4. Repeat for $p(s'|s,a)$6 episodes.

The essential innovation is the randomized exploration induced by sampling, which eliminates the need for explicit optimism or bonus-based confidence intervals (Osband et al., 2013, Mutti et al., 2023).

2. Regret Analysis and Theoretical Guarantees

PSRL achieves strong Bayesian regret guarantees, typically near-optimal in the sense of minimax lower bounds for RL exploration. For finite-horizon tabular MDPs, the Bayesian regret over $p(s'|s,a)$7 episodes satisfies

$p(s'|s,a)$8

where $p(s'|s,a)$9 is the horizon, $r(s,a)$0 and $r(s,a)$1 the number of states and actions, and $r(s,a)$2 hides logarithmic factors (Osband et al., 2013, Mutti et al., 2023, Osband et al., 2016).

The key proof mechanism decomposes regret into a model error term (controlled by high-probability confidence sets) and a sampling error term (amenable to martingale concentration via Azuma–Hoeffding inequality). PSRL’s regret bound systematically improves over standard OFU-based bounds by eliminating superfluous factors due to the stochastic directionality of the posterior samples (Osband et al., 2016).

For function approximation settings, recent analyses establish Bayesian regret bounds in terms of the intrinsic model dimension: e.g., $r(s,a)$3 for linear MDPs (Fan et al., 2020), $r(s,a)$4 for linear-mixture MDPs with prior-dependent constants (Li et al., 2024), and $r(s,a)$5 for Gaussian-process PSRL in continuous control, where $r(s,a)$6 is the maximum information gain (Flynn et al., 9 Mar 2026). Extensions under log-Sobolev inequalities support $r(s,a)$7 rates far beyond conventional log-concave posteriors (Jorge et al., 2024).

3. Extensions: Causal Priors, SSPs, and Zero-Sum Games

Causal Graph Priors (C-PSRL): When a (partial) causal graph over state and action variables is available, the prior can be specified as a (possibly incomplete) bipartite graph connecting features to successor features. The algorithm models the distribution over possible factorizations $r(s,a)$8 (where $r(s,a)$9 is the parent set of the $\mu$0-th transition factor), with Dirichlet priors over factored transitions. The joint posterior is hierarchical: $\mu$1 The regret bound interpolates smoothly between uninformative and informative priors, with prior knowledge degree $\mu$2 (known parent edges per factor) yielding

$\mu$3

Partial causal knowledge (larger $\mu$4) exponentially reduces sample complexity (Mutti et al., 2023).

Stochastic Shortest Path (PSRL-SSP): For SSP problems with absorbing goals and possibly improper policies, PSRL-SSP samples models at epoch boundaries defined by visits to the goal or by doubled visit counts. The Bayesian regret is $\mu$5 where $\mu$6 is the optimal cost upper bound, $\mu$7 is the state space size, and $\mu$8 the action space (Jafarnia-Jahromi et al., 2021).

Zero-Sum Stochastic Games (PSRL-ZSG): Posterior Sampling generalizes to adversarial two-player zero-sum stochastic games by sampling transition kernels and planning max-min policies over the sampled model. The Bayesian regret for arbitrary opponents is $\mu$9, matching known information-theoretic lower bounds up to log factors (Jafarnia-Jahromi et al., 2021).

4. Generalizations: Infinite-Horizon, Constraint, and Function Approximation

PSRL generalizes to a variety of settings:

• Infinite-/Continuing-Horizon RL: Extensions exist where "episodes" are defined by geometric resampling (Continuing PSRL: sample a new MDP with probability $H$0), yielding average-reward Bayesian regret $H$1 under weakly communicating MDP assumptions, with $H$2 the reward-averaging time (Xu et al., 2022). For standard non-episodic settings, practical PSRL variants mimic OFU-style episode triggers (e.g., policy switches at visit-count doublings). While proofs for these variants remain challenging, conjectured regret rates are $H$3 (Osband et al., 2016, Theocharous et al., 2017).
• Constrained RL: In constrained MDPs, PSRL can sample the transition model at the beginning of each epoch, solve the occupancy-measure LP (linear-formulation CMDP) or the Lagrangian-relaxed saddle-point problem, and execute the resulting policy until an epoch-ending condition is met. Empirically, PSRL-based algorithms outperform or match optimism-based baselines, with significantly lower computational overhead (Provodin et al., 2022).
• Partially Observable MDPs (POMDPs): The PSRL principle can be extended to POMDPs by maintaining a posterior over latent transition models and beliefs, solving for the optimal policy in the sampled POMDP, and executing it on the induced belief process. When the parameter set is finite, sub-logarithmic regret $H$4 is achieved; for continuous parameters, $H$5 rates are possible under technical conditions (Jafarnia-Jahromi et al., 2021).
• Function Approximation and Deep RL: Under linear, kernelized, or even non-linear function classes, a variety of posterior sampling-based deep-RL instantiations exist. These include tailored variational distributions (e.g., Gaussian dropout or event-based convolutional layers in EVaDE for deep MBRL (Aravindan et al., 16 Jan 2025)), successor uncertainties for randomized value-function exploration (Janz et al., 2018), and two-timescale Thompson sampling for large and continuous action spaces (Agarwal et al., 2022). Recent analysis delivers regret bounds in terms of effective dimension (e.g., $H$6 for Bayesian linear regression, matching OFU-style results) (Fan et al., 2020, Li et al., 2024).

5. Methodological Innovations and Computational Aspects

PSRL's analytical tractability arises from its intrinsic structure:

• Posterior Equivalence Lemma: At episode starts, the sampled model and true model are identically distributed conditional on past data; thus "optimism error" cancels in expectation, focusing analysis on the value estimation error.
• Confidence-Free Exploration: Unlike OFU methods, PSRL does not require confidence set construction or bonus parameter tuning. Randomization over posterior samples automatically induces sufficient exploration as uncertainty persists.
• Prior Knowledge Integration: Structural information—parametric, factored, or causal—can be encoded natively in the prior (e.g., via partial causal graphs (Mutti et al., 2023)). This approach allows practitioners to exploit domain knowledge to reduce sample complexity.
• Computational Complexity: Each episode requires solving an optimal control problem for a single sampled MDP, which is typically cheaper than the max-max or max-min optimization over confidence sets in optimistic paradigms (Osband et al., 2013, Mutti et al., 2023, Provodin et al., 2022). In large-scale or continuous domains, scalable variants utilize model parametrization, deterministic switching (DS-PSRL (Theocharous et al., 2017)), or function approximation for tractability.

6. Empirical Results and Benchmarking

Comprehensive empirical studies confirm PSRL’s efficiency:

• Tabular Benchmarks: On RiverSwim, Taxi, random gridworlds, and SSP domains, PSRL and its structural variants (C-PSRL, F-PSRL) exhibit strong sample efficiency, with regret substantially smaller than that of optimistic approaches. When partial structural priors are available, C-PSRL achieves near-oracle performance (Mutti et al., 2023, Osband et al., 2013, Osband et al., 2016).
• Constrained and Structured Domains: In box-pushing and Marsrover CMDPs, PSRL-based algorithms avoid the stagnation observed in OFU methods and converge to true optimal policies, even under cost constraints (Provodin et al., 2022).
• Continuous Control and Deep RL: In low/medium-dimensional control (Cartpole, Pendulum, Reacher, Pusher), model-based PSRL with Bayesian linear regression on neural network features matches or exceeds data efficiency of prior model-based and model-free baselines (e.g., MBPO, PETS, SAC) (Fan et al., 2020). Event-based variational formulation (EVaDE) yields improved exploration and human-normalized scores on Atari at 100K steps, besting SimPLe and CURL (Aravindan et al., 16 Jan 2025), and successor uncertainties yield superhuman performance on Atari with statistically efficient deep exploration (Janz et al., 2018).

Empirical summaries frequently report per-episode regret, cumulative regret curves, and convergence speed, consistently demonstrating PSRL’s superior or competitive performance relative to optimistic and heuristic exploration methods.

7. Open Directions and Future Developments

Despite its strong theoretical and empirical performance, several open questions and challenges remain:

• Computational Scalability: Efficient sampling and planning under non-conjugate posteriors, deep neural parameterizations, or unknown dynamics remain areas of active research, with advances (e.g., SARAH-LD for Langevin sampling (Jorge et al., 2024)) closing the gap.
• Regret for Adaptive/Infinite-Horizon Regimes: Rigorous regret guarantees for PSRL in infinite-horizon, non-episodic, or continuing environments with nontrivial switching schemes remain subtle, with conjectured $H$7 rates (Osband et al., 2016).
• Model-Free Posterior Sampling: Direct, non-parametric posterior sampling for Q-functions or policies—especially in the presence of function approximation or nonlinearity—presents statistical and algorithmic challenges, though recent advances match or beat the best known linear MDP bounds (Dann et al., 2022, Agarwal et al., 2022).
• Incorporation of Rich Priors: Causal, factored, or otherwise structured priors offer significant gains in sample efficiency but invite new questions regarding prior misspecification, identifiability, and computational tractability (Mutti et al., 2023, Agarwal et al., 2022).
• Beyond Standard RL: Zero-sum games, constrained RL, POMDPs, and adaptive recommender systems present fertile ground for further development of PSRL methods matching the theoretical guarantees of standard RL (Jafarnia-Jahromi et al., 2021, Jafarnia-Jahromi et al., 2021, Theocharous et al., 2017).

Overall, PSRL unifies statistically efficient exploration, flexible modeling, and practical tractability, and therefore occupies a central position in contemporary RL methodology and theory.