Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Space Models

Abstract: Model-Based Reinforcement Learning distinguishes between physical dynamics models operating on proprioceptive inputs and latent dynamics models operating on high-dimensional image observations. A prominent latent approach is the Recurrent State Space Model used in the Dreamer family. While epistemic uncertainty quantification to inform exploration and mitigate model exploitation is well established for physical dynamics models, its transfer to latent dynamics models has received limited scrutiny. We empirically demonstrate that latent transitions are biased toward well-represented regions of latent space, exhibiting an attractor behavior that can deviate from true environment dynamics. As a result, discrepancies in environment dynamics may not manifest in latent space, undermining the reliability of epistemic uncertainty estimates. Because these attractors often lie in high-reward regions, latent rollouts systematically overestimate predicted rewards. Our findings highlight key limitations of epistemic uncertainty estimation in latent dynamics models and motivate more critical evaluation of this method.

• The paper demonstrates that latent dynamics models (RSSM) exhibit attractor behavior, causing ensemble uncertainty to be suppressed despite accumulating physical state errors.
• The study reveals that reward overestimation in latent rollouts introduces an optimism bias in reinforcement learning policy updates.
• The findings underscore the need for architectural innovations to realign epistemic uncertainty metrics with true predictive errors in model-based planning.

Limitations of Epistemic Uncertainty Quantification in Latent Dynamics Models

Introduction

The paper "Biased Dreams: Limitations to Epistemic Uncertainty Quantification in Latent Space Models" (2604.25416) presents a systematic empirical analysis of epistemic uncertainty quantification within latent dynamics models, focusing on the widely used Recurrent State Space Model (RSSM) as popularized in the Dreamer family. While epistemic uncertainty estimation is a cornerstone of safety, robustness, and exploration strategies in model-based reinforcement learning (MBRL) leveraging physical dynamics models, its direct application to latent-space models remains largely under-examined in terms of reliability and actual error correspondence. This work establishes that latent dynamics models, unlike physical ones, can exhibit a pronounced attractor effect in latent space, which severely biases both the quantification and interpretation of epistemic uncertainty, leading to persistent overestimation errors in reward predictions used for downstream RL policy optimization.

Attractor Dynamics in Latent Space

The study demonstrates that latent rollouts generated by RSSM often converge, irrespective of their initialization, toward dominant, well-represented latent regions, a manifestation of attractor dynamics induced by the inductive bias of deep sequential variational inference. This attractor behavior ensures that out-of-distribution (OOD) initializations in the latent space are rapidly mapped back to familiar trajectories, even when these trajectories diverge significantly from the true physical system states.

Figure 1: PCA-embedded latent transition dynamics initiated from OOD states show RSSM trajectories converging to familiar latent regions with reduced uncertainty, while probabilistic ensemble (PE) rollouts in physical space rapidly accumulate error and high uncertainty.

Comparative analysis with physical dynamics models (e.g., probabilistic ensembles, PE) exposes that while physical models display rapidly compounding errors and increasing predictive uncertainty under OOD conditions, the RSSM's attractor behavior in latent space leads to apparent stability and decreased epistemic uncertainty, even as genuine physical errors accumulate. This effect violates the fundamental assumption underlying ensemble-based epistemic uncertainty: that high uncertainty should reliably signal model misalignment from the true environment, especially under distributional shift.

Quantitative Evaluation of Latent/Physical Mismatch

Systematic experiments across standard DMC Suite tasks (Cartpole Swingup, Cheetah Run, Hopper Hop, Walker Run) confirm that in both ID and OOD initializations, RSSM and its categorical variant (Cat-RSSM) generate prior rollouts with a monotonic increase in the physical state reconstruction error, whereas the associated ensemble epistemic uncertainty drops abruptly following the attractor pull into high-density regions. The physical discrepancy is quantified using a separately trained physical state decoder, and uncertainty via the geometric Jensen-Shannon divergence across five transition ensemble members.

Figure 2: Attractor evaluation for RSSM and PE under ID and OOD start states; RSSM exhibits attractor dynamics drawing OOD rollouts toward dominant latent flows, demonstrated across standard control tasks.

Figure 3: Attractor analysis of RSSM and Cat-RSSM shows similar attractor-induced latent convergence across Cartpole, Hopper, and Walker environments, for both ID and OOD conditions.

This decoupling is visualized through both PCA projections and rollout statistics. Notably, during OOD rollouts, initial uncertainty is elevated but diminishes rapidly as the trajectory is absorbed by the latent attractor, while reconstructed physical state error remains high throughout.

Reward Overestimation Bias

A key finding is the systematic overestimation of rewards along prior latent rollouts relative to actual environment-simulated rewards. These attractor regions in the latent space frequently overlap with states associated with high expected return in the learned model, introducing an optimism bias in model-based policy optimization.

Figure 4: Physical trajectory comparison in ID and OOD settings for a physical dynamics PE baseline highlights increasing divergence in OOD rollouts absent in latent space.

Figure 5: Reconstructed image trajectories in ID setting for RSSM and Cat-RSSM closely match ground truth, validating representational capacity in-distribution.

Figure 6: In OOD settings, reconstructed image trajectories for RSSM and Cat-RSSM are rapidly drawn to familiar regions, irrespective of true underlying environment dynamics.

Figure 7: Physical state rollouts in ID for both model variants align well with simulation, substantiating predictive accuracy when on-distribution.

Figure 8: In OOD, latent trajectory reconstructions provide false confidence, demonstrated by continued physical state mismatch while uncertainty estimates are suppressed.

This result has direct implications for RL updates: the agent systematically updates its policy towards behaviors that the latent model expects to be high performing but do not correspond to actual high-reward or feasible behaviors in the true environment. Posterior rollouts (i.e., rollouts leveraging current observation encodings at every step) remain aligned both in uncertainty and performance; however, these are unavailable for future hypothetical rollouts required by model-based planning.

Implications for Epistemic Uncertainty and MBRL

The empirical results undercut the practice of directly transferring ensemble-based epistemic uncertainty metrics from physical to latent dynamics models. The attractor phenomenon, inherent to ELBO-driven DVAE objectives in RSSM, means that low disagreement among latent transition ensemble members is not indicative of valid dynamics under OOD or compounding error conditions. This fundamentally limits the reliability of uncertainty-driven mechanisms for exploration, safety, and exploitation mitigation as deployed in standard latent world models.

Architectural changes are implicated as necessary: simply improving uncertainty metrics atop standard RSSM will not resolve the attractor-induced bias. Promising directions discussed include restructuring the latent space to enforce more locally-consistent mappings, as in Fourier-Latent-Dynamics [li2024fld] or contrastive/physically-regularized objectives [becker2023coral; peper2025four]. Additionally, more principled variational inference objectives may be required to align epistemic measures with true model error [becker2022vrkn].

Conclusion

This work demonstrates that ensemble-based epistemic uncertainty, when applied to standard RSSM-like latent dynamics models, can become decoupled from genuine predictive error due to attractor-induced bias in latent transition dynamics. As a result, policies optimized with such models are systematically exposed to optimism bias, and mechanisms for safe or exploratory RL grounded in these uncertainty metrics are rendered unreliable under significant distribution shift. These insights highlight the necessity of theory-driven latent space structuring or alternative inference schemes in future world model design, to recover meaningful epistemic uncertainty metrics and enable robust, safe, and performant model-based RL.