DreamerV3-style MBRL Agent

• DreamerV3-style MBRL agent is a reinforcement learning paradigm that uses learned predictive world models and recurrent state-space architectures for high-dimensional control.
• It employs ELBO-based training on recurrent state-space models and latent imagination rollouts, ensuring robust prediction, efficient policy optimization, and adaptable learning.
• Incorporating intrinsic motivation through latent disagreement and uncertainty-driven strategies, it significantly enhances sample efficiency and generalization across various environments.

A DreamerV3-style Model-Based Reinforcement Learning (MBRL) agent refers to a class of algorithms centered on learning and leveraging predictive world models for high-dimensional domains, generalizing the original DreamerV3 architecture to novel settings and extensions. This paradigm integrates a recurrent state-space model (RSSM) trained from raw sensory sequences, and optimizes policy and value networks entirely with imagined trajectories in compact latent space. The DreamerV3-style methodology targets sample efficiency, robust exploration, and broad applicability across discrete and continuous control, vision-based and proprioceptive environments, and is the foundation for multiple derivatives in modern MBRL research.

1. World Model Architecture and Training Objectives

DreamerV3 and its derivatives construct the agent’s world model as a learned generative and predictive architecture:

• Recurrent State-Space Model (RSSM): Maintains a deterministic hidden state $h_t$ (typically a GRU or LSTM cell) and a stochastic latent state $z_t$, which may be categorical or Gaussian. The sequence model is trained to capture transitions $h_t = \mathrm{GRU}(h_{t-1}, z_{t-1}, a_{t-1})$ and stochastic updates $z_t \sim q_\phi(z_t \mid h_t, o_t)$ with a prior $p_\phi(z_t \mid h_t)$ (Hafner et al., 2023).
• Observation and Reward Heads: The model reconstructs raw observations $o_t \sim p_\phi(o_t \mid h_t, z_t)$, predicts rewards $r_t \sim p_\phi(r_t \mid h_t, z_t)$, and episode continuation $c_t \sim p_\phi(c_t \mid h_t, z_t)$.
• Training Objective: The primary loss is a variational ELBO, decomposed into reconstruction, reward prediction, and dual KL divergences with "free-bits" clipping to prevent posterior collapse:

$$L_\text{world} = L_\text{pred} + \beta_\text{dyn} L_\text{dyn} + \beta_\text{rep} L_\text{rep},$$

where the losses cover predictive (likelihood), dynamics (KL from posterior to prior), and representation (KL from prior to posterior) terms (Hafner et al., 2023, Samsami et al., 2024, Khanzada et al., 7 Mar 2025).

Multiple extensions modify this core module, such as (i) ensemble world models for epistemic exploration (Khanzada et al., 7 Mar 2025, Khanzada et al., 21 Dec 2025), (ii) dynamic modulation for extracting motion cues (Zhang et al., 29 Sep 2025), (iii) spatio-temporal masking for robust representation (Sun et al., 2024), (iv) implicit models without pixel reconstruction (Burchi et al., 2024, Narendra et al., 26 Jun 2025), or (v) transformer-based memory (Chen et al., 2022).

2. Policy Optimization via Imagination in Latent Space

Policy and value optimization in DreamerV3-style agents is performed by imagination-driven actor-critic updates:

• Imaginative Rollouts: Starting from real or posterior latent states, the world model is used to generate $H$-step imagined rollouts in the latent space $(h_t, z_t)$ (Hafner et al., 2023). The action distribution $\pi_\theta(a_t \mid h_t, z_t)$ samples actions, which are fed into the recurrent world model.
• Critic (Value) Update: Targets are constructed as symlog-transformed $\lambda$-returns over the imagined trajectory:

$$R_t^{(\lambda)} = \sum_{n=0}^{H-1} \left( \prod_{i=0}^{n-1} \gamma_{t+i} \right) r_{t+n} + \left( \prod_{i=0}^{H-1} \gamma_{t+i} \right) V_\psi(z_{t+H}, h_{t+H}).$$

The critic is trained using distributional regression, usually with a two-hot target encoding (Hafner et al., 2023).

• Actor Update: The policy maximizes discounted imagined return plus a fixed entropy regularizer:

$$\mathcal{L}_\pi = -\mathbb{E} \left[ \sum_{n=0}^{H-1} \left( \prod_{i=0}^{n-1} \gamma_{t+i} \right) r_{t+n} \right] + \eta \, \mathbb{E} \left[ \text{KL} \left[ \pi_\theta(\cdot \mid z_t, h_t) \| \mathcal{N}(0, I) \right] \right]$$

(Khanzada et al., 7 Mar 2025).

Return scaling, free-bit KL, and normalization techniques enhance stability and domain generality (Hafner et al., 2023).

3. Intrinsic Motivation and World Model Uncertainty

Recent DreamerV3-style agents incorporate advanced exploration strategies to overcome reward sparsity and enable task-agnostic pretraining:

• Latent Disagreement Exploration: An ensemble of $K$ world models predicts the next latent; variance across ensemble predictions provides an intrinsic reward signal:

$$r_t^{\text{int}} = \frac{1}{K} \sum_{k=1}^K \| \mu_k(z_t, a_t) - \bar{\mu} \|^2, \quad \bar{\mu} = \frac{1}{K} \sum_{k=1}^K \mu_k(z_t, a_t)$$

Used for reward-free exploration, curriculum learning, and transfer (Khanzada et al., 7 Mar 2025, Khanzada et al., 21 Dec 2025).

• Hybrid Exploration (M3PO): Combines model-based planning with model-free uncertainty-bonus, using the discrepancy between model-based and model-free value estimation for exploration-augmented advantage:

$$\tilde{A}_t = \hat{A}_t + \alpha (\epsilon_t - \bar{\epsilon}), \;\; \epsilon_t = | Q^{MB}(z_t, a_t) - Q^{MF}(z_t, a_t) |$$

(Narendra et al., 26 Jun 2025).

• Distraction-Robust Exploration: Dynamic modulation (Zhang et al., 29 Sep 2025), spatio-temporal masking (Sun et al., 2024), and adversarial, object-based reconstruction weighting (Hutson et al., 2024) focus capacity on salient, task-relevant state changes and filter spurious observation correlations.

4. Specialized Model Architectures and Representation Learning

Several DreamerV3-style derivatives implement specialized world model architectures to target bottlenecks in generalization, expressivity, and sample efficiency:

• Transformer State-Space Models: TransDreamer replaces the RSSM with a transformer, providing longer temporal context and improved memory for sequential tasks (Chen et al., 2022).
• Implicit/Decoder-Free Models: MuDreamer (Burchi et al., 2024), M3PO (Narendra et al., 26 Jun 2025) eliminate pixel reconstruction—focusing all modeling capacity on next latent and reward prediction, improving performance under complex backgrounds and distraction.
• Kolmogorov–Arnold Networks (KANs): KAN-Dreamer replaces MLP/CNN blocks with KAN/FastKAN layers, analyzing the impact of univariate spline-based networks on sample efficiency, throughput, and interpretability (Shi et al., 8 Dec 2025).
• Dynamic/Distraction Robustness: DyMoDreamer augments the RSSM with dynamic modulator latents derived from inter-frame differencing, modeling object-level motion and changes (Zhang et al., 29 Sep 2025). HRSSM employs parallel mask and raw branches, bisimulation-based similarity loss, and spatio-temporal masking for robust task-relevant encoding (Sun et al., 2024).

5. Training Protocols, Hyperparameters, and Practical Guidelines

DreamerV3-style MBRL agents benefit from standardized and robust training routines:

• Replay Buffer and Sampling: Large FIFO replay buffers with uniform or prioritized sampling enable robust high-diversity sequence learning (Hafner et al., 2023, Chen et al., 2022).
• Hyperparameters: Common defaults include batch size 16–128, sequence length 50–64, learning rates $10^{-4}$–$4 \times 10^{-5}$, imagination horizon $H=15$, KL scaling factors, and free-bit thresholds (Hafner et al., 2023, Khanzada et al., 7 Mar 2025, Burchi et al., 2024).
• Optimization: Adam or LaProp with gradient clipping, normalization (LayerNorm, GroupNorm, RMSNorm), and symlog/percentile normalization of returns. Entropy scales (typically $\eta=3\times10^{-4}$) are domain-invariant.
• Domain Transfer / Fine-Tuning: Zero-shot policy transfer is facilitated by freezing the world model and fine-tuning only actor/critic heads on new tasks or domains; exploration-specific policies (latent disagreement) can remain fixed and yield robust transfer in unseen environments (Khanzada et al., 21 Dec 2025, Khanzada et al., 7 Mar 2025).
• Implementation Notes: For visual domains, convolutional encoders with four layers (channels 32-256) and transposed-CNN decoders; for LIDAR, fully connected MLP-VAEs suffice (Steinmetz et al., 3 Dec 2025). For KAN-FastKAN integration, explicit input clamping and basis selection is critical (Shi et al., 8 Dec 2025).

6. Empirical Performance and Benchmarks

DreamerV3-style agents define state-of-the-art in a variety of benchmarks by achieving high sample efficiency, robustness to distractions, and adaptability to new domains:

Environment / Task	DreamerV3	Variant(s) / Extension	Performance Effect	Reference
DeepMind Control Suite	Baseline	DyMoDreamer, MuDreamer	832 (+9.5%), 849.6 median	(Zhang et al., 29 Sep 2025, Burchi et al., 2024)
Atari 100k	Baseline	DyMoDreamer	156.6% mean human norm (vs 125%)	(Zhang et al., 29 Sep 2025)
CARLA lane-follow/collision	DreamerV3	InDRiVE (disagreement)	100% lane-follow SR (Town02, 10k steps) vs 64% (DreamerV3)	(Khanzada et al., 7 Mar 2025)
Turtlebot3 LIDAR control	DreamerV3	World-model LIDAR agent	100% mean SR with full LIDAR	(Steinmetz et al., 3 Dec 2025)
Multi-task setting	DreamerV3	M3PO (decoder-free, hybrid exploration)	On-par or higher SOTA, robust transfer	(Narendra et al., 26 Jun 2025)

Ablations confirm that latent disagreement, dynamic modulation, and explicit distraction-robustness provide marked improvements over the DreamerV3 baseline. Exploration bonuses and intrinsic rewards increase sample efficiency and out-of-distribution generalization in both synthetic and real-world settings.

7. Future Directions and Open Research

Current research seeks to extend DreamerV3-style agents in several directions:

• Better long-term memory: Alternative state-space models and transformer-based components (TransDreamer, R2I) address memory bottlenecks and credit assignment across long horizons (Chen et al., 2022, Samsami et al., 2024).
• Distraction/disentanglement: Masking, explicit dynamic object modeling, and policy-shaped reconstruction guide representation learning away from irrelevant but predictive content (Zhang et al., 29 Sep 2025, Hutson et al., 2024, Sun et al., 2024).
• Implicit and symbolic models: KAN/FastKAN and decoder-free world models push toward efficient, interpretable, and low-overhead latent modeling (Shi et al., 8 Dec 2025, Burchi et al., 2024).
• Intrinsic reward pretraining and scalability: Intrinsic disagreement and uncertainty-driven exploration enable task-agnostic pretraining and rapid adaptation (Khanzada et al., 21 Dec 2025, Khanzada et al., 7 Mar 2025, Narendra et al., 26 Jun 2025).
• Hybrid planning and on-policy learning: Integration of model-based planning via short-term MPC and on-policy trust-region optimization for stable training and multi-task robustness (Narendra et al., 26 Jun 2025).

Taken together, the DreamerV3-style agent paradigm provides a unified, general, and extensible blueprint for model-based RL at scale, serving as a foundation for ongoing advances in sample-efficient, robust, and adaptive control (Hafner et al., 2023, Khanzada et al., 7 Mar 2025, Zhang et al., 29 Sep 2025, Khanzada et al., 21 Dec 2025, Narendra et al., 26 Jun 2025, Burchi et al., 2024).