R2-Dreamer: Redundancy-Reduced World Models

• The paper introduces a novel decoder-free world model that eliminates pixel reconstruction in favor of an internal redundancy-reduction objective.
• It leverages a recurrent state-space model from the Dreamer lineage to generate latent trajectories for policy and value learning in visual control tasks.
• Empirical results show competitive performance on DeepMind Control Suite and Meta-World, with faster computation and enhanced focus on small, task-critical features.

R2-Dreamer is a decoder-free, augmentation-free world model for visual control introduced in "R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation" (Morihira et al., 18 Mar 2026). It is a model-based reinforcement learning method built on the Dreamer lineage of latent imagination, in which a recurrent state-space model (RSSM) supports policy and value learning from imagined latent trajectories rather than direct planning in pixel space (Hafner et al., 2019). Its defining modification is to remove pixel reconstruction and replace it with an internal redundancy-reduction objective that aligns the latent world state with the encoder embedding while decorrelating features. The stated objective is to learn compact, task-relevant latent representations without allocating capacity to large task-irrelevant image regions and without relying on external data augmentation; empirically, the method is reported as competitive with strong baselines on DeepMind Control Suite and Meta-World, substantially stronger on DMC-Subtle, and 1.59x faster than DreamerV3 in a reported wall-clock comparison on Walker Walk (Morihira et al., 18 Mar 2026).

1. Problem setting and historical placement

R2-Dreamer addresses image-based model-based reinforcement learning, where the central representational difficulty is to separate task-relevant information from irrelevant visual detail. In reconstruction-based world models, including Dreamer variants, the optimization target includes pixel-level likelihood. The data provided for R2-Dreamer states that this causes gradients to be dominated by large but task-irrelevant regions such as backgrounds, which incentivizes accurate reconstruction of irrelevant detail and diverts capacity away from small but crucial features. The same source identifies a second limitation in decoder-free methods: many avoid pixel reconstruction only by relying on external data augmentation, such as random shifts, to avert collapse and induce invariance. Those augmentation choices are described as task-dependent and potentially destructive when small objects or meaningful color cues are task-critical (Morihira et al., 18 Mar 2026).

Within the Dreamer family, the historical baseline is the original Dreamer, which established the latent-imagination paradigm: a learned RSSM summarizes experience, imagined latent rollouts are generated under the learned dynamics, and actor-critic learning proceeds entirely in latent space (Hafner et al., 2019). R2-Dreamer retains that overall style of control, but changes what the world model is asked to preserve. Instead of preserving visual fidelity through a decoder, it preserves information by coupling the encoder representation to the latent state via a Barlow Twins-style regularizer. A common misconception is that removing the decoder makes the system model-free; in fact, R2-Dreamer remains a full world-model method with an RSSM, reward predictor, continuation predictor, imagination rollouts, and actor-critic training.

2. Architecture and redundancy-reduction mechanism

R2-Dreamer adheres to DreamerV3’s architecture and training components except for eliminating the image decoder. Images $o_t$ are mapped by an encoder $f_o$ to embeddings

$e_t = f_o(o_t).$

The latent state is

$s_t = (h_t, z_t),$

with deterministic recurrent state $h_t$ and stochastic state $z_t$, updated through the standard RSSM factorization:

$$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$

The world model also includes a reward predictor $p_r(r_t \mid s_t)$ and a continuation predictor $p_c(c_t \mid s_t)$, where continuation is defined as $1$ minus termination probability. A lightweight linear projector $f_o$0 maps the latent state into the encoder feature space,

$f_o$1

The central innovation is the redundancy-reduction loss between $f_o$2 and $f_o$3. Over $f_o$4 samples, features are standardized along the batch dimension and assembled into a cross-correlation matrix

$f_o$5

The loss is

$f_o$6

with default redundancy weight $f_o$7. The two “views” required by the Barlow Twins-style objective are not generated by data augmentation; they are internal signals, namely the encoder embedding $f_o$8 and the projected latent state $f_o$9. For stability, $e_t = f_o(o_t).$0 is detached when computing $e_t = f_o(o_t).$1, while the encoder still receives gradients through the posterior and downstream task losses.

The diagonal term of $e_t = f_o(o_t).$2 pushes $e_t = f_o(o_t).$3 to predict $e_t = f_o(o_t).$4, which the source characterizes as a surrogate for maximizing $e_t = f_o(o_t).$5. The off-diagonal penalties decorrelate features, reduce total correlation, and prevent collapse. The intended consequence is a compact, factorized latent representation focused on informative dimensions rather than exhaustive pixel reconstruction (Morihira et al., 18 Mar 2026).

3. Training objective and latent-imagination pipeline

The training pipeline retains DreamerV3’s high-level structure: data collection, world model learning, imagination, and policy/value optimization. Interaction with the environment populates a replay buffer with tuples $e_t = f_o(o_t).$6. Replay segments are encoded, posterior latent states are inferred with the RSSM, and the world model is updated using reward prediction, continuation prediction, KL regularization, and redundancy reduction. Imagined trajectories are then unrolled from posterior states under the learned dynamics and current policy, and the actor and critic are optimized from replay and imagination (Morihira et al., 18 Mar 2026).

The world-model loss replaces reconstruction with $e_t = f_o(o_t).$7:

$e_t = f_o(o_t).$8

where

$e_t = f_o(o_t).$9

with coefficients $s_t = (h_t, z_t),$0, $s_t = (h_t, z_t),$1, and $s_t = (h_t, z_t),$2. KL balancing is implemented with free bits and stop-gradient asymmetry:

$s_t = (h_t, z_t),$3

$s_t = (h_t, z_t),$4

Policy/value learning follows the imagination-based recipe characteristic of Dreamer-style methods, but with the specific losses reported for R2-Dreamer. Imagined rollouts begin from posterior states inferred on replayed sequences and evolve under the learned dynamics. The return target is the $s_t = (h_t, z_t),$5-return

$s_t = (h_t, z_t),$6

with $s_t = (h_t, z_t),$7. The critic is trained on imagined and replay segments via

$s_t = (h_t, z_t),$8

where the target/EMA regularizer uses decay $s_t = (h_t, z_t),$9. The actor is trained on imagined trajectories only:

$h_t$0

with entropy coefficient $h_t$1 and dynamic return normalization scale

$h_t$2

Optimization and stabilization choices are specified as follows: LaProp with $h_t$3 and learning rate $h_t$4, Adaptive Gradient Clipping with threshold $h_t$5, RMSNorm and SiLU throughout, a $h_t$6 uniform mixture in latent and actor heads, and free bits thresholded at $h_t$7 nat. The default batch size is $h_t$8, with an ablation at $h_t$9 (Morihira et al., 18 Mar 2026).

4. Empirical results and benchmark profile

The reported evaluation covers three benchmark families. The first is the DeepMind Control Suite, with 20 pixel-based continuous-control tasks spanning locomotion and manipulation. The second is Meta-World MT1, with 50 single-task robotic manipulation benchmarks. The third is DMC-Subtle, introduced specifically to stress representational precision by shrinking task-critical objects in five DMC tasks; examples given include scaling the Reacher target to $z_t$0 and the Cartpole pole width to $z_t$1. The protocol is to train for up to $z_t$2M environment steps, evaluate with 10 episodes per seed, and use 5 seeds per method in a unified PyTorch codebase for fair comparisons across decoder-free variants (Morihira et al., 18 Mar 2026).

On DMC, R2-Dreamer is reported as competitive with DreamerV3, DreamerPro, TD-MPC2, and DrQ-v2 in mean and median returns across tasks. On Meta-World MT1, it achieves competitive mean and median success rates and is described as robust on contact-rich manipulation with small objects. On DMC-Subtle, the paper reports substantial gains over baselines and states that R2-Dreamer consistently outperforms decoder-based and data-augmentation-reliant agents, which is presented as confirmation that the method better preserves subtle, task-critical cues.

The wall-clock comparison given for DMC Walker Walk at $z_t$3M steps on a single RTX 3080 Ti is:

• R2-Dreamer: $z_t$4h
• Dreamer: $z_t$5h
• DreamerPro: $z_t$6h
• Dreamer (author’s JAX): $z_t$7h

The paper summarizes this as a 1.59x speedup over the authors’ DreamerV3 reproduction and attributes the gain to the removal of pixel reconstruction and data-augmentation processing. This suggests that, in the reported setting, the cost of computing the cross-correlation objective is materially smaller than the cost of image decoding (Morihira et al., 18 Mar 2026).

5. Ablations, analysis, and theoretical interpretation

The ablation results are organized around dependence on data augmentation, robustness to batch size, and representational focus. The paper states that DreamerPro collapses without data augmentation, whereas R2-Dreamer remains stable and strong without it. Adding data augmentation to R2-Dreamer yields only marginal gains on standard DMC and harms performance on DMC-Subtle, where preserving small features is critical. Halving the batch size from $z_t$8 to $z_t$9 barely affects performance, a result the authors describe as consistent with Barlow Twins’ robustness in correlation estimation. Qualitative occlusion-based saliency on DMC-Subtle Reacher shows R2-Dreamer’s policy sharply focusing on the tiny target, while baselines exhibit diffuse attention (Morihira et al., 18 Mar 2026).

The theoretical framing given in the source interprets the loss as optimizing a variational bound on an extended Sequential Information Bottleneck:

$$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$0

In that interpretation, fidelity is approximated by $$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$1, temporal compression by the KL between posterior and prior, and spatial compression by reducing total correlation through the off-diagonal penalties in the Barlow Twins-style loss. This does not constitute a proof of optimality in the provided material, but it offers a principled description of why the method may preserve task-relevant information while compressing redundant dimensions.

R2-Dreamer is also situated relative to several self-supervised learning families. The paper directly connects it to Barlow Twins through feature decorrelation via cross-correlation, and places it alongside VICReg and non-contrastive methods such as BYOL and SimSiam. Its distinguishing move is to adapt redundancy reduction to the RSSM setting by pairing encoder embeddings with projected latent states, thereby constructing internal views without external augmentation (Morihira et al., 18 Mar 2026).

6. Relation to other methods, limitations, and scope of the name

Relative to reconstruction-based Dreamer variants, R2-Dreamer eliminates the image decoder and its negative log-likelihood reconstruction term, while retaining the Dreamer-style RSSM and latent imagination pipeline. Relative to decoder-free methods such as DreamerPro, TD-MPC2, and DrQ-v2, its central distinction is that collapse prevention and invariance are driven by an internal regularizer rather than data augmentation. In this sense, R2-Dreamer can be read as a representational re-specification of the Dreamer program: it preserves the core imagination-based control architecture established by Dreamer, but changes the world-model target from visual fidelity to redundancy-reduced latent sufficiency (Hafner et al., 2019).

The limitations described in the source are concrete. First, augmentation can still be beneficial under certain conditions, such as robustness to camera jitter or strong domain shifts; evaluation on Distracting Control Suite is proposed. Second, the redundancy-reduction term requires computing a $$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$2 cross-correlation matrix per batch, so the overhead is non-zero even if smaller than decoding at moderate feature dimension and batch size. Third, although the method is reported as robust to halving batch size, extremely small batches may degrade correlation estimates, and feature dimension choices and projector capacity may affect performance. Fourth, scaling to very high-dimensional control, partially observable settings, or multi-camera setups is identified as an open test of generality. Practical guidance in the source recommends the default hyperparameters, batch size at least $$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$3, avoiding external data augmentation by default, and monitoring $$h_t = f_\theta(h_{t-1}, a_{t-1}), \qquad p_\phi(z_t \mid h_t), \qquad q_\psi(z_t \mid h_t, e_t).$$4 for feature collapse (Morihira et al., 18 Mar 2026).

The name should be interpreted narrowly. "R2-Dreamer" in this sense denotes the reinforcement-learning method of (Morihira et al., 18 Mar 2026), not the unrelated RF imaging system "Dreamer: Dual-RIS-aided Imager in Complementary Modes" (Wang et al., 2024), and not "R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies" (Xu et al., 15 Jun 2026). Those works share lexical overlap in title but belong to different technical domains.