DyMoDreamer: Dynamic Modulation for Visual RL

• DyMoDreamer is a model-based RL algorithm that augments DreamerV3 with a dynamic modulation mechanism focused on reward-relevant motion.
• The method employs differential masking of inter-frame changes to isolate moving objects, thereby enhancing temporal credit assignment and sample efficiency.
• Benchmark evaluations on Atari, DeepMind Visual Control Suite, and Crafter show notable improvements over standard DreamerV3 performance.

DyMoDreamer is a model-based reinforcement learning algorithm for visual control that augments a DreamerV3-style recurrent state-space model with a dynamic modulation mechanism specialized for reward-relevant motion. Its central design is to construct differential observations by masking inter-frame changes, infer a discrete dynamic modulator $d_t$ from those masked inputs, and use $d_t$ alongside the conventional stochastic latent $z_t$ in the transition and prediction heads. The reported outcome is improved sample efficiency on visual RL benchmarks, including a $156.6\%$ mean human-normalized score on Atari $100$k, a mean score of $832$ on the DeepMind Visual Control Suite, and a $9.5\%$ improvement over DreamerV3 on Crafter after $1$M steps (Zhang et al., 29 Sep 2025). A separate 2026 paper uses the name “Dreamer” for a depth-recurrent attention-mixture framework in language modeling and explicitly states that “DyMoDreamer” is not a separate model name there; that alternate usage is best treated as a nomenclature issue rather than as the same method (Knupp et al., 29 Jan 2026).

1. Motivation and conceptual basis

DyMoDreamer is motivated by a limitation of standard Dreamer-style world models: they process observations holistically, and the visual encoder and RSSM optimize reconstruction and reward or continuation prediction on all pixels. In the formulation reported for DyMoDreamer, this dilutes modeling capacity across large static backgrounds and nuisance visual details, even though the reward signal is often driven by small, fast-moving objects and their immediate local context, such as the ball and paddle in Pong or fists in Boxing (Zhang et al., 29 Sep 2025).

The method is designed around the claim that holistic modeling can exacerbate compounding model errors in imagined rollouts precisely on the dynamic cues that matter most for control. The details further note that VAE stochasticity tends to blur small moving objects, which degrades temporal credit assignment. DyMoDreamer addresses this by introducing a second discrete latent pathway dedicated to reward-relevant dynamics, while keeping the original stochastic representation $z_t$ to capture static context and non-dynamic task factors. This suggests a division of labor between latents: $z_t$ remains responsible for broad scene content, whereas $d_t$0 emphasizes dynamic visual features and their temporal evolution.

Relative to Dreamer, DreamerV2, and DreamerV3, the method builds on DreamerV3’s RSSM but adds a second discrete latent head for dynamics, conditions the GRU transition on $d_t$1, and adds both a differential reconstruction regularizer and losses for $d_t$2. Relative to object-centric methods such as OC-STORM, it does not require pre-trained segmentation or prior object counts. Relative to transformer or tokenized world models such as IRIS, $d_t$3-IRIS, STORM, and OC-STORM, it keeps the RSSM architecture lightweight and targets reward-relevant motion directly through temporal differencing and modulation. Relative to latent-difference approaches in model-free RL such as Latent Flow, it performs differencing in pixel space rather than using $d_t$4, with the stated goal of avoiding precision loss and missed small-object motions.

2. Differential observations and inter-frame masking

The differential-observation pathway begins from the raw image observation $d_t$5 and a temporal interval $d_t$6, with $d_t$7 reported as the empirical default. DyMoDreamer defines a vanilla binary differencing tensor $d_t$8 elementwise by thresholding the inter-frame difference with $d_t$9:

$z_t$0

Because $z_t$1 is often sparse, the method expands active regions by spatial convolution or dilation with a binary kernel $z_t$2 and threshold $z_t$3:

$z_t$4

The masked differential observation is then defined as

$z_t$5

This suppresses the static background while preserving, and locally expanding, motion-related regions (Zhang et al., 29 Sep 2025).

The functional role of this construction is explicit. A stochastic encoder receives $z_t$6 and produces $z_t$7, whereas a dynamic encoder receives $z_t$8 and produces $z_t$9. Since $156.6\%$0 is masked, the dynamic encoder primarily sees moving objects and their immediate context. The details characterize this as an online operation that explicitly encodes object-level motion cues and temporal transitions. The mask itself is deterministic; gradients flow through $156.6\%$1 to the dynamic encoder, but not through the hard threshold.

The sensitivity discussion places this mechanism within a concrete operating regime. The threshold $156.6\%$2, the dilation kernel size, and the differencing interval $156.6\%$3 trade sensitivity against noise. The data note that longer intervals $156.6\%$4 can suppress static flashes but may miss fast dynamics. A plausible implication is that the masking stage is both the source of DyMoDreamer’s inductive bias and one of its main hyperparameter sensitivities.

3. Dynamic modulation inside the RSSM

DyMoDreamer uses discrete latent variables for both the ordinary representation and the dynamic modulator. The representation latent and modulation latent are sampled as

$156.6\%$5

Both $156.6\%$6 and $156.6\%$7 are modeled as products of $156.6\%$8 independent categorical variables over $156.6\%$9 classes. In the default setting, $100$0 and $100$1 for both pathways. The encoder outputs groupwise logits, and sampling uses the straight-through estimator for discrete latents.

The recurrent transition is modulated directly by the previous dynamic latent:

$100$2

where $100$3 is a GRU. Priors over the representation and modulation latents are produced from $100$4,

$100$5

and the predictive heads are conditioned on the concatenated state $100$6:

$100$7

The implementation description characterizes $100$8 as a feature-wise modulator of both the transition and the predictive heads, implemented through concatenation or gating in the inputs (Zhang et al., 29 Sep 2025).

Architecturally, the world model contains two CNN VAEs with LayerNorm and SiLU, a GRU sequence model, MLP prior heads, a shared decoder, a reward predictor trained with symlog two-hot classification, and a continuation predictor trained with cross-entropy. The stated computational profile is close to DreamerV3 in parameters and FLOPs, with the added heads for $100$9 and the mask or regularizer computation contributing only small additional compute. The details further state that inter-frame differencing plus dilation is negligible compared to the CNN and GRU forward, and that no external vision model or optical flow is used.

A recurrent misconception is that differential observations alone explain the method’s gains. The ablation summary argues against that interpretation: removing dynamic modulation from the RSSM while still encoding differential observations causes a large drop, indicating that the integration of $832$0 into the transition and predictors is essential rather than incidental.

4. Objective function and end-to-end training procedure

The world-model loss combines prediction, dynamics KL, representation KL, and a differential divergence regularizer:

$832$1

with $832$2 and $832$3. The prediction term decomposes into reconstruction, reward, and continuation losses. The dynamics KL and representation KL are both clipped with free-bits at $832$4 nat and are defined separately for $832$5 and $832$6, with stop-gradient placed on opposite sides in the two terms to train prior prediction and posterior consistency respectively (Zhang et al., 29 Sep 2025).

A distinctive component is the differential divergence regularizer. Using temporal differences

$832$7

DyMoDreamer forms global softmax distributions over all pixels and channels using temperature $832$8, then minimizes a KL divergence between the reconstructed and ground-truth inter-frame change distributions. The stated role of this term is to align inter-frame change distributions of reconstructions to ground truth and to complement MSE-like intra-frame losses.

Policy learning follows DreamerV3 on imagined trajectories using model states $832$9. The critic uses an EMA target with decay $9.5\%$0, the actor uses a REINFORCE surrogate with entropy regularization, the imagination horizon is $9.5\%$1, the return uses $9.5\%$2, and the entropy scale is $9.5\%$3. The normalization term $9.5\%$4 is defined as the $9.5\%$5th–$9.5\%$6th percentile range of $9.5\%$7 in the batch. Imagined rollouts begin from posterior states and use the prior transition under the learned policy. Because the differential observation uses backward differencing, a random action is sampled at the first frame to seed $9.5\%$8 when $9.5\%$9.

The training loop is specified as a seven-stage pipeline: data collection; masked differencing; encoding of $1$0 and $1$1; RSSM update and prior prediction; world-model optimization; policy and value training on imagined rollouts; and repetition. Reproducibility details include replay sequences of length $1$2, batch size $1$3, world-model Adam with learning rate $1$4 and $1$5, actor-critic Adam with learning rate $1$6 and $1$7, gradient clip $1$8, $1$9, and discount horizon $z_t$0. On NVIDIA RTX 4090 with $z_t$1 CPU cores and $z_t$2 GB RAM, one Atari $z_t$3k run reportedly took approximately $z_t$4 hours in JAX.

5. Reported benchmark performance and ablation findings

The empirical evaluation covers Atari $z_t$5k, the DeepMind Visual Control Suite, and Crafter. The reported headline numbers are organized below (Zhang et al., 29 Sep 2025).

Benchmark	Reported DyMoDreamer result	Reported comparison points
Atari $z_t$6k (26 games)	Mean human-normalized score $z_t$7; median HNS $z_t$8	DreamerV3 $z_t$9; OC-STORM $z_t$0; DIAMOND $z_t$1
DeepMind Visual Control Suite (20 tasks, 1M steps, visual)	Mean score $z_t$2	DreamerV3 $z_t$3; TWISTER $z_t$4; TD-MPC2 $z_t$5
Crafter (1M steps)	Return $z_t$6	DreamerV3 $z_t$7; IRIS $z_t$8; $z_t$9-IRIS $d_t$00

The reported Atari improvements are especially pronounced in sparse small-object dynamics and high-speed small-object tasks, including Pong $d_t$01, Boxing $d_t$02, Krull $d_t$03, and Road Runner $d_t$04. On the DeepMind Visual Control Suite, the paper highlights tasks with sparse or localized moving objects such as Cheetah Run $d_t$05, Quadruped Run $d_t$06, and Finger Spin $d_t$07. On Crafter, the reported gain over DreamerV3 is $d_t$08, with the explanation that differential observations capture egocentric relative motion between the agent and environment components.

The ablation study isolates which components matter. Removing dynamic modulation from the RSSM while keeping differential observations yields a large drop, with Boxing decreasing from $d_t$09 to $d_t$10 and Road Runner from $d_t$11 to $d_t$12. Removing the differential divergence regularizer causes a moderate drop, such as Krull from $d_t$13 to $d_t$14 and Pong from $d_t$15 to $d_t$16, but the resulting model still remains above DreamerV3. Increasing the size of $d_t$17 without modulation to $d_t$18 groups and classes does not match the benefits of modulation, with Boxing $d_t$19, Krull $d_t$20, and Road Runner $d_t$21. Replacing $d_t$22 with latent differencing $d_t$23 underperforms, especially on tasks dominated by small objects, with Road Runner $d_t$24. Lowering the $d_t$25 dimension to $d_t$26 degrades performance, and using $d_t$27 helps smooth dynamics but can miss fast actions such as those in Boxing.

The qualitative analyses are consistent with the benchmark results. The reported differential masks tightly cover moving objects, and the dilated masks produce contiguous motion regions for small objects. Imagined reconstructions are said to exhibit substantially fewer hallucinations on dynamic patterns than DreamerV3, focusing on reward-relevant moving entities while reconstructing static backgrounds adequately via $d_t$28. With an auxiliary split decoder, an emergent specialization is observed in which $d_t$29 focuses on controllable dynamics and $d_t$30 captures static context and passive motion, despite the absence of explicit targets for that separation.

6. Relations, limitations, and terminological ambiguity

Within model-based visual RL, DyMoDreamer is positioned as an RSSM-based alternative to heavier object-centric or transformer-based approaches. Unlike OC-STORM, it does not require pre-trained segmentation or prior object counts. Unlike IRIS, $d_t$31-IRIS, STORM, and OC-STORM, it does not rely on larger transformer or token-based vision modules. Its stated contribution is instead to preserve the efficiency and simplicity of RSSM world models while explicitly extracting and modeling dynamic visual cues through differential masking and a discrete modulation latent (Zhang et al., 29 Sep 2025).

The method is reported to help most in tasks where rewards depend on sparse moving objects or localized interactions, in multi-phase tasks with distinct dynamics, and in continuous control from pixels where localized motion dominates return shaping. The limitations are correspondingly specific. In mostly static environments with negligible motion, or in settings where reward is tied to global appearance, the modulation mechanism may add little. Extremely subtle dynamics below the $d_t$32 threshold may require tuning $d_t$33, the dilation kernel, or multi-frame logical differencing. Noisy visuals or camera shake can cause naive differencing to highlight noise; the stated mitigation strategies include moving-average differencing and logical AND over frames. Proposed extensions include optical flow or event-based sensors, object-centric latents combined with modulation without pre-trained segmentation, and improved temporal credit assignment via hierarchical time scales or memory-augmented sequence models.

A separate point of clarification concerns the name itself. The 2026 paper “Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves” consistently uses the name “Dreamer” for a language-model architecture that combines sequence attention, depth attention, and sparse expert attention inside a depth-recurrent layer. That paper explicitly states that “DyMoDreamer” is not a separate model name in the paper; if the term appears in that context, it is best interpreted as the Dreamer framework instantiated with dynamic sparse expert attention together with depth recurrence, sequence attention, and depth attention (Knupp et al., 29 Jan 2026). The two usages therefore refer to distinct technical objects: one is a visual RL world model with dynamic modulation over differential observations, and the other is a depth-recurrent attention-mixture framework for latent reasoning in language modeling.

This naming ambiguity can obscure substantive differences. The RL DyMoDreamer centers on masked inter-frame differencing, a dynamic encoder, and modulation of an RSSM world model; the language-model Dreamer centers on depth recurrence, depth attention, and sparse expert attention. The overlap is lexical rather than methodological.