Optimal Objective for Representation Learning in Image-Based World Models

Determine the optimal objective function for learning latent state representations in image-based model-based reinforcement learning world models, such as those based on the Recurrent State-Space Model (RSSM), so that the learned representations emphasize task-essential information while avoiding overfitting to irrelevant visual details.

Background

The paper highlights that, despite the success of architectures like the Recurrent State-Space Model (RSSM) in world-model-based reinforcement learning, the choice of objective for learning latent representations from images is not settled. In high-dimensional visual settings, representation learning is especially challenging, making the selection of an effective objective crucial.

Decoder-based approaches optimize pixel-level reconstruction, which can overemphasize large but task-irrelevant background regions. Decoder-free approaches typically rely on data augmentation as an external regularizer to prevent representation collapse, but augmentation choices can distort task-critical information. These limitations motivate the explicit open question about what objective should guide representation learning in image-based world models.

References

While architectures like the Recurrent State-Space Model (RSSM) have achieved remarkable success (Hafner et al., 2025), a fundamental question remains open: What is the optimal objective function for learning the representation itself?

— R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation  (2603.18202 - Morihira et al., 18 Mar 2026) in Section 1 (Introduction)

Whether training against a reachability-aware criterion would produce a representation whose plain L2 is usable is an open and more interesting question.

— The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use  (2608.12959 - Singh, 13 Aug 2026) in Section 9, Limitations

Finally, we theoretically analyze the architecture's optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.

— NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games  (2609.01549 - Holeček et al., 1 Sep 2026) in Abstract; Section 3.4, “Model Convergence”; Section 5, “Conclusion”