Constructing a world model that fully clones deterministic 3D environments

Construct a world model that, trained purely from first-person observation–action trajectories, can fully clone and overfit a fixed deterministic 3D environment by reproducing its unique ground-truth sequence of observations for any given action sequence over long horizons.

Background

The paper argues that reliable long-horizon prediction in deterministic 3D environments is primarily bottlenecked by representation quality rather than dynamics modeling. It introduces Geometrically-Regularized World Models (GRWM) to improve latent spaces so that they align with the true topology of the environment, thereby enhancing rollout fidelity. Despite proposing GRWM as a step forward, the authors explicitly identify the broader task of building a model that can fully clone and overfit a deterministic 3D world as an open problem.

Deterministic environments here mean that for any initial state and sequence of actions, there exists exactly one resulting sequence of observations. The authors measure fidelity via frame-wise MSE between predicted and ground-truth observations over long horizons, and show that using true underlying states enables near-perfect predictions, underscoring the centrality of representation learning for solving this open problem.