---
title: 'DINO-WM: Zero-Shot Latent World Model'
url: https://www.emergentmind.com/topics/dino-wm
type: topic
---

# DINO-WM: Zero-Shot Latent World Model

DINO-WM (DINO World Model) is a world modeling framework that leverages spatial patch features from the pre-trained DINOv2 vision transformer to enable zero-shot planning in complex visual environments. Unlike classical world models that rely on generative pixel reconstructions or task-specific latent representations, DINO-WM operates fully in latent space, eschewing pixel reconstruction in favor of direct prediction of future DINOv2 patch embeddings. This paradigm facilitates offline training on diverse, pre-recorded trajectories and supports task-agnostic, zero-shot planning through direct optimization of action sequences to reach a desired visual goal [2411.04983].

## 1. Motivation and Core Principles

Learning accurate visual world models for control from pixels remains a significant challenge, primarily due to two limitations: the computational burden of pixel-level prediction (necessitating generative or diffusion models), and the lack of generality in existing latent-space approaches that entangle representation learning with task or reconstruction objectives. DINO-WM addresses these by using a frozen, spatially structured DINOv2 encoder, which provides object-centric, spatially localized patch embeddings learned from web-scale datasets. This decoupling allows world modeling to focus purely on dynamics in a compact, robust feature space, enabling efficient offline training from passive data and versatile test-time planning [2411.04983].

Key principles include:
- **Latent-only modeling:** All training and inference occur in DINOv2 patch embedding space; pixel reconstruction is unused except for optional qualitative inspection.
- **Offline, task-agnostic training:** The model is trained purely on trajectories of observations and actions, with no rewards, demonstrations, or inverse models.
- **Zero-shot planning:** At test time, goal-conditioning is achieved by optimizing over the DINO feature space, allowing immediate adaptation to new tasks by specifying different goal observations.

## 2. Model Architecture and Dynamics

DINO-WM formalizes dynamics prediction in partially observable MDPs by encoding each observation $o_t\in \mathbb{R}^{H\times W\times 3}$ as $N$ fixed DINOv2 patch embeddings, $z_t\in\mathbb{R}^{N\times E}$, where $N=(H/P)\times(W/P)$ and $E$ is the DINOv2 embedding dimension (typ. 384 for ViT-S/16). The encoder is frozen throughout.

The transition model is a causal vision transformer (ViT) parameterized by $\theta$, accepting as input the past $H$ DINO-patch latent frames $z_{t-H:t-1}$ and action embeddings $\phi(a_{t-H:t-1})$, producing a prediction of the next patch collection $\hat z_t$. Each action $a_t$ is mapped via a learned MLP $\phi$, and broadcasted to patch dimension. Causality is enforced so that only earlier frames inform the prediction at each timestep:
$$\hat z_t = p_\theta( z_{t-H:t-1},\, \phi(a_{t-H:t-1}) ).$$
An optional decoder $q_\psi$ can be used for qualitative inspection by mapping predicted latents $\hat z_t$ back to the pixel space, but this decoder is never used in either training or test-time planning.

## 3. Training Regimen and Data

Training proceeds fully offline on pre-collected trajectories of the form $\{ (o_0, a_0, o_1, a_1, ... ) \}$ without any reward or expert control signals. DINO-WM trains only the parameters $\theta$ (transformer weights in $p_\theta$) and $\phi$ (the action-to-latent MLP).

Prediction uses teacher forcing over windows of $H+1$ frames. For each such window, the model predicts the next $H$ steps of patch embeddings, supervised via a standard latent-space $L_2$ loss:
$$
\mathcal{L}_{\text{pred}} = \sum_{k=1}^H \| \hat z_{t-H+k} - z_{t-H+k} \|_2^2.
$$
No reconstruction, reward, terminal, or auxiliary losses are used. 
Trajectory collection is environment-specific but always uses random or noisy expert-like behavior, ensuring coverage of varied states (e.g., 2,000 random 50-step PointMaze trajectories, over 18,500 replayed or noisy expert Push-T sequences, and similar volumes for manipulation tasks) [2411.04983].

## 4. Zero-Shot Test-Time Planning

DINO-WM enables zero-shot planning by optimizing action sequences to minimize the $L_2$ distance in DINOv2-patch latent space between the model's roll-out and a goal observation's encoding. Formally, for a given start $o_0$ and goal $o_g$, actions $a_{0:H-1}$ are sought to minimize
$$
\mathcal{C}(a_{0:H-1}) = \| \hat z_H(a_{0:H-1}) - z_g \|_2^2,
$$
where $\hat z_0 = z_0$ and $\hat z_{t+1} = p_\theta( \hat z_{t-H+1:t}, \phi(a_{t-H+1:t}) )$.

The cross-entropy method (CEM) is the preferred optimizer, iteratively updating a Gaussian distribution over action sequences by selecting those with lowest planning cost, computing the mean and covariance, and resampling. A gradient-based shooting method is also possible via backpropagation through the world model, but is outperformed by CEM due to non-smooth latent dynamics.

Planning proceeds in an MPC loop: after executing the first $k$ (typically one) actions of the best sequence, the observation is re-encoded and planning resumes, closing the perception–action loop [2411.04983].

## 5. Empirical Evaluation and Comparison

DINO-WM is evaluated on a suite of 2D/3D environments involving navigation, manipulation, and multi-particle dynamics using 224×224 RGB observations:
- PointMaze (navigation),
- Push-T (block pushing),
- Wall/Two-Room (randomized barriers),
- Rope Manipulation (robotic rope pulling),
- Granular Manipulation (particle aggregation).

Zero-shot performance metrics include Success Rate (SR) on discrete goal tasks and Chamfer Distance (CD) for continuous shape-matching. DINO-WM substantially outperforms competitive baselines including IRIS, DreamerV3, and TD-MPC2. For example, on Push-T, DINO-WM achieves SR = 0.90, compared to IRIS (0.32), DreamerV3 (0.04), and TD-MPC2 (0.00). On Rope Manipulation, DINO-WM yields a CD of 0.41 (lower is better), compared to IRIS (1.11) and DreamerV3 (2.49).

### Zero-Shot Planning Performance

| Model         | PointMaze SR | Push-T SR | Wall SR | Rope CD | Granular CD |
|---------------|--------------|-----------|---------|---------|-------------|
| IRIS          | 0.74         | 0.32      | 0.04    | 1.11    | 0.37        |
| DreamerV3     | 1.00         | 0.04      | 1.00    | 2.49    | 1.05        |
| TD-MPC2       | 0.00         | 0.00      | 0.00    | 2.52    | 1.21        |
| DINO-WM       | 0.98         | 0.90      | 0.96    | 0.41    | 0.26        |

Ablation studies show spatial DINOv2 patch embeddings are critical: replacing them with global R3M, ResNet18, or DINOv2 CLS token leads to significantly degraded performance, especially on manipulation and spatial reasoning tasks [2411.04983].

DINO-WM demonstrates robust generalization to novel configurations, such as new object shapes, wall placements, or reduced particle number, maintaining or exceeding baseline performance (e.g., SR = 0.82 on unseen wall configurations vs DreamerV3 SR = 0.76, IRIS SR = 0.06).

Open-loop latent rollouts, decoded with the optional decoder, exhibit very low LPIPS (Push-T: 0.007; Wall: 0.0016) and high SSIM (Push-T: 0.985; Wall: 0.997), outperforming methods directly trained on pixel reconstruction.

## 6. Insights, Limitations, and Extensions

Several insights emerge from DINO-WM's results:
- The use of a frozen, high-quality spatial encoder (DINOv2 patches) enables the world model to focus exclusively on modeling environment dynamics, eliminating the confounding of feature and generative modeling objectives.
- Causal ViT-based latent predictors, trained entirely offline, are sufficient for precise prediction in domains requiring contact dynamics and complex object interactions.
- Latent-space planning using CEM is both straightforward and effective—no reward shaping or demonstration-derived inverse models are required.

Limitations include:
- The method requires access to aligned action data during offline training; applicability to settings lacking ground-truth actions is limited.
- Gradient-based action optimization may be hampered by latent non-smoothness.
- Current planning is single-level; hierarchical extensions or learning latent cost models for high-level goal following remain open directions.

Comparison to recent related latent world models, notably LaDi-WM [2505.11528], highlights a trend: latent-based dynamics using pre-trained visual feature spaces (e.g., DINO, CLIP/SigLip) substantially improve generalization and sample efficiency. LaDi-WM further integrates latent diffusion and semantic features for predictive manipulation, confirming the advantages of DINO-based encodings over pixel- or global-token models.

## 7. Context within Visual World Modeling

DINO-WM exemplifies the shift from pixel-centric to representation-centric visual world models, leveraging the semantic richness and spatial locality of transformer-trained visual encoders. Its decoupled, modular approach contrasts with end-to-end latent-pixel RL systems, supporting markedly improved zero-shot adaptability and interpretability. Future research may further unify geometric and semantic encodings (as in LaDi-WM), integrate language-conditioned goals, or extend to settings with partial or absent action annotation [2411.04983, 2505.11528].

Source: https://www.emergentmind.com/topics/dino-wm