DeltaWorld: Efficient Generative Video Forecasting

• DeltaWorld is a generative world modeling framework that compresses inter-frame visual dynamics into single delta tokens, achieving a 1,024× compression rate.
• It employs a 12-layer ViT-B transformer with a Best-of-Many extension to enable multi-hypothesis video forecasting in a single forward pass.
• Benchmark results reveal that DeltaWorld outperforms previous models using over 35× fewer parameters and 2,000× fewer FLOPs while maintaining diverse prediction quality.

DeltaWorld is an efficient generative world modeling framework that operates on "delta tokens," a highly compressed representation of inter-frame visual changes, enabling tractable, diverse, and computationally efficient video forecasting. It achieves a 1,024× compression rate relative to per-frame patch tokens and supports dense multi-hypothesis rollout in a single transformer pass, outperforming or matching prior state-of-the-art models while using over 35× fewer parameters and 2,000× fewer FLOPs (Kerssies et al., 6 Apr 2026).

1. DeltaTok Tokenization Mechanism

DeltaWorld leverages the DeltaTok tokenizer, which encodes video dynamics through the difference between consecutive frames embedded in the feature space of a Vision Foundation Model (VFM). Each input frame $v_t \in \mathbb{R}^{H' \times W' \times 3}$ is transformed by a frozen VFM $\phi$ (specifically DINOv3 ViT-B) into a spatial grid of visual patch tokens $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$, where $x_{t,h,w} \in \mathbb{R}^D$.

The DeltaTok encoder $g$ processes pairs of consecutive feature maps $(x_{t-1}, x_t)$, together with a learnable initialization token $z_{init}$, to produce a single $D$-dimensional "delta" token:

$z_t = g(x_{t-1}, x_t, z_{init}) \in \mathbb{R}^D$

which compresses the feature difference, akin to learning a compact approximation of $\Delta_t = x_t - x_{t-1}$.

For reconstruction, DeltaTok employs a decoder $\phi$0 such that:

$\phi$1

with mean squared error (MSE) used as the tokenizer's reconstruction loss:

$\phi$2

With $\phi$3 video and $\phi$4 patching ($\phi$5), spatial tokens per frame are $\phi$6; DeltaTok compresses this to a single $\phi$7, achieving $\phi$8 compression.

2. Generative Model Architecture and Best-of-Many Extension

The generative model is a 12-layer ViT-B transformer predictor $\phi$9 ($$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$0, eight heads), consuming the sequence of prior delta tokens $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$1, with a prepended "black" token $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$2. 1-D Rotary Position Embeddings (RoPE) are applied to the first 60 dimensions of each attention head, with a single causal mask enforcing autoregressive prediction.

To enable parallel multi-hypothesis prediction, the Best-of-Many (BoM) extension replaces a single learned query with $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$3 noise queries $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$4. For each $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$5, the model predicts a future delta token:

$$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$6

where $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$7 and $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$8 encode temporal context and target step.

Because each frame is condensed to one token, generating $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$9 possible futures is $x_{t,h,w} \in \mathbb{R}^D$0 a single pass through the transformer, sharply reducing computational cost compared to spatial models (which require $x_{t,h,w} \in \mathbb{R}^D$1 passes).

3. Training Methodology

DeltaWorld's training regime centers on the BoM loss and efficient context sampling. For each input sequence, $x_{t,h,w} \in \mathbb{R}^D$2 queries are used to generate $x_{t,h,w} \in \mathbb{R}^D$3 future delta token hypotheses. The loss is the smooth-L1 distance between prediction and the ground truth delta token, but only the best among the $x_{t,h,w} \in \mathbb{R}^D$4 predictions (the one with minimal distance) is used for supervision:

$x_{t,h,w} \in \mathbb{R}^D$5

with

$x_{t,h,w} \in \mathbb{R}^D$6

Training proceeds as:

1. Sample $x_{t,h,w} \in \mathbb{R}^D$7 context sequences $x_{t,h,w} \in \mathbb{R}^D$8 and target times $x_{t,h,w} \in \mathbb{R}^D$9.
2. Encode $g$0 into delta tokens via frozen DeltaTok.
3. Draw $g$1 noise queries.
4. Predict the $g$2 future delta tokens in parallel.
5. Compute $g$3 and backpropagate gradients through only the selected $g$4 branch.

Optimization is performed with AdamW (lr = $g$5; weight decay = $g$6; batch size = 1,024; sequence length = 8). The main experiments use $g$7. Predictors are trained for 300K steps and fine-tuned for 5K steps at $g$8 lower learning rate.

4. Inference, Multi-Modal Diversity, and Sampling

At inference, $g$9 random noise queries are sampled per timestep (typically $(x_{t-1}, x_t)$0 or $(x_{t-1}, x_t)$1). The predictor $(x_{t-1}, x_t)$2 outputs $(x_{t-1}, x_t)$3 candidate delta tokens in a single forward pass. For multi-step rollouts, each sequence appends its predicted $(x_{t-1}, x_t)$4 and repeats the process, yielding $(x_{t-1}, x_t)$5 diverse future video continuations.

Diversity is maintained by the BoM objective, which encourages the model to utilize different noise queries to access different plausible modes in future prediction space. This enables DeltaWorld to generate plausible multi-modal futures, such as variations in pedestrian trajectories or camera motion, within a single parallelizable inference (Kerssies et al., 6 Apr 2026).

5. Computational Efficiency and Quantitative Results

DeltaWorld achieves significantly reduced computational and parameter overhead compared to contemporary generative video models. The combined DeltaWorld predictor and tokenizer utilize approximately 0.3B parameters. In contrast, the Cosmos 4B/12B models require 4B/12B parameters plus a 7B-parameter diffusion decoder.

A breakdown for dense forecasting with 20 samples:

• DeltaWorld: 31 TFLOPs
• Cosmos-4B: ≈60,000 TFLOPs
• Cosmos-12B: ≈64,000 TFLOPs

This represents approximately $(x_{t-1}, x_t)$6 fewer FLOPs and $(x_{t-1}, x_t)$7 fewer parameters for DeltaWorld. A representative GFLOPs table (single sample, four-frame context at $(x_{t-1}, x_t)$8) is:

Component	DINO-world	DeltaWorld
VFM backbone (4 frames)	188.7	188.7
Future predictor	96.96	0.31
DeltaTok encoder (4 frames)	–	387.7*
Per-sample predictor	–	0.26
DeltaTok decoder (per step)	–	46.12

*shared across all samples

On dense forecasting benchmarks (VSPW segmentation, Cityscapes segmentation, KITTI depth), DeltaWorld's best-of-20 predictions outperform Cosmos by approximately 10 mIoU on Cityscapes and 4 mIoU on VSPW at a fraction of the compute. Mean scores are close to those from discriminative baselines, indicating the generated diversity remains realistic and faithful. Qualitative results depict plausible multi-modal futures, particularly in pedestrian and camera motion scenarios (Kerssies et al., 6 Apr 2026). The following table summarizes key metrics:

Model	GFLOPs	VSPW short	VSPW mid	City short	City mid	KITTI RMSE short	KITTI mid
DeltaWorld	$(x_{t-1}, x_t)$9	55.4(53.7)	50.1(46.7)	65.8(63.9)	55.4(51.3)	3.00(3.17)	3.88(4.17)
Cosmos-12B	$z_{init}$0	51.7(50.7)	47.7(45.5)	55.3(56.0)	53.3(51.2)	3.72(3.71)	4.01(4.14)

Numbers in parentheses are mean scores; others are best-of-20.

6. Implementation and Reproducibility

DeltaWorld relies on the following components:

• VFM: DINOv3 ViT-B, patch size 16.
• DeltaTok tokenizer: ViT-B encoder/decoder trained for 50K steps (AdamW, batch 1,024, MSE loss).
• Predictor: ViT-B trained for 300K steps (fine-tuned 5K); BoM with $z_{init}$1.
• Training data: ≈4M video clips (mean 11 s, 16 fps).
• Data augmentations: random resized crops, temporal $z_{init}$2 s.
• Benchmark evaluation: VSPW, Cityscapes, and KITTI.

Pretrained code and weights are publicly available: https://deltatok.github.io.

7. Context and Significance

DeltaWorld demonstrates that generative world modeling can effectively operate over highly compressed difference tokens in the feature space of a VFM. By collapsing the per-frame spatiotemporal representation to a one-dimensional temporal sequence, DeltaWorld substantially reduces the computational burden of diverse multi-step video forecasting. This framework enables best-of-many supervision and single-pass sampling for multi-hypothesis rollout, previously computationally prohibitive in generative world models. The substantial gains in efficiency and predictive performance suggest that delta-token-based approaches offer a practical, scalable alternative for complex video forecasting tasks (Kerssies et al., 6 Apr 2026).