DeltaTok: Efficient Video Tokenization

• DeltaTok is a video representation method that compresses high-dimensional frame differences into a single semantic token per frame.
• It employs a transformer encoder/decoder framework to capture spatio-temporal changes and reduce token count by 1024× compared to standard feature grids.
• DeltaWorld leverages DeltaTok with a multi-hypothesis best-of-many training strategy to predict diverse, accurate future states while drastically reducing computational resources.

DeltaTok is a video representation and generative modeling approach that collapses the temporal difference between high-dimensional video frames into a single semantic token per frame, enabling highly efficient autoregressive generation of diverse future states. This paradigm underlies DeltaWorld, a generative world model that predicts future frames by autoregressively modeling the one-dimensional sequence of delta tokens. The DeltaTok-DeltaWorld framework achieves orders-of-magnitude reductions in both parameter count and computational cost compared to prior generative video models, while producing diverse, plausible, and accurately forecasted futures (Kerssies et al., 6 Apr 2026).

1. Construction of the Delta Token

Let φ denote a frozen Vision Foundation Model (VFM), such as DINOv3 ViT-B, mapping each RGB frame $v_t \in \mathbb{R}^{H'\times W'\times 3}$ to a spatial grid of patch tokens $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$, where $D$ is the VFM token dimension (e.g., 768), and $H \times W$ the grid (e.g., $32 \times 32$ for 512 × 512 input).

Rather than naïvely using the per-patch difference

$$\Delta^{(h,w)}_t = x^{(h,w)}_t - x^{(h,w)}_{t-1} \in \mathbb{R}^D,$$

DeltaTok implements a transformer-based encoder $g$ that ingests the concatenated grids $x_{t-1}$, $x_t$ (with learned frame tags) and a single learned “start” token $z_{\mathrm{init}}$. The encoder $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$0 jointly attends over the spatial grids and produces a single $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$1-dimensional continuous delta token encompassing all spatio-temporal change. This process compresses the change between two high-dimensional feature maps into a single, compact token.

Recovery of $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$2 from $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$3 and $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$4 is achieved using a transformer decoder $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$5: $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$6 with $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$7 taking $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$8 (zero-initialized patch slots) and $$x_t = \phi(v_t) \in \mathbb{R}^{H \times W \times D}$$9 to reconstruct the full feature map. The encoder/decoder is trained via patchwise mean squared error (MSE) minimizing

$D$0

over random frame pairs.

2. Generative Modeling in DeltaWorld

DeltaWorld replaces the canonical spatio-temporal token grid with a one-dimensional temporal sequence of delta tokens $D$1, where $D$2 encodes the “absolute” start frame.

A transformer-based autoregressive model $D$3, equipped with 1D rotary positional embeddings, models $D$4 given the previous $D$5. For each timestep, $D$6 noise queries $D$7 are prepended to the token sequence; $D$8 then predicts $D$9 candidate next-step tokens: $H \times W$0 where $H \times W$1 denote timestamps.

Subsequent roll-outs for mid-horizon future forecasting are autoregressively performed in parallel for all $H \times W$2 hypotheses. Each candidate $H \times W$3 is appended to its own context, and prediction advances further, producing multi-step, multi-hypothesis futures in a single forward pass per step.

Decoded feature grids are obtained as $H \times W$4; optional translation back to RGB is performed by a separate diffusion or regularized autoencoder.

3. Multi-Hypothesis Training (Best-of-Many Loss)

DeltaWorld introduces a multi-hypothesis “best-of-many” (BoM) training paradigm. For each input, $H \times W$5 candidate futures are predicted. The loss is incurred only for the candidate closest (under $H \times W$6 or Huber distance) to the ground-truth delta token: $H \times W$7 where $H \times W$8 is smooth-L1 or $H \times W$9. Optionally, a decoded-feature/pixel loss may be applied, but in practice only token-space loss is used to maximize efficiency.

This approach implements tractable non-diffusion-based multi-hypothesis learning: at inference, sampling new noise queries yields diverse, plausible future predictions in a single forward pass.

4. Efficiency Gains

DeltaTok achieves a dramatic token reduction. For 512 × 512 (or 256 × 256) resolution, standard VFM features use $32 \times 32$0 spatial tokens per frame; DeltaTok compresses this to 1 token per frame—a $32 \times 32$1 reduction in sequence length.

Parameter and computational comparisons are as follows:

Model	Total Params (B)	Inference FLOPs (TFLOPs for 20 futures, 2 crops)
DeltaWorld	0.45	31
Cosmos-4B	4	60,000
Cosmos-12B	12	64,000

DeltaWorld thus requires $32 \times 32$2 fewer parameters and $32 \times 32$3 fewer FLOPs than state-of-the-art autoregressive or diffusion-based video world models on matched inference settings.

5. Empirical Evaluation on Dense Forecasting

Quantitative metrics include semantic segmentation mIoU (VSPW, Cityscapes; short and mid-horizon) and monocular depth RMSE (KITTI). Results are reported for “best-of-20” (i.e., oracle selection over $32 \times 32$4 candidates) and “mean” (averaged features decoded per sample).

Key mid-horizon results on Cityscapes:

• Copy-last baseline: mIoU ≈ 39.6
• DINO-world (discriminative): 49.8
• Cosmos-12B: best ≈ 53.3 / mean ≈ 51.2
• DeltaWorld: best ≈ 55.4 / mean ≈ 51.3

DeltaWorld’s best prediction exceeds all prior generative models by ≈2 mIoU, and its mean matches or surpasses them, despite using orders-of-magnitude fewer resources.

Qualitatively, the model generates diverse futures—e.g., distinct plausible pedestrian trajectories, accurate prediction of rare or small objects, effective modeling of unexpected events (e.g., crossing trains), and reduced temporal blurring.

6. Algorithmic Summaries and Pseudocode

DeltaTok Tokenization encodes two VFM feature grids into a single delta token using a transformer encoder:

$32 \times 32$7

DeltaWorld Generation Step samples $32 \times 32$5 noise seeds, predicts candidate deltas, and applies BoM loss:

$32 \times 32$8

Autoregressive Rollout maintains $32 \times 32$6 parallel hypotheses:

$32 \times 32$9

7. Context and Significance

DeltaTok and DeltaWorld jointly present a solution to the central challenge of anticipatory world modeling in video: scalable, efficient, and diverse future prediction. By reducing spatio-temporal video to a minimal sequence of semantic deltas, DeltaWorld makes multi-hypothesis generative video modeling feasible using a resource profile two orders of magnitude lighter than previous large autoregressive or diffusion models. Key to this performance is the combination of semantic feature deltas (aligned with VFM structure) and algorithmic best-of-many supervision enabled by the collapse to one token per frame. This enables efficient exploration of the space of plausible futures across dense video prediction tasks, setting new benchmarks for both efficiency and accuracy in generative world models (Kerssies et al., 6 Apr 2026).