A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens

Abstract: Anticipating diverse future states is a central challenge in video world modeling. Discriminative world models produce a deterministic prediction that implicitly averages over possible futures, while existing generative world models remain computationally expensive. Recent work demonstrates that predicting the future in the feature space of a vision foundation model (VFM), rather than a latent space optimized for pixel reconstruction, requires significantly fewer world model parameters. However, most such approaches remain discriminative. In this work, we introduce DeltaTok, a tokenizer that encodes the VFM feature difference between consecutive frames into a single continuous "delta" token, and DeltaWorld, a generative world model operating on these tokens to efficiently generate diverse plausible futures. Delta tokens reduce video from a three-dimensional spatio-temporal representation to a one-dimensional temporal sequence, for example yielding a 1,024x token reduction with 512x512 frames. This compact representation enables tractable multi-hypothesis training, where many futures are generated in parallel and only the best is supervised. At inference, this leads to diverse predictions in a single forward pass. Experiments on dense forecasting tasks demonstrate that DeltaWorld forecasts futures that more closely align with real-world outcomes, while having over 35x fewer parameters and using 2,000x fewer FLOPs than existing generative world models. Code and weights: https://deltatok.github.io.

• The paper introduces DeltaTok, a novel tokenizer that compresses frame differences into single tokens, reducing video tokenization by up to 1024×.
• DeltaWorld leverages a Transformer-based architecture with a Best-of-Many objective to generate diverse future predictions in a single forward pass.
• Empirical results demonstrate superior semantic segmentation and depth forecasting performance with significantly lower parameters and FLOPs than previous models.

Delta Tokens for Efficient Generative World Modeling: DeltaWorld

Problem Statement and Motivation

The paper "A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens" (2604.04913) addresses the longstanding inefficiency in generative world models for visual forecasting. Traditional generative models employ dense spatio-temporal tokenization and operate in latent spaces optimized for pixel fidelity, leading to substantial computational and memory overhead. While prior works, especially discriminative models using vision foundation model (VFM) feature spaces, have reduced parameter and FLOPs costs, they inherently collapse multimodal futures into deterministic mean predictions. Generative models capable of forecasting diverse plausible futures are critical for reliable downstream decision-making in domains such as autonomous driving and robotics. However, the cost for generating multiple sample futures remains prohibitively high, bottlenecked by the multi-pass autoregressive or diffusion-based inference common in the field.

DeltaTok and DeltaWorld: Methodological Advances

This paper introduces DeltaTok—a tokenizer that encodes the difference in VFM features between consecutive video frames into a single continuous delta token. The key innovation is reducing video representation from a 3D spatio-temporal grid to a 1D temporal sequence; for example, for $512 \times 512$ resolution, this yields a $1024\times$ reduction in tokens per frame. By capturing only the structured, typically low-dimensional temporal dynamics between frames, DeltaTok eliminates spatial redundancy and enables vastly compact representations without compromising semantic integrity.

DeltaWorld, the proposed generative model, leverages DeltaTok to conditionally generate future sequences of delta tokens. Unlike prior pixel or feature grid-based generative models, DeltaWorld predicts multiple diverse futures in a single forward pass and supervises only the best hypothesis during training using the Best-of-Many (BoM) objective. This objective exploits the capacity of noise queries to diversify the trajectories of sampled futures, aligning with the requirement of efficient multi-hypothesis generation for practical applications.

Figure 1: DeltaWorld generates diverse sampled futures, with upper row showing context and oracle, and lower row visualizing four distinct DeltaWorld predictions in VSPW segmentation task.

Architecture and Training

DeltaWorld is architected as a compact Transformer-based predictor operating on sequences of delta tokens. Frame features are first encoded via a frozen VFM (DINOv3 ViT-B), then DeltaTok compresses the delta between consecutive features into a single token. The predictor receives the token sequence along with stochastic noise queries and autoregressively extends it, allowing diverse future trajectories depending on the noise realization. The spatial feature maps for each predicted future are reconstructed by the DeltaTok decoder, explicitly conditioned on the previous frame's features, inherently capturing temporal consistency.

The BoM training objective further enhances diversity: for each input context, $K$ noise queries are sampled, and only the prediction closest to ground truth is supervised, effectively mapping different noise queries to distinct plausible futures.

Empirical Evaluation

The paper evaluates DeltaWorld on semantic segmentation (VSPW, Cityscapes) and monocular depth (KITTI) forecasting tasks at multiple prediction horizons. Compared against generative (Cosmos-4B, Cosmos-12B) and discriminative (DINO-world) baselines, DeltaWorld consistently exhibits superior alignment with ground-truth futures. Specifically, it achieves:

• Over $35\times$ fewer model parameters and $2{,}000\times$ fewer FLOPs relative to Cosmos-12B, with substantially improved mIoU and RMSE metrics.
• Strong best-of-many performance: for Cityscapes mid-horizon segmentation, DeltaWorld yields best mIoU of $55.4$ compared to Cosmos-12B's $53.3$. Mean predictions are also competitive, demonstrating that sampled diversity covers plausible future modes rather than mere noise.

  Figure 2: DeltaWorld demonstrates enhanced mean and best sample prediction relative to Cosmos-12B for KITTI mid-horizon depth estimation.

Qualitative analysis further highlights DeltaWorld's ability to capture stochasticity inherent in real-world sequences, such as pedestrian movement and camera ego-motion, with diverse plausible trajectories from a single context.

Figure 3: DeltaWorld mid-horizon autoregressive rollouts conditioned on noise queries, showing segmentation and depth task head visualization across datasets.

Computation and Efficiency Analyses

The shift from dense spatial token grids to single delta tokens per frame translates to an orders-of-magnitude reduction in compute (predictor accounts for less than 1% of overall FLOPs during multi-sample inference), without performance degradation. Ablation studies confirm that delta compression preserves prediction quality better than naïve frame compression, which suffers from representational bottlenecks.

BoM sample scaling experiments demonstrate continued improvement in best prediction as training query count $K$ increases, validating the hypothesis that more diverse training queries yield increasingly specific futures without degrading mean performance.

Discussion, Implications, and Future Directions

Theoretical implications include the paradigm shift toward explicitly modeling temporal change rather than absolute spatial content, aligning with foundational video coding insights yet unexploited in contemporary deep generative frameworks. This has ramifications for scaling generative world models: predictor size, context length, and rollout depth can be extended with minimal additional overhead, enabling real-time forecasting for physically embodied AI agents. Moreover, the highly compact delta token representation opens avenues for integrating action-conditioning and enhanced downstream planning, potentially facilitating efficient reinforcement learning and model-based control in dynamic environments.

Figure 4: Demonstrates DeltaWorld's capacity for multiple plausible mid-horizon rollouts from shared context, with rich diversity in prediction outcomes.

Figure 5: RGB visualization of DeltaWorld's diverse mid-horizon rollouts for three datasets, underlining its generalization and semantic fidelity.

Practically, DeltaWorld's architecture makes scalable multi-future forecasting feasible for large-scale deployment in perception, robotics, and simulation. The absence of expensive iterative diffusion or multi-pass autoregressive generation dramatically reduces latency and hardware requirements, bringing efficient generative forecasting within reach of real-time, resource-constrained applications.

Future developments should address limitations of BoM's implicit distribution modeling to ensure better coverage and calibration of predictive uncertainty. Enhancing error accumulation mitigation during long-horizon rollouts and integrating explicit action conditioning are promising directions. The delta tokenization principle could also generalize to non-visual modalities for unified spatio-temporal world modeling across multimodal foundation models.

Conclusion

DeltaWorld, through its DeltaTok tokenizer and efficient generative architecture, establishes a new standard for tractable and performant multi-future world modeling in video forecasting. By compressing only the temporal differential and leveraging single-pass BoM training, it achieves competitive or stronger predictive metrics than prior generative and discriminative models, while drastically reducing computational burden. The methodology is broadly applicable across vision-based tasks and poised to influence future advances in scalable world modeling for embodied AI.