World Model Self-Distillation: Training World Models to Solve General Tasks

Abstract: Pretrained video generators are promising visual world models that exhibit emergent task-solving abilities; however, their reliance on detailed textual descriptions limits their direct use for planning and decision-making. Existing approaches either outsource this reasoning to language or vision-LLMs, or rely on supervised fine-tuning with paired task-execution videos, which are costly to collect and difficult to scale. We propose a scalable framework that elicits task-solving ability in such models by combining self-distillation with reinforcement learning. Given an unlabeled scene image, a vision-LLM generates a candidate task and a detailed step-by-step solution. The solution conditions a pretrained video diffusion model, the Demonstrator; we distill its behavior into an Executor conditioned only on the image and a short task prompt. This transfers execution knowledge from caption-guided generation to instruction-conditioned task solving without curated task-video supervision. We further improve the Executor with reinforcement learning from VLM feedback, exploiting the asymmetry between judging whether a sampled video satisfies a task and generating the solution. Experiments on our proposed WorldTasks-Benchmark and the DreamGen robotics benchmark show that the Executor surpasses the Demonstrator under our VLM-based evaluation protocol and transfers competitively to robotic tasks.

• The paper presents WMSD, a novel framework that transforms caption-conditioned video diffusion models into instruction-driven task solvers using synthetic supervision and reinforcement learning.
• It demonstrates that on-policy self-distillation significantly outperforms off-policy approaches, achieving notable improvements such as +44.5 points in navigation and +38.3 points in object interaction tasks.
• The approach scales efficiently and generalizes to unseen domains, enabling Executors to surpass their demonstrator teachers and achieve robust agent grounding of up to 86% in first-person scenarios.

World Model Self-Distillation for General Task-Solving in Video World Models

Introduction and Motivation

This paper introduces World Model Self-Distillation (WMSD), a framework for training pretrained video world models to solve general tasks directly from high-level instructions, without requiring extensive paired task-execution supervision. In contrast to previous work that either outsources reasoning to language/vision-LLMs (VLMs) or depends on costly paired data, WMSD blends self-distillation with reinforcement learning from vision-LLM feedback. The centerpiece of this approach is the transformation of a caption-conditioned video diffusion model into an instruction-conditioned task solver by leveraging synthetic supervision and online RL-based policy improvement.

The appeal of this methodology lies in its data efficiency, scalability, and ability to surpass the capabilities of its teacher (the Demonstrator) under VLM-based evaluation protocols. This represents a shift in world-model training: rather than passively mimicking detailed textual prompts, a distilled Executor learns to synthesize plausible, goal-oriented video trajectories given only an initial observation and a concise instruction.

Figure 1: Overview of WMSD. Candidate tasks and stepwise solution prompts are generated by a VLM, which conditions a video diffusion Demonstrator; a distilled Executor learns to solve tasks from high-level prompts, further optimized by RL from VLM feedback.

Methodology

Task-Conditioned Video Generation

The target model is a conditional flow-matching video generator. For each instance, the system starts from an initial scene observation $\mathcal{I}$ and a short instruction $\mathcal{T}$, seeking to model $p(\tau|\mathcal{I},\mathcal{T})$, where $\tau$ denotes the action trajectory. The teacher (Demonstrator) receives a richer context—an execution description $\mathcal{D}$—and generates reference rollouts via a pretrained video diffusion model. The student (Executor), whose parameters are trainable, only receives $\mathcal{T}$ and $\mathcal{I}$ as input and must learn to solve the task directly.

Two-Stage Self-Distillation Pipeline

1. Dataset Synthesis with Vision–LLMs: For each sampled scene image, a VLM generates a set of candidate (task, solution) pairs, spanning a wide variety of environments, agent types, and task complexities.
2. Self-Distillation: The Executor is trained to match the Demonstrator’s behavior given the same initial scene but using only the high-level instruction. Both off-policy (matching on teacher’s trajectories) and on-policy (matching on student rollouts) distillation objectives are evaluated, with the on-policy variant showing markedly superior task-solving effectiveness.

Reinforcement Learning Augmentation

Crucially, to overcome the Demonstrator performance upper bound, RL is introduced. The Executor's sampled videos are evaluated by a VLM for task success, agent correctness, and physical plausibility. RL updates increase the likelihood of high-reward rollouts:

• Reward Formulation: Combined VLM-based reward signals (task completion, agent attribution, and consistency) are weighted together with a distillation-based reward that stabilizes training.
• Optimization: RL is performed using group-relative policy optimization (GRPO/AWM) adapted for flow-matching generators, efficiently leveraging few-step rollouts.

Theoretical Control of Drift

A Grönwall-based argument ensures that on-policy matching of student and teacher velocity fields controls divergence between Executor and Demonstrator distributions, justifying the stability of the proposed joint distillation and RL approach.

Empirical Evaluation

Datasets and Benchmarks

• WorldTasks Dataset: Contains 20,000 curated scene images with 146,440 diverse VLM-generated task prompts. Prompts are broad, covering navigation, manipulation, perception, and agent types including human, first-person, vehicles, etc.
• WorldTasks-Bench: A set of 200 challenging instructions for systematic evaluation; rollouts are scored by VLMs for task success, agent attribution, and physics/realism.

  Figure 2: Examples from WorldTasks, showing initial frames, agent/task prompts, and detailed solution descriptions across diverse agent types and task domains.

  

Figure 3: Addressed-agent categories distribution—demonstrating task-agent diversity in the dataset.

Main Results

• On-Policy vs. Off-Policy Distillation: On-policy self-distillation outperforms the off-policy baseline by a wide margin, with additional gains from the distillation reward component.

  Figure 4: Ablation of distillation variants on WorldTasks-Bench; on-policy self-distillation yields higher overall metrics.

• RL Surpassing the Demonstrator: Executor models trained with on-policy self-distillation and RL not only close the gap to the Demonstrator but consistently exceed it on composite task benchmarks, highlighting the practical benefit of VLM-driven RL in this setting.
• Baseline Comparisons: Across both LTX-2 and HunyuanVideo-1.5 backbones, WMSD substantially improves VLM-judged task and agent scores with no compromise in generation speed. Supervised fine-tuning with auto-annotated task-video pairs provides negligible gains—confirming that high-level distilled knowledge and VLM feedback offer better scalability and generalization than weak automatic curation.

Fine-Grained Analysis

Figure 5: Performance stratified by task and agent type—navigation and object interaction tasks, as well as first-person and human-character prompts, benefit most from WMSD.

WMSD shows particularly strong improvements in navigation (+44.5 pts) and object interaction (+38.3 pts) tasks; agent grounding for first-person/POV prompts improves to 86%, suggesting robust attribution in ambiguous settings.

Qualitative Behavior

Generated videos from the WMSD-enhanced Executor demonstrate stronger agent-environment interaction, more plausible motion, and marked reduction of physical inconsistencies or reward hacking behaviors. The effect of RL plus anchor-based distillation is evident in visual quality and temporal coherence.

Generalization Capabilities

WMSD-trained Executors generalize competitively to unseen robotic domains. On the DreamGen benchmark, zero-shot models trained via WMSD achieve performance comparable to models fine-tuned with supervised robot-specific data, especially on complex manipulation and environment-affecting tasks. However, without in-domain data, fine-grained robot dynamics remain imperfect, reflecting the limits of pure data-free transfer.

Figure 6: Example of cross-domain generalization—Executor solves a compound manipulation task on the DreamGen robotics benchmark.

Methodological and Practical Implications

WMSD sidesteps the need for explicit task-video labeled pairs by using VLM-generated solutions as a form of scalable soft supervision. This unlocks several advantages:

• Data efficiency: Any scene image can be used to generate a variety of task-solution pairs without manual annotation.
• Model scalability: The approach naturally extends to ever-larger video models and broader task domains, as shown by robust results across different generator architectures.
• Improvement beyond demonstration: RL with VLM feedback enables Executors to exceed the capabilities of their pretrained teachers.
• Robustness and drift control: The Demonstrator anchor regularizes Executor behavior, maintaining visual and task realism despite noisy RL signals.

Theoretically, the methodology clarifies the value of asymmetric generation-verification tasks, whereby recognizing a successful solution is substantially easier than synthesizing one—a property leveraged via VLM-based feedback.

Limitations and Future Directions

• Reward Model Coverage: VLM-based evaluation is not immune to errors or ambiguity, especially for fine-grained or out-of-distribution tasks. Reward hacking remains a risk, albeit mitigated by explicit consistency terms.
• Robot-Specific Dynamics: Purely data-free approaches lack capacity for learning nonvisual or platform-specific behaviors; in-context learning or hybrid continuation strategies may close this gap.
• Ongoing Scaling: Further work on stability of advanced distillation objectives (e.g., DMD), improved reward modeling, and even tighter integration with real-world planning agents is warranted.

Conclusion

World Model Self-Distillation (WMSD) delivers a scalable, generalizable recipe for converting powerful video generative models into instruction-driven task solvers without the bottleneck of paired supervision. By uniting VLM-generated synthetic supervision, on-policy self-distillation, and RL refinement, WMSD-trained Executors achieve state-of-the-art performance across generalist and robotics-oriented video benchmarks and robustly transfer agent-centric reasoning into actionable visual dynamics. This framework offers a promising avenue for advancing large-scale world modeling, instruction-following control, and scalable training pipelines for embodied AI systems.

Figure 7: Example dataset samples—scene, task, and corresponding stepwise solution used for demonstrator distillation.

Figure 8: Additional dataset samples, demonstrating rich agent and instruction diversity foundational to WMSD's success.