WorldTasks-Benchmark Overview

• WorldTasks-Benchmark is a benchmark that tests visual world models’ ability to generate task-execution videos from a single image and a brief instruction.
• It employs automated task generation and strict VLM-based filtering to assess performance on task completion, agent attribution, and physical realism.
• The benchmark covers diverse tasks such as navigation, object interaction, and compound sequences, leveraging self-distillation and RL to improve execution.

WorldTasks-Benchmark, or WorldTasks-Bench, is a benchmark introduced in "World Model Self-Distillation: Training World Models to Solve General Tasks" for evaluating whether a visual world model can take an initial scene image and a short high-level instruction, then generate a task-execution trajectory without relying on curated task–video supervision (Stapf et al., 10 Jun 2026). It is designed around “general task-solving” from a single observation, with compact prompts of the form “[Agent description]: [Task instruction],” and it evaluates generated videos using binary judgments for task completion, correct agent attribution, and physical realism and temporal consistency.

1. Purpose and problem setting

WorldTasks-Bench was introduced to close the gap between the emergent task-solving abilities of pretrained video world models and their reliance on detailed textual descriptions for execution (Stapf et al., 10 Jun 2026). In the formulation used by the benchmark, a model under test receives an initial scene image $I$ and a short task instruction $T$, and must internally generate a task-execution video rather than depending on an externally supplied long-form solution.

The benchmark targets tasks that require planning and decision-making under minimal textual specification. The reported task families include navigation and positioning, object interaction and manipulation, perception and attention, action execution, vehicle control, UI interaction, and compound sequences. Representative examples include “Walk toward the building entrance on the right,” “Align the car’s front bumper with the white track curb ahead,” “Use the right hand to pick up the pink bottle and pour water on the flower,” and “Turn right to face the welcome sign on the easel.” This scope suggests an emphasis on instruction following in diverse visual scenes rather than on narrowly defined robotics-only control.

A central design choice is that evaluation starts from a single unlabeled scene image with minimal text. This distinguishes the benchmark from settings in which execution is guided by detailed captions or supervised task–video pairs. The intended question is whether a world model can transform a compact instruction into a coherent execution trajectory.

2. Dataset composition and benchmark structure

The underlying WorldTasks dataset contains 20,000 starting images sourced from video-game environments and real-world scenes, largely based on MiraData (Stapf et al., 10 Jun 2026). For each image, a VLM, specified as Qwen3.5-27B, pre-generates 8 task–solution pairs. After filtering, the training split contains 146,440 task prompts. WorldTasks-Bench itself is a controlled evaluation set of 200 randomly selected image–task pairs. Validation and test splits are not separately reported.

The benchmark mixes synthetic and real-world visual sources. The initial observations are unlabeled scene images. The Executor input is the initial scene image $I$ plus the short task instruction $T$, and the output is a generated video, described in experiments as a short clip of 121 frames. During training, the Teacher or Demonstrator uses the same image together with a detailed execution description $D$.

The addressed-agent distribution is skewed toward first-person and human-character prompts.

Addressed agent	Training split	WorldTasks-Bench
First-person views	50.7%	53.5%
Human characters	39.0%	37.5%
Vehicles	5.2%	6.0%
Inanimate objects/landmarks	2.4%	—
Creatures	1.9%	—
Animals	0.6%	—
Crowds	0.1%	—

The task-family distribution covers both low-level and multi-step behaviors.

Task family	Training split	WorldTasks-Bench
Positioning	22.2%	22.0%
Navigation	20.3%	22.5%
Object interaction	19.0%	17.0%
Perception	14.3%	16.0%
Combat / action execution	6.6%	11.0%
Compound / multi-step	4.6%	1.5%
Vehicle control	4.1%	3.5%
UI interaction	3.7%	2.5%
Other	5.1%	4.0%

These distributions matter because they delimit the benchmark’s notion of “general tasks.” The coverage is broad, but it is not uniform across agent types or task families.

3. Task generation and curation pipeline

For each unlabeled image, Qwen3.5-27B generates 8 candidate pairs consisting of a short task instruction and a detailed step-by-step solution (Stapf et al., 10 Jun 2026). The instruction follows the format “[Agent description]: [Task instruction],” while the solution is a multi-sentence script describing actions over time. The detailed solution is not the evaluation input for the Executor; it is used to condition the Demonstrator during training.

The curation pipeline includes several filtering stages. Low-level image quality filters use the variance of Laplacian for blur, mean luminance thresholds, and pixel ratios to filter under- or over-exposed or degenerate frames, with thresholds reported as min_laplacian_var = 12.0, min_mean_luma = 20.0, max_mean_luma = 235.0, max_black_ratio = 0.85, and max_white_ratio = 0.85. A CLIP-based aesthetics score keeps the top 90%. A further “VLM-based dataset quality filtering” prompt removes non-actionable or unsuitable scenes. Qwen3.5-27B is also used to characterize the prompt set by addressed-agent category and task type.

The benchmark therefore combines automatic task generation with semantic filtering rather than manual task annotation. A plausible implication is that scale is obtained by leveraging VLM synthesis and screening, while benchmark validity is maintained through strict curation and separate VLM-based evaluation.

4. Evaluation protocol and metrics

WorldTasks-Bench evaluates one generated video per image–task pair under a VLM-based protocol with three binary criteria (Stapf et al., 10 Jun 2026). The criteria are task completion, correct agent attribution, and physical realism and temporal consistency. The prompts are explicitly named the “Task Solved Evaluation Prompt,” the “Correct Agent Attribution Prompt,” and the “Physical Realism Prompt.”

The benchmark reports three corresponding metrics. Task Score is the VLM-judged success rate that the instructed task is completed. Agent Score is the VLM-judged correctness that the intended agent performs the action. Realism Score is the VLM-judged plausibility of physics and temporal coherence. Aggregation is by proportion over valid judgments, and samples with evaluator failures are excluded. The paper does not provide a formal LaTeX definition for these metrics; they are simple averages of binary judgments.

Evaluator robustness is handled procedurally. If the VLM returns a malformed output or an API error occurs, that sample is discarded. Failure rates are reported per method, with the largest observed failure rate stated as 3.0%. The prompts emphasize strict evidence-based judgments and explicitly penalize ambiguity, misattribution, and physical inconsistencies. Appendix reporting includes denominators and failure rates; for example, for LTX-2 (8-step)+WMSD the reported counts are Task 121/200, Agent 134/194, Realism 172/195, and Avg 143/197.

This evaluation protocol measures whether the generated video satisfies the instruction under a strict visual judgment model. It does not define success by likelihood, caption similarity, or framewise reconstruction quality.

5. World-model interface and relation to self-distillation

The benchmark is introduced together with a training framework in which a caption-guided Demonstrator transfers execution knowledge to an instruction-conditioned Executor (Stapf et al., 10 Jun 2026). The world-model interface is written as

$p(\tau \mid \mathcal{I}, \mathcal{T}),$

with latent trajectory $\tau = \{x_t\}_{t \in [0,1]}$. The student dynamics are given by

$$\frac{d x_t}{d t} = v_{\theta}(x_t,t \mid c_{\mathrm{E}}), \qquad x_0 \sim p_0.$$

The conditioning interface distinguishes the two roles. For the Executor, $c_E = (I, T)$; for the Demonstrator, $c_D = (I, D)$. The Demonstrator is a caption-guided video diffusion or flow model conditioned on detailed execution descriptions $T$0, while the Executor is the instruction-conditioned world model that must solve tasks from only the initial image and short instruction.

The paper defines off-policy distillation, on-policy local discrepancy, on-policy distillation, a key gradient decomposition, distillation-as-reward, an anchor loss, and a final objective. The total reward is

$T$1

and the VLM task-success reward is

$T$2

Reinforcement learning is implemented with Flow-GRPO, and Advantage Weighted Matching is also used. The latter is described as aligning RL updates with the flow-matching loss through advantage weighting and reducing variance versus stepwise likelihood objectives.

The experimental instantiations reported for WorldTasks-Bench include LTX-2-19b-distilled (ltx2_i2v) and HunyuanVideo-1.5-480p_i2v (hy15_i2v). Fine-tuning uses LoRA rank 64 and alpha 128. The optimizer is Adam with learning rate $T$3 for LTX-2 and $T$4 for HY1.5, betas $T$5, and weight decay $T$6. For LTX-2-based runs, training resolution is $T$7, evaluation resolution is $T$8, the number of frames is 121, and inference uses 8 steps for both training and evaluation in the distilled 8-step variant, with batch size 32 and group size 24. For HunyuanVideo-1.5-based runs, training and evaluation are at $T$9, with 121 frames, 10 training steps, 40 evaluation steps, and group size 16. The frame rate is not explicitly stated, and the benchmark reports binary criteria per video independent of FPS.

6. Empirical results, ablations, and transfer

On WorldTasks-Bench, the reported LTX-2 (8-Step)+WMSD result is Task Score 0.605, Agent Score 0.691, Realism 0.882, Average 0.726, and end-to-end time 10.1 s (Stapf et al., 10 Jun 2026). The corresponding LTX-2 (8-Step)+VLM baseline, which outsources solution prompting, is reported as Task 0.495, Agent 0.572, Realism 0.732, Average 0.598, and end-to-end time 10.5 s. The base LTX-2 (8-Step) result is Task 0.285, Agent 0.391, Realism 0.694, and Average 0.455. For HunyuanVideo-1.5, the base model reports Task 0.464, Agent 0.540, Realism 0.780, and Average 0.597, while HY1.5+WMSD reports Task 0.574, Agent 0.630, Realism 0.828, and Average 0.673.

The paper states that combining on-policy self-distillation with RL enables the student to surpass the demonstrator under VLM evaluation. Ablations further report that on-policy variants with distillation reward continue to improve beyond approximately 60 steps and surpass off-policy on both WorldTasks score and PickScore, whereas standard RL alone plateaus after approximately 50 steps. Alternating teacher–student updates are described as slower and less stable. With respect to distillation strength, the best performance is reported at $I$0; smaller values weaken guidance, while larger values overly constrain RL-driven improvements.

Category-specific improvements are also reported for the Executor with WMSD. Navigation Task Score rises from 31.1% to 75.6%, object interaction from 17.6% to 55.9%, perception from 40.6% to 68.8%, positioning from 27.3% to 50.0%, and combat actions from 27.3% to 36.4%. Agent grounding improves from 42.1% to 86.0% for first-person prompts and from 36.0% to 76.0% for human-character prompts; vehicles reach approximately 50.0% on a small sample of 12.

Transfer results are reported on the DreamGen robotics benchmark. Despite being trained without robot-specific supervised data, LTX-2+WMSD achieves Object 70.0, Behavior 57.4, and Environment 58.6. Cosmos (SFT) is reported as Object 62.0, Behavior 61.7, and Environment 65.5, while a zero-shot baseline such as LTX-2 reports Object 20.0, Behavior 29.8, and Environment 41.4. The interpretation given is that WorldTasks training transfers to robotics-like manipulation tasks, but robot-specific dynamics remain imperfect without targeted data. WorldTasks-Bench and DreamGen are presented as complementary: the former focuses on general visual tasks from single images, whereas the latter evaluates robotics trajectories and physical interactions.

7. Limitations, future directions, and reproducibility

Several limitations are explicitly identified for the benchmark and its evaluation pipeline (Stapf et al., 10 Jun 2026). VLM-based judgments can be noisy; strict prompts and a consistency reward mitigate but do not eliminate errors. Discarding evaluator failures introduces slight sample-size variability, which is why denominators are reported. Reward hacking can occur without a consistency reward, with examples including teleported objects or implausible insertions; this is mitigated by a “visual quality & temporal consistency” reward prompt.

Dataset bias is also acknowledged. Training prompts are skewed toward first-person and human agents, while vehicles and UI interactions are smaller slices. Game-style environments may differ from strict real-world physics, although physical plausibility is explicitly evaluated through the realism score. These points are important when interpreting reported generalization.

Future directions proposed in the paper include additional task families such as puzzle solving and longer-horizon compound tasks, more standardized evaluator ensembles using multiple VLM judges, consensus, and confidence calibration, as well as video continuation and in-context learning to incorporate robot-specific appearance and dynamics from short context. Other proposed improvements include better rewards for physics and object permanence and augmenting evaluation with pairwise comparisons or preference aggregation.

The benchmark is accompanied by public resources: a project page at https://sebastian-stapf.github.io/world-model-self-distillation/, code at https://github.com/sebastian-stapf/world-model-self-distillation, and the WorldTasks dataset at https://huggingface.co/datasets/sebastian-stapf/WorldTasks. Reproducibility materials include evaluator prompts, training reward prompts, hyperparameter tables for LTX-2 and HunyuanVideo-1.5 runs, and VLM evaluator failure rates and denominators. The main reported results used 128 GH200 GPUs, while ablations used 16 GH200 GPUs over 12 hours.

For evaluation of a new model, the benchmark procedure is explicit: use the 200 image–task pairs, provide each pair as $I$1 to the model, generate a short video trajectory, run the three evaluator prompts, discard malformed evaluator outputs, and report Task Score, Agent Score, Realism Score, their average, denominators, failure rates, inference resolution, number of frames, inference steps, and end-to-end time. Optional reporting includes PickScore, qualitative examples, and per-category breakdowns. In this sense, WorldTasks-Bench is not only a test set but also a standardized protocol for measuring whether an instruction-conditioned world model can convert a single visual observation and compact instruction into a plausible execution video.