DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models

Abstract: Robotic manipulation requires sophisticated commonsense reasoning, a capability naturally possessed by large-scale Vision-LLMs (VLMs). While VLMs show promise as zero-shot planners, their lack of grounded physical understanding often leads to compounding errors and low success rates when deployed in complex real-world environments, particularly for challenging tasks like deformable object manipulation. Although Reinforcement Learning (RL) can adapt these planners to specific task dynamics, directly fine-tuning VLMs via real-world interaction is prohibitively expensive, unsafe, and sample-inefficient. To overcome this bottleneck, we introduce DreamPlan, a novel framework for the reinforcement fine-tuning of VLM planners via video world models. Instead of relying on costly physical rollouts, DreamPlan first leverages the zero-shot VLM to collect exploratory interaction data. We demonstrate that this sub-optimal data is sufficient to train an action-conditioned video generation model, which implicitly captures complex real-world physics. Subsequently, the VLM planner is fine-tuned entirely within the "imagination" of this video world model using Odds Ratio Policy Optimization (ORPO). By utilizing these virtual rollouts, physical and task-specific knowledge is efficiently injected into the VLM. Our results indicate that DreamPlan bridges the gap between semantic reasoning and physical grounding, significantly improving manipulation success rates without the need for large-scale real-world data collection. Our project page is https://psi-lab.ai/DreamPlan/.

• The paper introduces DreamPlan, which trains an action-conditioned video world model on 2,056 exploratory robot trajectories and uses Best-of-K preferences with ORPO to fine-tune a vision-language planner without further real-world reinforcement learning.
• DreamPlan raises the average manipulation score from 0.35 for the strongest zero-shot baseline to 0.60 across rope, cloth, and toy-arm tasks, showing that physical grounding can outperform simply scaling the VLM.
• The method distills world-model verification into policy weights, achieving about 1 second and 15 TFLOPs per decision versus 2,605 seconds and roughly 156,000 TFLOPs for deployment-time verification of eight candidates.

DreamPlan addresses a persistent weakness of zero-shot vision-LLM (VLM) planners in robotic manipulation: they reason semantically well but lack grounded physical understanding of how objects respond to contact, particularly for deformable objects where small action deviations produce large topological changes. Rather than fine-tuning the VLM through costly real-world reinforcement learning (RL) or through simulators that suffer from a severe sim-to-real gap on deformable dynamics, DreamPlan performs RL entirely within the rollouts of an action-conditioned video world model trained from sub-optimal exploratory data (2603.16860).

Framework overview

The method proceeds in three stages. First, a zero-shot VLM planner generates candidate keypoint-based actions, which are executed on the robot to collect diverse action–observation trajectories. Second, this exploratory data—dominated by failures and sub-optimal executions—is used to fine-tune an action-conditioned video diffusion world model $W_\phi$ approximating the transition distribution $\mathcal{P}(o_{t+1} \mid o_t, a_t)$. Third, the planner is optimized against world-model-generated preferences via Odds Ratio Policy Optimization (ORPO), with no further real-world interaction.

A central design choice is the discrete, keypoint-based action space: given keypoints detected independently in the current and goal images, the planner selects a grasp keypoint and a target placement keypoint rather than regressing continuous coordinates. This grounds reasoning in meaningful regions and simplifies downstream optimization.

Action-conditioned video world model

Conditioning a video diffusion model (VDM) on low-level robot actions is the main modeling challenge. DreamPlan renders the kinematic configuration of the robot arms corresponding to each commanded motion sequence and injects these rendered trajectories into CogVideoX-5B (image-to-video) through a ControlNet-style residual branch added to the frozen backbone, enforcing pixel-level alignment between action signals and generated video. To focus capacity on deformation rather than background, lighting, or arm appearance, the model predicts cropped object-only videos rendered on white backgrounds; this design improves PSNR from 24.70 (full-scene prediction) to 26.25 on the final-frame object region.

Two claims deserve emphasis. First, sub-optimal exploratory data suffices: several hundred action sequences collected in roughly four hours of robot interaction yield a stable world model producing plausible deformable rollouts. Second, qualitative comparisons show that strong image-to-video baselines conditioned only by text prompts—CogVideoX-5B-I2V and Wan2.2-I2V-A14B—frequently ignore specified actions and produce physically inconsistent deformations, whereas the rendered-action conditioning yields action-consistent, physically plausible motion. This positions the world model as a usable verifier for planner fine-tuning, which text-prompted baselines are not.

The automated collection pipeline itself is notable: SAM2 masks are propagated across frames to provide grounding, farthest point sampling over masks supplies candidate points, the VLM selects grasp/target pairs per arm, and a fixed Cartesian primitive executes the pick-place cycle. This yielded 2,056 trajectories without continuous human teleoperation.

Reinforcement fine-tuning via Best-of-$K$ preferences

Because a single video diffusion inference takes approximately one minute, online RL inside the generative loop is impractical. DreamPlan decouples the two: for each state–goal pair, $K$ candidate actions are sampled from the planner, the world model rolls out each once, GPT-4o scores predicted outcomes against the goal image, and the best action becomes the positive sample while the rest serve as negatives. ORPO then maximizes the log-odds of preferred over disfavored actions:

$$\mathcal{L}_{\text{ORPO}} = \log\sigma\left(\log \pi_\theta(a^*|s) - \log \pi_\theta(a^-|s)\right)$$

No value function or repeated world-model queries are needed during optimization. The backbone is Qwen3-VL-8B.

Experimental results

Experiments run on a bimanual Franka FR3 platform with a RealSense D435i camera, evaluating three tasks—rope straightening, cloth folding, and toy arm repositioning—with 10 randomized trials per task scored on a {0, 0.5, 1} scale after a single action primitive.

Method	Rope	Cloth	Toy Arm	Avg
Qwen3-VL-4B	0.10	0.15	0.40	0.22
Qwen3-VL-8B	0.20	0.10	0.70	0.33
Qwen3-VL-32B	0.30	0.05	0.70	0.35
GPT-4o	0.30	0.00	0.55	0.28
Qwen3-VL-8B + DreamPlan	0.60	0.35	0.85	0.60

The fine-tuned 8B model roughly doubles the average score of the strongest baseline (0.60 vs. 0.35) and outperforms the substantially larger Qwen3-VL-32B despite its smaller size—an explicit demonstration that physical grounding, not scale, is the binding constraint. Absolute gains over zero-shot counterparts range from 15% to 40% across tasks.

The efficiency comparison against sampling-based explicit verification at deployment is the paper's strongest numerical claim: verifying $N=8$ candidates at inference costs 2,605 seconds per decision at roughly $1.56\times10^5$ TFLOPs for an average score of 0.50, whereas DreamPlan attains 0.60 in about 1 second and 15 TFLOPs—a reduction in inference time of more than three orders of magnitude while also improving performance. The practical implication is that physics-informed supervision can be distilled into the policy weights, eliminating the need for world-model inference at deployment.

Limitations and open questions

Several constraints bound these results. Evaluation uses only 10 trials per task with a coarse three-level score, so differences such as the 0.60 versus 0.55 gap carry substantial sampling uncertainty. The reward signal depends on GPT-4o comparing predicted rollout frames to goal images, inheriting whatever biases a proprietary judge introduces; no analysis of verifier error or its propagation through preference construction is provided. The world model is trained per-task on single-step-horizon primitives, leaving open whether it supports long multi-step imagined rollouts, compounding error behavior, or cross-object generalization beyond cloth, rope, and soft toys. The framework also assumes the fixed pick-place action primitive and bimanual setup; extension to richer action repertoires or different embodiments is untested. Finally, because preference learning optimizes relative odds among candidates sampled from the base planner, performance may remain bounded by the coverage of the initial zero-shot proposal distribution.

Conclusion

DreamPlan demonstrates that a task-specific action-conditioned video world model trained on a few hours of sub-optimal autonomous interaction can serve as both simulator and verifier for offline RL fine-tuning of a VLM planner, yielding large success-rate gains on deformable manipulation at negligible deployment cost. Its efficiency stems from structurally separating expensive diffusion inference from policy optimization via Best-of-$K$ preference construction and ORPO.