$\texttt{WEAVER}$, Better, Faster, Longer: An Effective World Model for Robotic Manipulation

Abstract: The potential impacts of world models (WMs, i.e., learned simulators) on robotics are far-reaching -- policy evaluation, policy improvement, and test-time planning -- all with limited real-world interaction. To unlock these downstream capabilities, a WM needs to jointly satisfy three desiderata: $\textit{(i)}$ fidelity (i.e., producing simulated trajectories that correlate with reality), $\textit{(ii)}$ consistency (i.e., producing simulated trajectories that are coherent over long horizons), and $\textit{(iii)}$ efficiency (i.e., producing simulated trajectories quickly). We propose $\texttt{WEAVER}$ (World Estimation Across Views for Embodied Reasoning): a WM architecture that simultaneously achieves all three desiderata, providing state-of-the-art results on robotic manipulation tasks. $\texttt{WEAVER}$ is a multi-view WM trained to predict future latents and reward values via a flow-matching loss. We distill the key design decisions across model architecture, memory, and prediction objectives required to unlock the kinds of long-horizon dynamic manipulation tasks that have confounded prior world modeling approaches. We apply $\texttt{WEAVER}$ in robotic hardware, demonstrating its effectiveness at policy evaluation ($ρ$=0.870 correlation with real-world success rate), policy improvement (real-world success rate improvement of $38\%$ on top of the $π_{0.5}$ robot foundation model), and test-time planning (real-world success rate improvement of $14\%$ with a $5-10\times$ speedup over prior WMs). $\texttt{WEAVER}$ also demonstrates better performance than prior WMs when evaluated on out-of-distribution scenarios. Code, models, and videos at: https://arnavkj1995.github.io/WEAVER/ .

• The paper presents WEAVER, a world model using multi-view encoding, diffusion forcing, and memory augmentation to create high-fidelity, long-horizon rollouts.
• It employs key-value caching and rectified-flow distillation to reduce inference time by up to 16×, enabling real-time test-time planning and robust policy evaluation.
• Empirical results demonstrate WEAVER outperforms baselines in video fidelity metrics and achieves up to 38% improvement in policy success through synthetic data augmentation.

WEAVER: An Effective World Model for Robotic Manipulation

Introduction

This work presents WEAVER (World Estimation Across Views for Embodied Reasoning), a scalable world model (WM) architecture engineered for robotic manipulation that jointly achieves high-fidelity, long-horizon rollouts, and inference efficiency. The paper rigorously positions WEAVER in the context of existing WM architectures and motivates its development around three critical desiderata: (i) high-fidelity predictions correlated with physical reality, (ii) rollouts remaining consistent across long temporal horizons, and (iii) practical inference speed that supports real-time control and planning. WEAVER is empirically validated on a set of five challenging real-world manipulation tasks, outperforming contemporary baselines like Ctrl-World and Dreamer-v4 across standard video fidelity and metric benchmarks, and enables strong policy evaluation, offline policy improvement, and test-time planning.

Figure 1: WEAVER architecture satisfies fidelity, consistency, and efficiency, unlocking real-world downstream applications in policy evaluation, improvement, and test-time planning.

Architecture and Model Design

WEAVER innovates several architectural principles specifically for visuomotor robotic environments:

• Multi-View Encoding: By leveraging information from both external and wrist-mounted cameras, the model reduces partial observability, handles occlusions, and models fine-grained contact dynamics critical for manipulation.
• Pretrained Visual Encoder: The use of the Stable Diffusion 3 VAE accelerates learning, enhances robustness to out-of-distribution (OOD) states, and supports real-to-sim transfer.
• Latent Dynamics via Flow Matching and Diffusion Forcing: The core latent dynamics module predicts long-horizon latent rollouts, optimized with flow matching and regularized using diffusion forcing to sustain coherence under noisy and high-dimensional sensory streams.
• Sparse Memory and Proprioception: Augmented with sparse long-term memory, short-term histories, and proprioceptive state prediction, WEAVER maintains performance on contact-rich and deformable-object tasks that were limiting for image-only or single-view models.

  Figure 2: Overview of the WEAVER architecture, including multi-view encoding, memory/history conditioning, and outcome selection via latent-space reward/critic heads.

Efficient Inference and Value Estimation

The paper addresses the main computational bottlenecks in transformer-based WM inference:

• Key-Value Caching: Memory and history tokens are cached, reducing redundant computation per diffusion step.
• Optimized Noise Schedules: Cosine and power schedules allocate more denoising computation near the clean end of the latent space, yielding higher fidelity for fewer denoising steps.
• Rectified-Flow Distillation: Post-training with a teacher-student setup allows the distilled student to generate high-fidelity rollouts in a fraction of the steps, substantially reducing planning latency.

For value estimation, WEAVER includes a reward model and a value head operating directly on latent representations, avoiding slow decoding and the need for external large vision-LLMs for reward assignment.

Figure 3: WEAVER's predicted latent rewards closely match ground-truth rewards, and advantage-based sample selection enables robust test-time trajectory filtering.

Benchmarking and Quantitative Results

WEAVER delivers marked improvements in both video prediction benchmarks and downstream robotics performance:

• Video Generation Quality: On the DROID and OOD datasets, WEAVER outperforms Ctrl-World in FID, FVD, and LPIPS, especially at low-number-of-function-evaluation (NFE) budgets—a critical regime for real-time use. Inference time per rollout is reduced by up to $16\times$ compared to diffusion baselines without a substantial drop in quality.

  Figure 4: WEAVER enjoys lower FID at every rollout horizon compared to strong baselines.

  

  Figure 5: Pareto frontier of FVD vs. inference time demonstrates WEAVER's superior quality-time tradeoff.

• Policy Evaluation: On long-horizon real rollouts, WEAVER's predicted success rates achieve a Pearson correlation of $\rho=0.87$ with real-world outcomes, exceeding prior architectures. The model is resilient to OOD conditions that degrade other WMs.

  Figure 6: WEAVER robustly tracks the true outcome distribution in both evaluation and OOD settings.

• Policy Improvement (Data Distillation): Policies finetuned with 100% synthetic data filtered via WEAVER's advantage achieve within $4\%$ of real-data finetuned policies, and combinations of real plus synthetic reach higher absolute performance. Improvements are significant ($+38\%$) over foundation policies even without further real-world experience.

(Figures 7, 8, 11)

Figure 7: Visual policy improvement; actions refined using WEAVER-imagined data improve both stability and task completion.

• Test-Time Planning: Batched candidate selection (best-of-$N$ using latent advantage) yields rapid replanning and up to $20\%$ improvement in aggregate success rate over foundation policies, with a practical latency below $2$s per candidate batch, a regime previously infeasible for diffusion-based WMs.

Qualitative Evaluation and Long-Horizon Consistency

Qualitative analyses demonstrate WEAVER's superiority in maintaining rollouts that are physically and visually consistent over 10–20 s horizons, outperforming baselines in challenging scenarios such as stacking deformable objects and pouring granular materials. Notably, WEAVER consistently tracks the passage and final states of invisible objects or materials under occlusion, a challenging aspect for both classical and deep generative models.

(Figure 8, 13, 14, 15, 16, 17, 18)

Figure 9: Example of long-horizon towel placement; only WEAVER maintains consistent, accurate imagination of the scene and task outcome.

Ablations and Further Analysis

Ablation studies support the contributions of each architectural and optimization choice: multi-view input, proprioceptive prediction, advanced noise schedules, and rectified-flow distillation each yield additive improvements to fidelity, consistency, and efficiency. Additionally, results highlight that model-based improvements saturate if the reward model remains weak, emphasizing the continued need for better reward function design or large-scale success/failure annotation.

Limitations and Future Directions

The paper identifies several open challenges:

• Partial Observability: Despite multi-view conditioning, purely visual observations leave ambiguities unsolved; integration of tactile and force signals is posited as a critical next step.
• Physical Priors: Tasks with high-dimensional, non-rigid, or stochastic dynamics (deformables, granulars) expose the limitations of purely data-driven modeling; incorporating explicit physics priors or hybrid models may be necessary for further progress.
• Planning Horizon and Latency: While WEAVER is fast, test-time planning horizons remain bounded by computation; further advances in distillation, hierarchical planning, or more aggressive compression are necessary for deployment in high-frequency or multi-stage, longer-horizon problems.
• Reward Model Noise: Current reward heads are trained on per-frame progress annotations, which may miss subtle error modes; improving robustness and calibration of reward and value estimation is an ongoing research focus.

Conclusion

WEAVER advances the state of the art in robot learning world models by balancing high-generation fidelity, temporal consistency, and computational efficiency, enabling not only realistic video synthesis but also practical control algorithm deployment. Its design enables robust policy simulation, improvement from synthetic data, and fast online test-time planning, backed by strong numerical and qualitative benchmarks. Future research should address multi-modal sensing, better physical inductive bias, and enhance reward supervision for scaling reliable autonomous robot learning.