Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model (2510.27607v1)

Abstract: Recently, augmenting Vision-Language-Action models (VLAs) with world modeling has shown promise in improving robotic policy learning. However, it remains challenging to jointly predict next-state observations and action sequences because of the inherent difference between the two modalities. To address this, we propose DUal-STream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict and enhances the performance of VLAs across diverse tasks. Specifically, we propose a multimodal diffusion transformer architecture that explicitly maintains separate modality streams while still enabling cross-modal knowledge sharing. In addition, we introduce independent noise perturbations for each modality and a decoupled flow-matching loss. This design enables the model to learn the joint distribution in a bidirectional manner while avoiding the need for a unified latent space. Based on the decoupling of modalities during training, we also introduce a joint sampling method that supports test-time scaling, where action and vision tokens evolve asynchronously at different rates. Through experiments on simulated benchmarks such as RoboCasa and GR-1, DUST achieves up to 6% gains over baseline methods, while our test-time scaling approach provides an additional 2-5% boost. On real-world tasks with the Franka Research 3, DUST improves success rates by 13%, confirming its effectiveness beyond simulation. Furthermore, pre-training on action-free videos from BridgeV2 yields significant transfer gains on RoboCasa, underscoring DUST's potential for large-scale VLA pretraining.

• The paper introduces Dual-Stream Diffusion (DUST), a framework that augments VLA models with explicit world-modeling by decoupling action and observation streams for bidirectional causal reasoning.
• The methodology utilizes a multimodal diffusion transformer with independent noise schedules and asynchronous joint sampling, yielding up to 13% higher success rates in real-world robotic tasks.
• Experimental results on simulated and real-world benchmarks demonstrate enhanced data efficiency and superior performance through transfer learning from action-free videos and tailored inference strategies.

Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model

Introduction and Motivation

The paper introduces Dual-Stream Diffusion (DUST), a framework for augmenting Vision-Language-Action (VLA) models with explicit world-modeling capabilities. The motivation stems from the limitations of existing VLA architectures, which, despite leveraging powerful vision-LLMs (VLMs) for perceptual grounding and instruction following, often fail to capture the causal dynamics between actions and their effects on the environment. Prior approaches either concatenate modalities into a unified latent space or employ causal, uni-directional conditioning, both of which suffer from modality mismatch and restricted information flow.

DUST addresses these challenges by maintaining separate streams for action and future observation modalities, enabling bidirectional cross-modal knowledge transfer via shared attention mechanisms, and employing independent noise schedules and decoupled flow matching losses during training.

Figure 1: Comparison of world-model augmented VLA architectures: unified joint diffusion, causal diffusion, and dual-stream diffusion.

DUST Architecture

DUST is built upon a multimodal diffusion transformer (MMDiT) backbone, which processes action and vision tokens in parallel, with modality-specific timestep embeddings and normalization. The architecture consists of:

• A frozen VLM (Eagle-2) for extracting semantic features from current observations and task instructions.
• Separate token streams for actions and future visual observations, each processed through dedicated pathways.
• Shared cross-modal attention layers for bidirectional information exchange.
• Modality-specific DiT blocks for fine-grained denoising and specialization.

This design avoids the pitfalls of a unified latent space, allowing each modality to retain its statistical structure while facilitating causal reasoning between actions and future states.

Figure 2: Dual-stream diffusion (DUST) architecture with separate action and vision streams, cross-modal attention, and modality-specific denoising.

Decoupled Training Algorithm

The training procedure leverages independent noise schedules for each modality, inspired by diffusion forcing. For each training sample, separate timesteps $\tau_A$ and $\tau_o$ are sampled for actions and future observations, respectively. This enables the model to learn both forward and inverse causal relationships:

• Predicting future observations from noisy actions (forward dynamics).
• Inferring actions that could have led to a given future state (inverse dynamics).

The joint objective is decomposed into unimodal flow matching losses, weighted by a hyperparameter $\lambda_{\textrm{WM}}$, allowing for flexible balancing between action and world-modeling objectives.

Asynchronous Joint Sampling and Test-Time Scaling

During inference, DUST employs an asynchronous joint sampling strategy, updating vision tokens more frequently than action tokens. This is motivated by the higher dimensionality and complexity of image embeddings, which benefit from more denoising steps, whereas action tokens converge with fewer steps. The ratio $q$ controls the number of vision updates per action update, enabling a tunable trade-off between inference speed and accuracy.

Figure 3: Overview of vision-action joint sampling, illustrating asynchronous denoising of vision and action tokens.

Experimental Results

Simulated Benchmarks

DUST is evaluated on RoboCasa and GR-1, two challenging simulated environments for robotic manipulation. Across all data regimes (100, 300, 1000 demonstrations per task), DUST consistently outperforms GR00T-N1.5 and FLARE baselines, achieving up to 6% higher success rates. The gains are robust across pick-and-place, contraption open/close, and miscellaneous tasks.

Real-World Robotic Manipulation

On the Franka Research 3 platform, DUST demonstrates a 13% improvement in success rate over baselines across diverse pick-and-place tasks, confirming its effectiveness in real-world settings. Qualitative rollouts show that DUST's explicit world-modeling enables more precise end-effector alignment and physically consistent predictions.

Figure 4: Example pick-and-place task setup for real-world evaluation.

Figure 5: Qualitative comparison on a real-world pick-and-place task, highlighting DUST's superior alignment and grasping.

Transfer Learning from Action-Free Video

DUST's dual-stream design facilitates pretraining on large-scale action-free video datasets (BridgeV2), optimizing only the world-modeling loss. When finetuned on RoboCasa, DUST achieves significant gains in data efficiency and generalization, demonstrating the utility of passive video data for downstream policy learning.

Test-Time Scaling

Increasing the number of vision denoising steps ($N_o$) while keeping action steps fixed yields additional 2–5% improvements in success rates, validating the asynchronous joint sampling strategy. Synchronous scaling of both modalities degrades performance, underscoring the importance of modality-specific inference schedules.

Ablation Studies

Ablations confirm that both the dual-stream MMDiT architecture and decoupled noise scheduling are essential for optimal performance. The best results are obtained with 12 MMDiT layers and a balanced loss weighting ($\lambda_{\textrm{WM}} \in [0.5, 2.0]$). Increasing vision denoising steps without increasing action steps is critical for leveraging the high-dimensional structure of image embeddings.

Figure 6: Modified MMDiT block with separate timestep embeddings for each modality.

Implementation Considerations

• Computational Requirements: Training on RoboCasa and GR-1 requires significant GPU resources (A100/H200), with batch sizes and iteration counts tuned for convergence.
• Architecture Modularity: DUST's design is compatible with any VLM backbone and can be extended to additional modalities (e.g., audio, tactile).
• Deployment: The asynchronous sampling strategy allows for real-time inference with adjustable accuracy-speed trade-offs, suitable for practical robotic systems.

  Figure 7: Real-world experimental setting with Franka Research 3 and multi-view cameras.

Implications and Future Directions

DUST advances the state-of-the-art in world-model augmented VLA models by resolving modality conflicts and enabling scalable, bidirectional integration of action and observation streams. The framework's compatibility with large-scale video pretraining opens avenues for leveraging internet-scale passive data, reducing reliance on expensive robot demonstrations. Future work may explore:

• Extension to multi-agent and multi-modal settings.
• Integration with reinforcement learning for closed-loop policy optimization.
• Further scaling of world-modeling capacity via larger VLMs and more expressive diffusion architectures.

Figure 8: Example GR-1 rollouts demonstrating dexterous manipulation.

Figure 9: Example RoboCasa rollouts for diverse household tasks.

Figure 10: Example Franka Research 3 rollouts for real-world pick-and-place tasks.

Conclusion

Dual-Stream Diffusion (DUST) establishes a robust and extensible paradigm for world-model augmented VLA models, achieving superior performance in both simulated and real-world robotic manipulation. Its architectural innovations, training methodology, and inference strategies collectively address longstanding challenges in cross-modal integration and causal reasoning. DUST's demonstrated scalability, transferability, and practical deployment potential mark it as a significant contribution to the field of embodied AI and robotics.