Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments

Abstract: Embodied intelligence is often studied through specialized models for individual tasks such as manipulation or navigation, resulting in fragmented capabilities and limited generalization across tasks, environments, and robot embodiments. In this work, we study whether heterogeneous embodied decision-making problems can be unified within a single vision-language-action model. We present Qwen-VLA, a unified embodied foundation model that extends Qwen's vision-language modeling stack from perception, understanding, and reasoning to continuous action and trajectory generation through a DiT-based action decoder. Qwen-VLA is trained with a large-scale joint pretraining recipe over diverse data sources, including robotics manipulation trajectories, human egocentric demonstrations, synthetic simulation data, vision-and-language navigation data, trajectory-centric supervision, and auxiliary vision-language data. To support multiple robot platforms, we introduce embodiment-aware prompt conditioning, where robot-specific textual descriptions specify the current embodiment and control convention. We further cast manipulation, navigation, and trajectory prediction into a unified action-and-trajectory prediction framework, enabling transferable visual grounding, spatial reasoning, and continuous action generation across robot morphologies, task families, and environments. Experiments on manipulation, navigation, and trajectory-centric benchmarks show consistent multi-task performance and out-of-distribution generalization under variations in scene layout, background, lighting, object configuration, and robot embodiment. Qwen-VLA-Instruct achieves 97.9% on LIBERO, 73.7% on Simpler-WidowX, 86.1%/87.2% on RoboTwin-Easy/Hard, 69.0% OSR on R2R, 59.6% SR on RxR, 76.9% average OOD success in real-world ALOHA experiments, and 26.6% zero-shot success on DOMINO dynamic manipulation.

• The paper introduces a unified framework that casts embodied decision-making as conditional prediction within a multimodal action space.
• It employs a novel DiT-style action decoder and a staged training curriculum (T2A, CPT, SFT, RL) to integrate visual, linguistic, and embodiment cues.
• Empirical results demonstrate strong out-of-distribution performance on manipulation and navigation tasks across diverse robot platforms.

Unifying Embodied Intelligence: The Qwen-VLA Vision-Language-Action Model

Introduction and Motivation

Qwen-VLA presents a unified vision-language-action (VLA) framework designed to generalize across manipulation, navigation, and trajectory-centric embodied AI tasks on a wide array of robot platforms. Addressing the fragmentation of prior approaches—where specialist models are trained for isolated tasks or embodiments—Qwen-VLA formulates embodied decision making as conditional prediction in a unified action-and-trajectory space, grounding all predictions in multimodal (visual, linguistic, embodiment) context.

Figure 1: Overview of Qwen-VLA, a unified embodied model trained on mixed manipulation, navigation, and vision-language understanding data to generate both robot actions and textual responses.

Key architectural innovations include an embodiment-aware prompting scheme, a DiT-style (Diffusion Transformer) flow-matching action decoder, and a single multimodal backbone for all tasks. The model is trained with a staged curriculum, spanning text-to-action compression, multimodal grounding, multi-task supervised fine-tuning, and reinforcement learning, on a heterogeneous data mixture covering robot/human trajectories, synthetic simulations, and VLN corpora.

Model Architecture and Unified Formulation

Qwen-VLA's architecture consists of a natively multimodal backbone (Qwen3.5) and a DiT-based policy head. All tasks, regardless of embodiment or output structure, are cast as predicting a target action sequence $y_{t:t+H-1}$ given current observation $o_t$, language instruction $x$, and embodiment description $e$. Robot and human trajectories, navigation waypoints, and trajectory-centric motion are represented in a unified masked tensor; platform-specific details (e.g., control frequency, action space) are encoded in textual prompts prepended to each sample.

This generalization is facilitated by zero-padded channel layouts and per-sample validity masks, enabling masking of irrelevant output dimensions while preserving compatibility with the shared DiT action decoder. The model thus supports simultaneous multi-task training without architecture modifications.

Training Curriculum and Data Strategy

The staged training pipeline is critical for stability and multi-domain transfer:

Figure 2: Qwen-VLA’s curriculum: T2A pretrains the DiT decoder from language/embodiment alone, CPT adds vision grounding, SFT aligns with curated demonstrations, and RL directly optimizes task success.

• Stage I—Text-to-Action Pretraining (T2A): Only language and embodiment prompts are used as inputs. The DiT action decoder is trained to reconstruct high-D trajectories, learning a language-indexed action prior. This decouples semantics from vision and ensures the decoder learns rich, compositional structural mappings before overfitting to visual cues.
• Stage II—Continued Pretraining (CPT): Vision is introduced, and both the backbone and DiT are trained jointly. This stage grounds the DiT’s action prior in visual perception, leveraging the full heterogeneous mixture of real, simulated, and synthetic data.
• Stage III—Supervised Fine-Tuning (SFT): High-quality demonstrations from target tasks and robot platforms are used to specialize the policy head and reinforce detailed action conventions, with data mixtures balanced across tasks and embodiments.
• Stage IV—Reinforcement Learning (RL): Closed-loop task success in simulation is directly optimized through PPO/GAE, further refining execution competence.

Data Mixture and Embodiment Prompting

The success of unified modeling hinges on heterogeneous, high-coverage training corpora. Qwen-VLA draws from:

• Extensive real and simulated robot manipulation (74%)
• Egocentric human demonstrations, with structured pose data (6%)
• Synthetic, automatically verified simulation via RoboInF (3.7%)
• Navigation and vision-language navigation datasets (7.5%)
• Auxiliary vision-language and spatial grounding corpora (9.8%)

  Figure 3: Dataset examples generated with RoboInF; diverse “short-” and “long-horizon” tasks systematically compositionalized for the synthetic simulation set.

Each sample carries an embodiment-specific text prompt describing the platform, control conventions, and other relevant meta-data. This enables the single DiT policy head to adapt to platform-specific semantics without explicit rigidity or fragmentation.

Empirical Results and Out-of-Distribution Generalization

Qwen-VLA is extensively evaluated on manipulation (single/dual-arm, tabletop/humanoid, bimanual, deformable), navigation, and dynamic manipulation benchmarks, both in simulation and on real robots (ALOHA platform). Notable findings include:

• Generalist Outperforms Most Specialists: The Qwen-VLA-Instruct multi-task model achieves 97.9% on LIBERO (single-arm manipulation), 73.7% on Simpler-WidowX (real-to-sim), 86.1/87.2% on RoboTwin-Easy/Hard (bimanual), and strong results on RoboCasa-GR1.
• Robust Real-World Transfer: On real-robot ALOHA tasks, pretraining raises OOD (out-of-distribution) success from 36.2% (no pretraining) to 76.9% with pretraining. Gains are especially pronounced for color, instance, background, and instruction OODs.

  Figure 4: Schematic overview of ALOHA platform tasks for real-world evaluation: pick/place, cleaning, stacking, folding, fine manipulation.

  

  Figure 5: Qwen-VLA-Base demonstrates out-of-distribution generalization on the ALOHA dual-arm robot with color-conditioned grasping, novel objects, unseen backgrounds, and compositional instructions.

• Navigation: On VLN-CE, Qwen-VLA-Instruct attains state-of-the-art (e.g., 59.6% SR on RxR Val-Unseen), matching or exceeding highly optimized specialist baselines.
• Dynamic Manipulation: On the DOMINO benchmark, Qwen-VLA-Instruct achieves 26.6% SR and 39.5 MS (manipulation score), outperforming strong methods fine-tuned for dynamic tasks, despite only using current-frame observations and no tailored adaptation to dynamic motion.

Analysis and Ablation

A series of controlled ablations probe key design choices:

Figure 6: T2A pretraining ablations. Key results: mixing synthetic/real data (20/80) achieves the best SFT performance; full-sequence > chunked; vision-free T2A is essential; Sigmoid-Normal at T2A and Beta at SFT is optimal; extended T2A brings diminishing returns or overfitting.

• T2A Stage: Mixing 20% synthetic and 80% real data maximizes SFT performance. Full-sequence supervision and vision-free T2A are superior; chunked, vision-conditioned, or overtuned T2A degrade transfer.
• RL Post-Training: RL optimization in a single environment transfers mild gains to other domains without catastrophic forgetting.

  Figure 7: VL+VLA co-training provides clear gains on complex, compositional benchmarks. Pretrained DiT decoders converge faster, showing efficient transfer.

• Vision-Language Co-Training: VL data does not interfere with embodied learning and yields clear gains in object-centric and compositional tasks. Pretrained DiT significantly accelerates and stabilizes convergence on new backbones compared to from-scratch baselines.

Other findings:

• Multi-embodiment training with zero-padding/validity-masks maintains specialist-level performance while slashing architectural complexity.
• State conditioning offers only marginal benefits versus raw visual-only context, given rich multi-view input and relative action decoding.

Implications and Future Directions

Qwen-VLA demonstrates that a single, unified VLA model can achieve robust, generalizable embodied control across a broad spectrum of tasks, environments, and embodiments. The critical insight is that the action prediction landscape for manipulation, navigation, and trajectory-centric problems can be jointly modeled via architectural and data curation strategies—eschewing rigid output heads and per-task fragmentation.

Key practical implications include:

• Multi-embodiment scaling: Unified models with prompt-conditioned adaptation offer practical, maintainable deployment routes for heterogeneous robot fleets.
• Reduced need for task-specific engineering: Transferable visual-linguistic grounding and action priors mitigate data scarcity in downstream, long-tail, or OOD settings.
• Foundation for next-generation agents: The approach enables seamless scaling to new tasks, platforms, and multimodal inputs, defining a flexible interface for embodied AI agents.

Theoretically, Qwen-VLA reinforces the view that embodied skills can emerge from large-scale, diverse pretraining when combined with structured, staged optimization strategies—reminiscent of recent advances in generalist LLMs and VLMs. Future work remains to close gaps in long-horizon planning, rare-object generalization, real-world robustness, and integration of physical feedback (tactile, force, etc.), as well as scaling autonomous interaction data via autonomous data engines and simulation.

Conclusion

Qwen-VLA substantiates a robust framework for generalist, prompt-conditioned embodied control, achieving strong in- and out-of-distribution performance across manipulation and navigation benchmarks. Its staged pretraining, unified architecture, and rigorous data strategy provide a scalable path for embodied agents to map perception and language into executable continuous actions with high transferability across tasks, platforms, and environments (2605.30280).