Qwen-RobotManip: VLA Model for Manipulation

• Qwen-RobotManip is a unified Vision-Language-Action model that enables cross-task and cross-embodiment robotic manipulation through canonical state–action representations.
• It employs a three-pronged alignment framework combining canonical state-action encoding, camera-frame Δ-pose parameterization, and in-context policy adaptation.
• Pretrained on 38,100 hours of multi-modal data, the model demonstrates emergent zero-shot instruction following, robust recovery, and effective cross-embodiment transfer.

Qwen-RobotManip is a generalizable Vision-Language-Action (VLA) foundation model for robotic manipulation, specifically designed to enable genuine cross-task and cross-embodiment generalization at scale. It directly addresses the alignment and scaling challenges posed by the inherent heterogeneity, cost, and narrowness of robotic manipulation data relative to modalities such as text. Qwen-RobotManip achieves this by introducing a unified alignment framework—across representation, motion, and behavior—that enables coherent large-scale multi-source training. The system leverages a rigorous data curation pipeline and a synthetic human-to-robot demonstration synthesis process to pretrain on an unprecedented corpus, enabling emergent capabilities in zero-shot instruction following, robust recovery, and cross-embodiment transfer, including demonstrated success in real-robot deployment scenarios (Yuan et al., 16 Jun 2026).

1. Model Architecture and Unified Alignment Framework

Qwen-RobotManip builds atop the Qwen-VL vision-language transformer backbone (Qwen3.5-4B) and incorporates a continuous-action “Diffusion Transformer” (DiT) policy head. Three principal alignment innovations are employed to make multi-embodiment training coherent:

• Canonical State–Action Representation: All supported robot embodiments use a shared 80-dimensional action vector, which consists of two 29-dimensional "arm blocks" (7 joint angles, 3D end-effector position, 6D rotation, 1 gripper, 12 dexterous joints per arm), supplemented by 22 reserved dimensions for additional degrees of freedom (e.g., mobile bases). Zero-padding and a per-dimension binary mask $m$ ensure only meaningful slots contribute to gradient updates.
• Camera-Frame Δ-Pose Parameterization: Actions are encoded as small rigid-body transforms (4×4 pose-deltas) in the camera frame rather than in base or EEF-local coordinates, promoting invariance and alignment. The action pose-delta is:

$$a_p = \begin{bmatrix} {}^c_{\!e}R \;{}^e_{\!e^*}R\;{}^e_{\!c}R & {}^c_{\!e}R\,{}^e t_{e^*} \ 0 & 1 \end{bmatrix}$$

where ${}^c_{\!e}R$ and ${}^e_{\!c}R$ are calibration extrinsics.

• In-Context Policy Adaptation: At inference, recent executed history $(o_h, s_h, a_h)$ is appended as tokens to the prompt, functioning as an implicit, contextual robot embodiment identifier. Context tokens are handled through causal attention in the VLM, allowing the DiT head to ground output actions jointly in current observation and history.

The DiT action expert consists of 10 Transformer blocks with alternating self-attention (on state/action) and cross-attention (to vision/language tokens), and is conditioned on timestep $t$, camera embedding (CaPE), EEF-type, and a calibrated-camera flag.

Training Objectives: Qwen-RobotManip is co-trained on two objectives:

• Flow-Matching Loss: For continuous action chunks, the loss is

$$\mathcal{L}_{FM} = \frac{1}{B}\sum_{i=1}^B \frac{\sum_{t,j}m_{i,t,j}(f_\theta(x_{i,t},t_i,s_i,o_i)_j-v_{i,t,j})^2}{\sum_{t,j}m_{i,t,j}}$$

where $x_t = (1-t)\epsilon + t a$, $\epsilon \sim \mathcal{N}(0, I)$, $t \sim \text{Beta}(1, 1.5)$.

• VLM Next-Token Cross-Entropy:

$$a_p = \begin{bmatrix} {}^c_{\!e}R \;{}^e_{\!e^*}R\;{}^e_{\!c}R & {}^c_{\!e}R\,{}^e t_{e^*} \ 0 & 1 \end{bmatrix}$$0

• Total Loss:

$$a_p = \begin{bmatrix} {}^c_{\!e}R \;{}^e_{\!e^*}R\;{}^e_{\!c}R & {}^c_{\!e}R\,{}^e t_{e^*} \ 0 & 1 \end{bmatrix}$$1

2. Data Curation, Human-to-Robot Synthesis, and Pretraining Corpus

Qwen-RobotManip is pretrained on a ~38,100 hour corpus spanning three modalities:

• Real-Robot Demonstrations (~11,000 h): Aggregated from open datasets (e.g., OXE, RoboMIND/2.0, DROID, RH20T, AgibotWorld, RoboCOIN, Galaxea), covering 15 robot platforms (Franka, UR5(e), AgileX ALOHA, ARX-L5, xArm7, Sawyer, Kinova Gen3, IIWA, Jaco, FR3, UR10e, ViperX, WidowX, Piper, YAM).
• Egocentric Human Videos (~1,933 h): Example datasets include EgoDex (829 h), VITRA (~247 h), and EgoVerse (~954 h), with hand pose recovery via MANO mapping.
• Human-to-Robot Synthesized (~24,808 h): Egocentric hand videos are algorithmically retargeted into robot action sequences using:
  • Action alignment by virtual-finger retargeting and Savitzky–Golay trajectory smoothing
  • Base pose search via inverse kinematics feasibility maximization:

  $$a_p = \begin{bmatrix} {}^c_{\!e}R \;{}^e_{\!e^*}R\;{}^e_{\!c}R & {}^c_{\!e}R\,{}^e t_{e^*} \ 0 & 1 \end{bmatrix}$$2 - Visual alignment by background removal (SAM3 + ProPainter), MuJoCo rendering with occlusion compositing. - A five-stage filtering pipeline enforces kinematic, trend, and consistency checks, with cross-modal filtering for instruction-video alignment and quality.

3. Training Regime and Scaling Effects

Qwen-RobotManip employs dual-stream co-training with a 90% manipulation : 10% vision-language data mixture, incorporating 28M examples of general VQA, spatial reasoning, OCR, STEM, instruction following, and embodied chain-of-thought prediction. Training uses batch size ~128, mixed-precision across hundreds of GPUs, and DiT action head amortization via 8-fold noise sampling per batch. Pretraining spans ~1M gradient steps ($$a_p = \begin{bmatrix} {}^c_{\!e}R \;{}^e_{\!e^*}R\;{}^e_{\!c}R & {}^c_{\!e}R\,{}^e t_{e^*} \ 0 & 1 \end{bmatrix}$$3), followed by domain-specific supervised fine-tuning.

Scaling ablations show that only the unified 80d state-action slot architecture and camera-frame EEF parameterization yield log-linear improvements on held-out error as the dataset fraction increases, indicating the necessity of alignment innovations for exploiting scale. Naïve concatenation of data results in non-monotonic or plateauing learning curves.

4. Emergent Generalization and Empirical Evaluation

Qwen-RobotManip demonstrates robust out-of-distribution (OOD) generalization, zero-shot instruction following, and effective cross-embodiment skill transfer in both simulation and on real platforms. Standard IID benchmarks are insufficient to distinguish pretraining effects, necessitating rigorous OOD evaluation.

Key OOD Benchmarks and Results:

Benchmark/Setting	Qwen-RobotManip	π₀.₅	Absolute Δ
LIBERO-Plus (7 pert)	89.0%	84.4%	+4.6
RoboTwin-Clean2Rand Hard	69.4%	47.9%	+21.5
RoboCasa365 Composite-Unseen	14.9%	5.4%	+9.5
EBench Overall SR / Score	45.6% / 60	27.1% / 41	+18.5 / +19
RoboTwin-IF (instruction following)	72.2%	49.6%	+22.6
RoboTwin-XE (cross-embodiment, EEF ctrl)	23.9%	7.5%	+16.4
AgileX ALOHA OOD (real robot)	87.5%	37.5%	+50.0
RoboChallenge Table30-v1 (Generalist)	45% SR/59.8	-	1st / +20%

Emergent skills include robust retry strategies for failed grasps, precise placement in clutter, reactive error correction in long-horizon tasks, and bimanual sequencing.

5. Key Findings, Limitations, and Future Directions

Qwen-RobotManip establishes that a unified alignment framework—combining canonical state-action representation, camera-frame motion encoding, and in-context adaptation—enables foundation models to exploit large, heterogeneous robot and human video datasets for manipulation. The model’s scaling and alignment design allow it to synthesize training from open data sources, making proprietary collection unnecessary.

Summary of Findings:

• IID benchmarks fail to probe real generalization; Qwen-RobotManip’s superiority appears only under OOD testing.
• Synthetic hand-to-robot pipelines and data curation are essential to scale data diversity and quality.
• Ablations confirm that canonical alignment is critical for scaling; without it, learning curves stagnate.

Limitations:

• Synthetic artifacts from the video-to-robot pipeline may limit action fidelity.
• Most evaluation remains in simulation, with some real-world deployments.
• Model operates on fixed-length state/action chunks and is not optimized for sub-100 ms reflexes, imposing latency constraints.

Future work: Prospective advances include biomechanical models for retargeting, expansion to mobile/humanoid morphologies, integration of physical contact simulators, real-world OOD testbed development, and leveraging generative world models for plan-execute closed-loop control (Yuan et al., 16 Jun 2026).

6. Relation to Qwen-VLA and Broader Embodied Foundation Models

Qwen-RobotManip’s approach is characterized by a rigorous focus on alignment innovations tailored for manipulation, while general foundation models such as Qwen-VLA unify vision-language-action modeling across manipulation, navigation, and trajectory domains, leveraging similar architectural (Qwen3.5-4B backbone + DiT expert) and masking strategies. Qwen-VLA’s embodiment-aware prompt conditioning and unified tensor interface support a wide spectrum of embodied tasks, demonstrating multi-task generalization with strong OOD performance on manipulation and navigation (Wang et al., 28 May 2026).

A plausible implication is that the architectural and alignment principles underlying Qwen-RobotManip can be extended to other domains of embodied intelligence, provided the data representations and supervision signals are harmonized in a similarly canonical manner.