XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations (2511.02776v1)

Abstract: Recent progress in large-scale robotic datasets and vision-LLMs (VLMs) has advanced research on vision-language-action (VLA) models. However, existing VLA models still face two fundamental challenges: (i) producing precise low-level actions from high-dimensional observations, (ii) bridging domain gaps across heterogeneous data sources, including diverse robot embodiments and human demonstrations. Existing methods often encode latent variables from either visual dynamics or robotic actions to guide policy learning, but they fail to fully exploit the complementary multi-modal knowledge present in large-scale, heterogeneous datasets. In this work, we present X Robotic Model 1 (XR-1), a novel framework for versatile and scalable VLA learning across diverse robots, tasks, and environments. XR-1 introduces the \emph{Unified Vision-Motion Codes (UVMC)}, a discrete latent representation learned via a dual-branch VQ-VAE that jointly encodes visual dynamics and robotic motion. UVMC addresses these challenges by (i) serving as an intermediate representation between the observations and actions, and (ii) aligning multimodal dynamic information from heterogeneous data sources to capture complementary knowledge. To effectively exploit UVMC, we propose a three-stage training paradigm: (i) self-supervised UVMC learning, (ii) UVMC-guided pretraining on large-scale cross-embodiment robotic datasets, and (iii) task-specific post-training. We validate XR-1 through extensive real-world experiments with more than 14,000 rollouts on six different robot embodiments, spanning over 120 diverse manipulation tasks. XR-1 consistently outperforms state-of-the-art baselines such as $π_{0.5}$, $π_0$, RDT, UniVLA, and GR00T-N1.5 while demonstrating strong generalization to novel objects, background variations, distractors, and illumination changes. Our project is at https://xr-1-vla.github.io/.

• The paper introduces Unified Vision-Motion Codes (UVMC) as discrete latent representations that enable precise alignment between visual dynamics and robotic motion.
• It employs a three-stage training paradigm using a dual-branch VQ-VAE and large-scale, heterogeneous datasets to achieve cross-modality and cross-embodiment generalization.
• Experimental results on 120+ tasks across diverse robotic platforms demonstrate strong few-shot adaptation and superior performance over state-of-the-art baselines.

XR-1: Unified Vision-Motion Representations for Versatile Vision-Language-Action Robotics

Introduction and Motivation

XR-1 introduces a scalable framework for vision-language-action (VLA) learning, targeting the persistent challenges in robotic policy design: precise low-level action generation from high-dimensional sensory inputs and robust generalization across heterogeneous data sources and robot embodiments. The core innovation is the Unified Vision-Motion Codes (UVMC), a discrete latent representation learned via a dual-branch VQ-VAE, which jointly encodes visual dynamics and robotic motion. This approach enables multimodal alignment and cross-embodiment control, facilitating the exploitation of large-scale, heterogeneous datasets including both robot and human demonstration videos.

Figure 1: Overview of XR-1, highlighting the UVMC latent space and the three-stage training paradigm for cross-embodiment, cross-modality, and cross-data learning.

Methodology: Three-Stage Training Paradigm

Stage 1: Learning Unified Vision-Motion Codes

The dual-branch VQ-VAE architecture consists of a vision encoder (SigLIP + ViT) and a motion encoder (causal convolutions + transformer), both mapping their respective modalities into a shared discrete codebook. The vision branch predicts future frames from image pairs, capturing temporal dynamics, while the motion branch reconstructs action sequences from proprioceptive and action data. A cross-modality alignment loss (KL divergence) regularizes the latent spaces, ensuring that visual codes are grounded in motor dynamics, which is critical for leveraging action-free human demonstrations.

Stage 2: UVMC-Guided Pretraining

UVMC codes are injected into the VLA policy backbone (e.g., PaliGemma or Florence-2) via learnable tokens, serving as auxiliary supervision during large-scale pretraining. The policy is trained to predict both UVMC and actions, encouraging the backbone to internalize structured vision-motion representations. This design is model-agnostic and supports integration with various VLA architectures.

Stage 3: Task-Specific Post-Training

The pretrained policy is fine-tuned on embodiment-specific datasets, adapting to the dynamics and morphology of the target robot. This stage sharpens performance for downstream tasks while retaining the generalization benefits of UVMC pretraining.

Experimental Setup and Dataset Curation

XR-1 is evaluated on six robot embodiments (Tien Kung 1.0/2.0, Single-/Dual-Arm UR-5e, Dual-Arm Franka, AgileX Cobot Magic 2.0) across 120+ manipulation tasks and 14,000+ rollouts. The pretraining dataset aggregates Open-X, RoboMIND, Ego4D, and XR-D, totaling over 1.2M episodes and 110M frames, with balanced sampling to prevent overfitting to dominant sources.

Figure 2: Experimental setup showing the diversity of robot embodiments and manipulation tasks used for XR-1 evaluation.

Figure 3: Overview of the pretraining datasets, illustrating the scale and heterogeneity leveraged for UVMC learning.

Results: Performance, Adaptation, and Generalization

XR-1 consistently outperforms state-of-the-art baselines ($\pi_{0.5}$, $\pi_0$, RDT, UniVLA, GR00T-N1.5) across all embodiments and task types, including bimanual collaboration, dexterous manipulation, deformable object handling, contact-rich interactions, dynamic environments, and long-horizon tasks.

Figure 4: Representative evaluation tasks, demonstrating the breadth of manipulation scenarios addressed by XR-1.

Fast Adaptation and Embodiment Transfer

XR-1 demonstrates strong few-shot adaptation: on unseen robots (e.g., Tien Kung 2.0), it achieves high success rates on 15 novel tasks with only 20-shot demonstrations per task, outperforming baselines trained per task.

Figure 5: Fast adaptation results on Tien Kung 2.0, showing XR-1's superior performance with limited demonstrations on an unseen embodiment.

Figure 6: Fast adaptation on Dual-Arm UR5e, highlighting XR-1's ability to generalize and adapt across tasks and embodiments.

Generalization to Out-of-Distribution Scenarios

XR-1 exhibits robust generalization to novel objects, background variations, distractors, and illumination changes, with significant gains over $\pi_0$ in all tested conditions. The UVMC latent space enables embodiment-agnostic knowledge transfer, as evidenced by ablation studies showing a 15% improvement in cross-embodiment skill learning.

Implementation Considerations

• Model Scale: Main XR-1 instantiation uses PaliGemma (SigLIP + Gemma, ~3B parameters); XR-1-Light uses Florence-2 (~230M parameters) for resource-constrained deployment.
• Training Cost: Pretraining requires ~38,400 GPU hours per stage on 80 A100 GPUs; fine-tuning per embodiment uses 576 GPU hours on 8 A100s.
• Inference: Action chunk inference at 5 Hz, action-level inference at ~200 Hz on a single RTX 4090 GPU.
• Data Efficiency: Monotonic scaling law observed; increasing pretraining data volume directly improves downstream performance.

Theoretical and Practical Implications

XR-1's UVMC framework advances the field by demonstrating that joint vision-motion latent spaces, regularized for cross-modal alignment, are essential for scalable, generalist robotic policies. The ability to leverage heterogeneous data—including action-free human videos—addresses a major bottleneck in data collection for robotics. The model-agnostic design and demonstrated transferability suggest that UVMC-based supervision can be broadly adopted in future VLA architectures.

The strong empirical results, especially in few-shot adaptation and cross-embodiment transfer, challenge the prevailing reliance on unimodal or action-centric latent representations. The ablation studies provide evidence that multimodal alignment is not only beneficial but necessary for robust generalization in embodied AI.

Future Directions

Potential avenues for further research include:

• Extending UVMC to incorporate additional sensory modalities (e.g., tactile, audio).
• Investigating hierarchical or compositional latent spaces for more complex task decomposition.
• Scaling to broader embodiment classes (e.g., mobile robots, aerial drones).
• Exploring online adaptation and continual learning with UVMC-guided policies.

Conclusion

XR-1 establishes a new paradigm for vision-language-action learning in robotics by introducing Unified Vision-Motion Codes and a three-stage training pipeline. The framework achieves robust, scalable, and generalizable control across diverse robots and tasks, outperforming leading baselines and demonstrating strong adaptation and generalization capabilities. The results underscore the importance of multimodal alignment and cross-domain exploitation for the development of general-purpose embodied agents.