Xiaomi-Robotics-0: Real-Time VLA Control

• Xiaomi-Robotics-0 is an open-sourced vision-language-action model that integrates a CNN+ViT backbone and diffusion transformer for precise, dexterous robotic control.
• It employs a two-stage pre-training pipeline combining large-scale robot trajectories and image-text pairs to robustly learn multi-modal representations.
• Innovative post-training strategies like Λ-mask and RoPE offset enable asynchronous, near-real-time execution on consumer-grade GPUs.

Xiaomi-Robotics-0 is an open-sourced vision-language-action (VLA) model designed for high-performance, real-time control in robotics. It achieves seamless integration of visual perception, natural language understanding, and precise low-level action generation for dexterous bimanual tasks. The architecture combines state-of-the-art vision-language modeling with a diffusion-based action-generation transformer, leveraging large-scale cross-embodiment data and novel alignment strategies to enable fast, robust deployment on consumer-grade GPU hardware (Cai et al., 13 Feb 2026).

1. Model Architecture and Principal Components

The foundation of Xiaomi-Robotics-0 is a modular architecture comprising a vision-language backbone, a diffusion transformer action head, and a suite of specialized modules for stability and real-time execution.

• Vision–Language Backbone: At its core, the Qwen3-VL-4B-Instruct VLM (4B parameters) processes RGB observations $o_t$ through a hybrid CNN + ViT encoder and language instructions $l$ via a Transformer-based text encoder. Output key-value caches from the VLM condition all downstream modules.
• Diffusion Transformer (DiT) Action Head: The action head consists of a 16-layer DiT that generates continuous action chunks $a_{t:t+T}$ via flow-matching. DiT's input sequence includes a learnable [SINK] token, the robot's proprioceptive state $s_t$ (via MLP), and either noisy or clean-plus-noisy action prefix tokens, depending on training phase. AdaLN layers inject flow-matching timestep $\tau$, and causal self-attention is enforced within each action chunk.
• Special Modules & Embeddings: These include choice-policy action tokens ($[A_i]$ for $i=1...T$) and a score token $[S]$ (for candidate chunk scoring), a learnable attention sink token (for DiT stability), RoPE positional encoding with offsets (disambiguating noisy/clean tokens), and a Λ-shape attention mask (restricting attention post-training to increase policy reactivity).

This compositional architecture fuses multi-modal information through attention over frozen VLM feature caches, enabling rich task- and context-aware action synthesis.

2. Pre-training Datasets and Objectives

Xiaomi-Robotics-0 utilizes an expansive dataset and a two-stage pre-training pipeline:

• Cross-Embodiment Robot Trajectories: Aggregates ≈200 million timesteps from sources such as DROID, MolmoAct, 338 hours of teleoperated Lego Disassembly, and 400 hours of Towel Folding.
• Vision–Language Data: Comprises ≈80 million image-text pairs from Conceptual Captions, CC12M, FineVision, and robot-centric VL slices; supports tasks like grounding, VQA, captioning, and embodied reasoning.
• Two-Stage Pre-training:

1. Co-Training (Stage 1): VLM is co-trained with VL data (next-token cross-entropy) and robot trajectories (via choice-policy supervised learning). Given each step $t$, the VLM predicts $N$ candidate action chunks $l$0 and corresponding scores $l$1. Only the candidate with minimum $l$2 error (to ground-truth chunk) is propagated, promoting robustness under a 1:6 VL-to-trajectory sampling ratio.
2. Flow-Matching on Frozen VLM (Stage 2): VLM parameters are frozen. DiT is trained on pure trajectory data using the loss:

   $l$3

   where $l$4, $l$5, and $l$6 (favoring noisy steps).

This regimen preserves vision-language capabilities while endowing the action head with generalizable skills from diverse robot embodiments.

3. Post-training for Asynchronous Real-Time Execution

To ensure smooth real-world deployment under non-negligible inference latency ($l$7 ms on RTX 4090), Xiaomi-Robotics-0 introduces critical post-training mechanisms:

• Real-Time Control (RTC) Baseline: Prefixing $l$8 past actions attempts to bridge inference windows, but can induce undesirable "action copying," diminishing vision/language reactivity.
• Λ-shape Attention Mask: During post-training, the Λ-mask constrains attention such that noisy tokens $l$9 can attend to the clean prefix (enabling smooth chunk transition), while tokens $a_{t:t+T}$0 cannot, ensuring reliance on VLM features for new decision-making.
• RoPE Offset: Positional indices for noisy tokens are distinctly offset (e.g., $a_{t:t+T}$1), preventing aliasing with clean prefixes.
• Dynamic Loss Re-Weighting: For sampled $a_{t:t+T}$2, samples with larger $a_{t:t+T}$3 prediction error receive higher flow-matching loss weight, inductively focusing training on difficult cases and mitigating policy stagnation.

Together, these augmentations enable asynchronous policy rollouts with high stability and throughput.

4. Deployment Strategies and Real-Time Alignment

Deployment supports both synchronous and asynchronous policies, with explicit mechanisms for timestep alignment:

• Synchronous Execution: Predict and execute the first $a_{t:t+T}$4 steps of each action chunk; robot idles during DiT inference.
• Asynchronous Execution (Preferred): As soon as $a_{t:t+T}$5 steps are executed, the next inference is triggered (conditioned on a rolling prefix of actions), and execution of the chunk tail continues during inference. New action chunks are executed immediately upon availability, provided prefix length $a_{t:t+T}$6 (inference window coverage). Sensor inputs (RGB, proprioception) are synchronized at 30 Hz via timestamp resampling, maintaining temporal coherence.

This strategy eliminates physical idle gaps, sustaining high control frequencies under hardware constraints.

5. Empirical Performance and Benchmarking

Xiaomi-Robotics-0 achieves state-of-the-art results across simulated and real-robot tasks:

Benchmark	Metric	Baseline	Xiaomi-Robotics-0
LIBERO (T=10)	Spat./Obj./Goal/Long/Avg	EO-1: 98.2% Avg	98.7% Avg
CALVIN (T=10, ABCD→D)	Completed Tasks (1–5)	FLOWER: 4.67	4.80
SimplerEnv	Google Robot Matching	EO-1: 76.5%	85.5%
Bimanual Lego	Success Rate (per brick)	~90–95%	~90–95%
Towel Folding	Throughput (pcs/min)	~1.0	1.2

Vision-language capability is maintained: Xiaomi-Robotics-0 matches or exceeds Qwen3-VL-4B on 10 VL + ERQA benchmarks, and ablation without VL data erases all VL ability.

In comparative ablations, synchronous variants may slightly improve grasp success rates on precision-limited tasks but at the expense of throughput. Real-time control baselines exhibit shortcut behaviors in deformable objects, while Xiaomi-Robotics-0's Λ-mask and RoPE offset yield stable and reactive asynchronous policies with maximal throughput.

6. Analysis, Limitations, and Ablation Studies

Ablations elucidate key design contributions:

• Asynchronous vs Synchronous: While synchronous control can mitigate grasp errors under tight tolerances, asynchronous execution with Λ-masking and RoPE offset consistently delivers higher overall throughput and smoother rollouts, especially under variable inference latencies.
• RTC "Copy-Shortcut": Naive RTC settings induce copying artifacts, notably in deformable folding, with repetitive motions and reduced policy novelty.
• Δt_c Sampling & Loss Re-weighting: Randomizing $a_{t:t+T}$7 during post-training promotes DiT robustness to varying prefix lengths, while loss re-weighting by prediction error focuses learning capacity effectively.

A plausible implication is that careful attention modulation and data weighting are essential for high-velocity, low-latency robot control with transformer-based models.

7. Hardware, Software Implementation, and Practical Guidance

Execution is optimized for consumer hardware:

• Hardware: Single NVIDIA GeForce RTX 4090 (24 GB VRAM).
• Inference Latency: Approximately 80 ms for five flow-matching steps plus full VLM and DiT forward pass.
• Software Stack: DeepSpeed ZeRO-2 for scalable training, PyTorch with FlashAttention. Total model size is approximately 4.7 billion parameters.
• Sensor Synchronization: All vision and proprioceptive inputs are resampled to a 30 Hz common clock, critical for deployment synchrony.
• Chunk and Prefix Tuning: Recommended action chunk length $a_{t:t+T}$8 (1 s at 30 Hz); set $a_{t:t+T}$9 to bridge inference cycles.

Open-source code and checkpoints are hosted at https://xiaomi-robotics-0.github.io, facilitating reproduction and further research (Cai et al., 13 Feb 2026).

---

Xiaomi-Robotics-0 demonstrates that a large-scale, diffusion-transformer-based real-time VLA stack—trained with cross-embodiment and VL data, and carefully aligned for inference latency—achieves state-of-the-art results in both simulation and real-robotics, preserving general vision-language abilities while maximizing throughput under hardware constraints.