RLDX-1: Unified VLA for Dexterous Robotics

• RLDX-1 is a unified vision-language-action framework that integrates motion, long-term memory, and physical sensing for advanced robotic dexterity.
• It employs a multi-stream action transformer and a synthetic data pipeline to robustly handle complex, contact-rich manipulation tasks.
• Optimized with aggressive inference strategies, RLDX-1 consistently outperforms prior VLA systems in both simulated and real-world benchmarks.

RLDX-1 is a unified Vision-Language-Action (VLA) framework for general-purpose robotic dexterous manipulation that integrates motion awareness, long-term memory, and physical sensing into a single learned robotic policy. Built on the Multi-Stream Action Transformer (MSAT) architecture, RLDX-1 is designed to overcome the limitations of prior VLAs in complex real-world tasks that demand broad functional capabilities beyond general versatility. RLDX-1 features a temporally- and physically-augmented VLM backbone, a synthetic data synthesis pipeline for rare manipulation scenarios, a three-stage training scheme specialized for human-like dexterous skills, and aggressive inference optimization enabling real-time deployment. Empirical results across standard simulation and real-robot benchmarks demonstrate RLDX-1's consistent superiority over contemporary VLA systems in tasks requiring dynamic motion, memory, and contact-rich physical interaction (Kim et al., 5 May 2026).

1. Architectural Components

RLDX-1 comprises two principal stages: (1) a temporally-aware Vision-LLM (VLM) backbone and (2) a flow-matching diffusion action model with MSAT-based cross-modal prediction.

• VLM Backbone: Based on Qwen3-VL 8B, the VLM ingests a language prompt and $K+1$ video frames ($H \times W$). It is fine-tuned on robot-specific Visual Question Answering (VQA) data and integrates three augmentations:
  • Motion Module: Mid-level encoder layers implement space–time self-similarity (STSS) for dynamic cue extraction, refining features via $$v_t^{(i)} \to \tilde{v}_t^{(i)} = v_t^{(i)} + \phi_{\mathrm{stss}}(v_t^{(i)})$$.
  • Temporal Compression: Initial LLM blocks process multi-frame tokens, collapsing past frames via average pooling to a single "context" token, lowering computational load.
  • Memory Module: Maintains an $n_{mem}$-length queue of past cognition tokens, processed through a lightweight Transformer to yield a memory feature $m_t$.
• Action and Physics Model: Uses a flow-matching diffusion model to predict both an action chunk $a_{t:t+H}$ and, when available, future physical signals $p_{t+1:t+L}$. The denoising process is handled jointly by the Multi-Stream Action Transformer.
• Multi-Stream Action Transformer (MSAT): MSAT extends MM-DiT to $N$ streams, typically "cognition" (C), "action" (A), and physical "physics" (P). Initial blocks process modality-specific streams independently, while later blocks merge and jointly process all tokens via cross-modal self-attention. The sequence proceeds as follows for each stream $s \in \{C, A, P\}$:
  • Compute query, key, value: $Q_s = W_q^{(s)} X_s$, $H \times W$0, $H \times W$1.
  • Concatenate streams: $H \times W$2, $H \times W$3, $H \times W$4.
  • Compute joint self-attention, split output streams, apply stream-specific FFNs with RMSNorm and SwiGLU. RoPE and a learnable "time" token encode sequence and diffusion timestep information.

This design enables RLDX-1 to explicitly encode and fuse scene, action, and physical context with inductive bias toward each modality.

2. Synthetic Data Generation and Filtering

A specialized synthetic data pipeline addresses the scarcity of rare, dexterous, or contact-rich humanoid scenarios:

• Task Augmentation (VLM-guided): Factorized composition over behavior, object, placement, hand type; skill-primitive conditioning enables re-sampling of scenes and instructions.
• Scene Augmentation: Uses FLUX.2 for I2I editing (Canny-conditioned), Cosmos-Predict2.5 for I2V, and edge-conditioned V2V style transfer to diversify visual context.
• Action Annotation: An inverse dynamics model (IDM) trained by flow-matching on real demonstrations predicts $H \times W$5 for synthetic videos, creating pseudo-labeled actions.
• Automatic Video Filtering:
  • VLM evaluates instruction following and trajectory plausibility (1–5).
  • Motion-consistency is verified by simulating IDM-predicted actions, embedding pairs with V-JEPA2 plus a cross-attention probe, and filtering out misaligned samples.

This pipeline scales the available training scenarios, particularly in functionally sparse regions, improving downstream generalization.

3. Training Protocols

RLDX-1 adopts a three-stage training pipeline:

• Pre-training: Trained for 100K steps (1.5M episodes, batch 8192) on diverse multi-embodiment data, including 150K synthetic GR-1 samples, using flow-matching loss for actions:

$H \times W$6

• Mid-training: 25K steps (batch 1024) on embodiment-specific data, activating the STSS, memory ($H \times W$7), and physics streams, with auxiliary flow-matching loss for physical signals:

$H \times W$8

• Post-training: 30K steps (batch 128) of behavior cloning from teleoperated demos followed by "RECAP-style" RL refinement. A text-based VLM critic $H \times W$9 autoregressively predicts integer-valued advantages, guiding policy updates. Iterative cycles of rollout collection and advantage re-annotation enhance targeted behaviors.

This suggests that curriculum and RL-aided fine-tuning are critical to achieving human-like dexterous skill.

4. Inference Optimization

Several inference-time strategies accelerate deployment on hardware such as RTX 5090:

• Static Graph Conversion: Precomputes RoPE and attention masks, compiling the full forward pass into a single CUDA graph, eliminating launch overhead.
• Kernel Fusion: Custom Triton kernels group frequently adjacent ops (e.g., RoPE → RMSNorm → FlashAttention, AddLayerNorm, AddRMSNorm, SwiGLU, grouped MLPs), reducing global memory operations.

Achieved latency reductions are summarized below:

Configuration	Latency (ms)
PyTorch eager	71.2
CUDA Graph + TorchCompile	59.6
Static graph	48.9
Static graph + kernel fusion	43.7

End-to-end latency improves by 1.63× over the baseline.

5. Empirical Performance and Benchmark Results

RLDX-1 outperforms π₀.₅ and GR00T N1.6 across simulation and real-robot benchmarks, with notable gains on tasks requiring dynamic perception, long-horizon reasoning, and physical manipulation.

• Simulation (selected):
  • LIBERO-Short/Long: 98.6% / 95.3% (π₀.₅: 98.0% / 92.0%; GR00T: 97.4% / 94.4%)
  • RoboCasa Kitchen: 70.6% (π₀.₅: 62.1%; GR00T: 66.2%)
  • GR-1 Tabletop: 58.7% (π₀.₅: 15.4%; GR00T: 47.6%)
• Real-World (OpenArm Humanoid, ALLEX Humanoid, Franka Research 3):
  • ALLEX Humanoid, Conveyor Pick-n-Place (unseen speed): 75.0% (π₀.₅: 29.2%; GR00T: 50.0%)
  • ALLEX, Object-in-Box Selection: 91.7% (π₀.₅: 33.3%; GR00T: 29.2%)
  • Franka Research 3, Shell Game: 91.7% (π₀.₅: 50.0%; GR00T: 50.0%)
  • Card Slide-and-Pick (tactile+torque): 97.2 versus π₀.₅ (~60), GR00T (~50)

Ablations show performance sensitivity to backbone feature extraction layer (e.g., Layer 18: 60.9% success on RoboCasa Kitchen vs. Layer 8: 51.1%) and scaling with synthetic data (GR-1 Tabletop: real-only 41.0%; +100% synth: 50.1%).

6. Analysis, Limitations, and Future Directions

Several key insights derive from RLDX-1's design and evaluation:

• Explicit Motion Modules: STSS and frame compression enable anticipation of dynamic object trajectories, critical for fast-moving scenarios.
• Memory Mechanisms: Retention and transformation of past cognition tokens support long-horizon reasoning in tasks such as object selection and shell games.
• Physics Stream: Auxiliary flow-matching for tactile and torque sensor signals allows nuanced, contact-rich manipulation where vision alone is insufficient.
• MSAT Projections: Separate per-stream projection followed by joint self-attention preserves inductive biases and enables robust modality fusion.

Identified limitations include:

• Architectural size and inference cost, suggesting LoRA or distillation for efficiency.
• Synthetic data efficacy, currently gated by video model fidelity.
• Memory module horizon ($$v_t^{(i)} \to \tilde{v}_t^{(i)} = v_t^{(i)} + \phi_{\mathrm{stss}}(v_t^{(i)})$$03 context chunks), possibly insufficient for ultra-long-horizon reasoning.
• Open questions on safety and robustness in adversarial or highly unstructured environments.

These observations position RLDX-1 as evidence that unifying VLM priors with dedicated modules for motion, memory, and physical sensing, large-scale/synthetic data, and aggressive optimization yields a single VLA policy capable of human-like, contact-rich, real-world dexterous manipulation (Kim et al., 5 May 2026).