UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

Abstract: Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-action (VLA) models without explicit robot action supervision. However, latent actions derived solely from RGB observations primarily encode appearance-driven dynamics and lack explicit 3D geometric structure, which is essential for precise and contact-rich manipulation. To address this limitation, we introduce UniLACT, a transformer-based VLA model that incorporates geometric structure through depth-aware latent pretraining, enabling downstream policies to inherit stronger spatial priors. To facilitate this process, we propose UniLARN, a unified latent action learning framework based on inverse and forward dynamics objectives that learns a shared embedding space for RGB and depth while explicitly modeling their cross-modal interactions. This formulation produces modality-specific and unified latent action representations that serve as pseudo-labels for the depth-aware pretraining of UniLACT. Extensive experiments in both simulation and real-world settings demonstrate the effectiveness of depth-aware unified latent action representations. UniLACT consistently outperforms RGB-based latent action baselines under in-domain and out-of-domain pretraining regimes, as well as on both seen and unseen manipulation tasks.

• The paper introduces UniLARN, a shared-codebook RGB-D inverse and forward dynamics framework that creates unified latent actions encoding both visual appearance and 3D geometry.
• UniLACT improves CALVIN performance from 2.40 to 3.10 under out-of-domain pretraining, a 29.2% relative gain over Moto, while using only RGB at inference.
• Real-world tests show a 10% average success-rate improvement over the RGB-only baseline, although results vary by task and depend partly on estimated depth quality.

UniLACT addresses a specific deficiency in latent action pretraining for vision-language-action (VLA) models: because existing latent action methods derive their action abstractions exclusively from RGB video, the resulting representations encode appearance-driven dynamics while omitting explicit 3D geometric structure. The authors argue this omission is consequential for contact-rich manipulation, where reasoning about reachability, clearance, and collision is essential. Their remedy is to inject depth into the latent action space itself during unsupervised pretraining, rather than using depth only as a pixel- or feature-level auxiliary input as prior depth-augmented VLAs do (2602.20231).

Method: UniLARN and three-stage training

The framework comprises two components trained over three stages. In stage one, UniLARN (Unified Latent Action leaRNing) learns latent actions from paired RGB-D frames via inverse dynamics models (IDMs) and forward dynamics models (FDMs). For each modality $m \in \{\mathrm{r}, \mathrm{d}\}$, an IDM maps an observation pair $(o_t^m, o_{t+H}^m)$ to a continuous embedding that is vector-quantized against a codebook shared across both modalities, yielding modality-specific discrete latents $z_t^m$. These are concatenated, linearly projected, and re-quantized with a second codebook to produce a unified latent $z_t^u$. Crucially, $z_t^u$ conditions modality-specific FDMs that must reconstruct future observations in each modality separately; this disentangled reconstruction objective forces the unified latent to retain complementary appearance (RGB) and geometric (depth) information.

In stage two, the UniLACT transformer — a GPT-2 causal backbone with frozen ViT-L visual encoders and frozen T5 language encoding — is pretrained autoregressively to predict UniLARN's latent tokens conditioned on observations and task instructions, using next-token cross-entropy. In stage three, action query tokens are appended and a lightweight decoder outputs 7-DoF end-effector deltas ($\Delta\mathbf{p} \in \mathbb{R}^3$, $\Delta\mathbf{r} \in \mathbb{R}^3$, binary gripper), trained with L1 regression plus binary cross-entropy on 30 teleoperated demonstrations per task, jointly with the latent prediction loss.

A notable design property is that depth is required only during training; at inference UniLACT consumes static RGB and a task instruction alone. This yields identical model size (89.8M parameters) and per-step latency (27 ms) to the RGB-only baseline Moto (Chen et al., 2024), meaning the geometric gains incur zero inference-time overhead.

Results

On CALVIN ABC$\rightarrow$D, UniLACT reaches an average sequence length of 2.86 under in-domain pretraining versus 2.60 for reproduced Moto, and 3.10 versus 2.40 (+29.2% relative) under out-of-domain Open X-Embodiment pretraining — the paper's strongest headline number. It surpasses baselines with comparable inputs (RT-1, Robo-Flamingo, SuSIE) using only static RGB, though GR-1 retains the top overall score (3.06 in-domain) while relying additionally on gripper-view RGB and proprioception. Task-level analysis indicates RGB latents remain competitive on appearance-driven tasks (stacking, block rotation) while unified latents dominate geometry-centric tasks (slider movement, light bulb).

Real-world experiments on an xArm7 with four tabletop tasks show a +10% average success improvement over Moto (62.5% vs. 52.5%), with gains on three of four tasks including both unseen tasks (+10% lift-block, but −10% on put-eggplant-in-plate). Qualitative examples attribute failures of the RGB baseline to inaccurate depth estimation causing collisions. Because most OXE data lacks native depth channels, the authors generate training depth maps with Depth-Anything-V2 (Yang et al., 2024) — a dependency worth noting, since the geometric signal is thus mediated by a monocular estimator whose errors propagate into the pseudo-labels.

Ablations

Three findings stand out:

• Shared-space learning matters: naive independent supervision by RGB or depth latents (alternating batches) degrades below either single modality (2.08 vs. 2.60/2.40), whereas joint learning in a shared space achieves 2.859. This supports the claim that cross-modal interaction, not merely additional modality data, drives the benefit.
• Asymmetric supervision schedule: pretraining with all three latent target types (U$\rightarrow${U,R,D}) followed by fine-tuning on unified latents only outperforms predicting all types throughout (2.859 vs. 2.299). Cross-modal targets enrich the representation early, but compact unified targets optimize better at fine-tuning.
• Objective interference: multi-task prediction of all latent types within each step underperforms single-target prediction (2.055 vs. 2.859), suggesting joint optimization across latent heads introduces interference.

Limitations and open questions

The paper concedes several constraints implicitly. Real-world evaluation covers only four tasks with 10 trials each, and the one regression (eggplant-in-plate) shows the depth advantage does not transfer uniformly across tasks. Depth supervision relies on a learned monocular depth estimator when sensor depth is unavailable, leaving open how estimator error affects the quality of unified latents at scale. The evaluation uses small-scale models (89.8M parameters); whether the benefit persists for larger VLA backbones is untested. Finally, the mechanism by which shared-codebook quantization aligns RGB and depth semantics is not analyzed directly — no probing or interpretability study isolates what geometric information the unified codes actually capture beyond downstream task success.

Conclusion

UniLACT demonstrates that embedding depth into the latent action representation itself — through a shared-codebook IDM/FDM pipeline producing unified discrete latents — improves VLA policy performance in simulation (+29.2% relative under OOD pretraining) and modestly on real hardware, at no inference-time cost since depth is discarded after pretraining. The work establishes depth-aware latent pretraining as a viable alternative to pixel-level depth integration in VLAs, while leaving open questions of scalability, robustness to estimated depth, and direct analysis of the learned geometric structure.