What Matters for Latent Actions in Robot Learning

Abstract: Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-LLM (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.

• The paper systematically evaluates 41 latent-action design choices across three simulation benchmarks and a Franka Panda, finding that plain LAPO, 32-dimensional unnormalized latents, and joint latent-action integration provide strong general performance.
• Optical flow consistently underperforms simpler pixel or semantic frame differences, while regularization strength matters more than regularizer type; VAE is a practical default, whereas VQ-VAE improves zero-shot robustness under environmental shifts.
• Latent-action fine-tuning improves real-world success from 64.75% to 79.25% across 400 rollouts, and FDM reconstruction metrics offer useful coarse screening but cannot reliably rank models across latent dimensions.

Latent Action Models (LAMs) promise to bridge the gap between abundant unlabeled video and scarce robot action data by learning compact latent actions that surrogate physical actions. However, research in this area has been fragmented: methods such as LAPO, LAOF, CoMo, StaMo, and Motus each introduce design choices evaluated under inconsistent experimental settings, leaving it unclear which factors genuinely determine downstream manipulation performance. The paper "What Matters for Latent Actions in Robot Learning" (2608.19613) addresses this gap with the first comprehensive empirical study of latent action learning for robotic manipulation. The authors unify representative LAM methods within a common autoencoding framework and systematically evaluate 41 design choices across three dimensions—latent action modeling paradigms, learning objectives and regularization, and latent action integration strategies—on LIBERO, LIBERO-Plus, and RoboTwin2.0, with real-world validation on a Franka Panda platform.

Problem formulation and unified framework

The work formalizes latent action learning as follows: given consecutive frames $(o_t, o_{t+1})$, an inverse dynamics model (IDM) infers a latent action $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$ that captures transition dynamics without access to ground-truth physical actions. To compare heterogeneous methods fairly, the paper introduces a unified notation based on consecutive-frame differences $\Delta_t = \Delta(o_{t+1}, o_t)$, where $\Delta$ may be instantiated as pixel-wise differences ($\Delta$RGB), semantic feature differences ($\Delta$DINO), or optical flow estimated by RAFT or SEA-RAFT.

Two paradigms are distinguished. The implicit IDM-FDM framework jointly trains an IDM and a forward dynamics model (FDM) under self-supervised reconstruction, encompassing LAPO, LAOF (which adds optical flow prediction as auxiliary supervision), and CoMo (which feeds frame differences rather than the future frame to the IDM to mitigate causal leakage). The explicit CFD-AE framework directly autoencodes $\Delta_t$ without future-frame synthesis. All variants share the same 700M spatiotemporal Transformer backbone and identical latent dimensionality, eliminating architectural confounds.

The full training pipeline spans three stages: Stage I pretrains the LAM on video only; Stage II uses the pretrained IDM as an automatic annotator to fine-tune a Qwen3-VL-4B backbone on $(o_t, z_t, l)$ triplets; Stage III post-trains on robot action data for policy learning, following the OpenVLA-OFT recipe.

Latent action modeling paradigms

Averaged across all benchmarks and five integration strategies, the overall ranking is LAPO (0.733) > $\Delta$DINO (0.728) > CoMo (0.717) > $\Delta$RGB (0.697) > LAOF (0.691) > SEA-RAFT (0.643) > RAFT (0.643). Three findings stand out:

First, the original LAPO remains a remarkably strong baseline when trained directly on raw data, requiring no preprocessing. This is a somewhat contrarian result relative to recent proposals that add auxiliary signals or leakage-mitigation mechanisms.

Second, optical flow is not an effective inductive bias for latent action learning. Both RAFT and SEA-RAFT consistently rank last—even below simple $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$0RGB—and adding optical flow supervision to LAPO (LAOF) consistently degrades performance. The authors argue that transforming inter-frame changes into pixel-wise displacements discards visual cues relevant to transition dynamics (contact, occlusion, deformation), while flow estimation errors propagate into the learned representation, particularly under the Noise perturbation of LIBERO-Plus. Notably, better optical flow quality does not translate into better latent actions: RAFT outperforms SEA-RAFT under distribution shift despite producing lower-quality flow.

Third, semantic feature differencing suffices: $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$1DINO consistently outperforms $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$2RGB and approaches or exceeds LAPO on LIBERO, indicating that a strong pretrained semantic prior (DINOv2) can itself supply effective motion structure. Meanwhile, CoMo's anti-leakage modification consistently underperforms LAPO, suggesting that—at the tested dimensionality—the causal leakage concern has limited practical impact, whereas removing future-frame information from the IDM compromises its modeling capacity.

Regularization: type matters less than strength

Across continuous regularizers—VAE (KL), Sparsity (with VCM), and SIGReg (Epps–Pulley statistics on random one-dimensional projections)—downstream success rates on LIBERO span only 0.908–0.915 when strengths are tuned appropriately. The regularization strength is far more consequential than its form: excessively large coefficients suppress transition dynamics, degrading both FDM reconstruction metrics and probe losses. Recommended settings are $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$3 for VAE, $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$4 for Sparsity, and $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$5 for SIGReg, with VAE preferred for simplicity and SIGReg penalized for slower training.

Discrete regularization behaves differently. VQ-VAE achieves the strongest zero-shot generalization on LIBERO-Plus (0.517) versus AE (0.445), VAE (0.433), Sparsity (0.468), and SIGReg (0.467). The discrete bottleneck appears to encourage reusable action primitives that transfer under environmental perturbation—but this advantage does not carry over to in-distribution benchmarks, where discretization's reduced continuity limits gains on saturated tasks. The practical guidance is therefore split: continuous VAE by default, VQ-VAE when zero-shot robustness dominates.

Integration strategies

Five action head architectures are compared: DAP (latent actions used only for backbone fine-tuning, then discarded), LAP (latent-to-action as an intermediate code), JAP (joint parallel prediction of both), and hybrids JAP-DAP/JAP-LAP. A clear ordering emerges: DAP is weakest, LAP improves substantially, and JAP performs best. The hybrid results localize the source of improvement: because JAP-DAP and JAP-LAP retain their advantage even after the joint head is replaced, the primary benefit comes from how joint optimization shapes the backbone representation during Stage II/III, not from the head architecture itself. The design principle is that latent actions should remain involved throughout downstream training as a structured auxiliary objective, not serve solely as a pre-training target. Concretely, LAP is recommended when physical actions are unavailable during backbone fine-tuning; JAP-LAP when they are accessible.

Proxy metrics are useful but insufficient

Four proxy metrics are examined: Linear and MLP probes decoding physical actions from latent actions, and two normalized FDM reconstruction metrics, SSIM Gain and MSE Gain, measuring how much better the FDM predicts the future frame than copying the current frame. Correlation analysis over 29 configurations yields a nuanced verdict. Pearson correlations are high, but Spearman correlations are markedly lower—these proxies support coarse-grained screening, not fine-grained ranking, and cannot reliably identify the single best model.

FDM reconstruction metrics correlate more strongly with downstream performance than probe metrics, plausibly because they directly assess world-modeling capability, whereas probes entangle decoder capacity and annotation ambiguity. All correlations deteriorate when comparing models across different latent dimensionalities, since higher-dimensional latents reduce probe error via capacity alone and may enable reconstruction shortcuts; cross-dimensionality comparisons should be avoided. An additional observation ties proxy behavior to control modality: end-effector delta control (LIBERO) yields lower probe losses than joint position control (RoboTwin2.0), consistent with differential actions being more recoverable from consecutive-frame motion.

Dimensionality and normalization

A latent dimensionality of $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$6 provides the best overall trade-off on both the 7-DoF single-arm and 14-DoF dual-arm platforms. Single-arm benchmarks saturate quickly ($z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$7 reaches 0.926 on LIBERO, essentially tied with 32 at 0.922), but bimanual control demands more capacity: on RoboTwin2.0, $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$8 reaches 0.856 versus 0.802 at $z_t \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$9. Excessively large spaces yield no consistent gain.

On normalization, the evidence is decisive: across 33 method–benchmark combinations, unnormalized latent actions outperform normalized ones in 28 cases, with an average gain of +0.0115 (+0.0160 on LIBERO-Plus). When pretraining regularization already imposes sufficient structure on the latent space, additional normalization suppresses magnitude information useful for control.

Scaling behavior and real-world validation

Scaling Stage-II fine-tuning data from Robotwin+Liberoplus (14.5%) to the full ~59M-frame mixture including Open X-Embodiment data consistently improves downstream performance across all benchmarks, benefiting both DAP and LAP. The largest gain appears on LIBERO-Plus (up to +9.0%), indicating that data scaling primarily strengthens generalization under distribution shift rather than in-domain fit—a finding consistent throughout with the observation that latent-action pretraining helps most where generalization demands are highest.

Real-world experiments on a Franka Panda with four tabletop tasks (100 rollouts per task per method) show that a Qwen3-VL-4B backbone fine-tuned with latent actions (OpenVLA-OFT "LA-Tuned") succeeds on 317/400 rollouts versus 259/400 for the baseline—an improvement from 64.75% to 79.25% (+14.5 points, +22.4% relative). The LA-Tuned model also converges faster, exceeding the baseline's final performance (76.25% at 40k steps) after only 10k steps (85.0%), and exhibits visibly stronger task-conditioned grounding, remaining focused on task-relevant objects amid distractors where the baseline switches to incorrect task modes.

Limitations and open questions

Several constraints bound these conclusions. The study restricts Stage-I learning to video-only settings to avoid embodiment-specific confounds, so interactions between latent action learning and robot-state or physical-action priors—as explored by Villa-X, CLAM, and LatBot—remain outside its scope. The scaling-law claim rests on a single doubling of fine-tuning data composition rather than a multi-point scale sweep, so the functional form of any scaling law is not established. Proxy-metric validity is demonstrated only within fixed dimensionality and within the tested configuration family; whether FDM reconstruction metrics remain predictive for fundamentally different LAM architectures is unresolved. Finally, the authors note that all conclusions derive from arm-based manipulation, and generalization to dexterous hands, quadrupeds, and humanoids remains untested. Elevating latent actions from task-specific supervision to foundation-level representations integrated into multimodal pretraining, and validating findings on in-the-wild internet video, are left explicitly open.

Conclusion

This paper converts a fragmented design space into an empirically grounded set of guidelines for latent action learning in robotic manipulation. Its principal results—that plain LAPO remains a near-optimal paradigm, that optical-flow-based motion signals hurt rather than help, that regularization strength outweighs regularization choice, that $\Delta_t = \Delta(o_{t+1}, o_t)$0 with unnormalized latents suffices across embodiments, that JAP-LAP-style integration maximizes benefit, and that FDM reconstruction metrics are the most reliable (though ranking-insufficient) proxies—are supported by controlled comparisons over 41 configurations, three benchmarks, and real-robot deployment with a +14.5-point success-rate gain. For practitioners building embodied foundation models from unlabeled video, the study provides a concrete, reproducible default recipe and clarifies precisely which design decisions merit attention and which do not.