--- title: Latent Action Q-Learning url: https://www.emergentmind.com/topics/latent-action-q-learning-laq type: topic --- # Latent Action Q-Learning Latent Action Q-Learning (LAQ) is an offline reinforcement learning (RL) method designed to recover high-quality value functions from state-only experience, where action labels are never observed. The central premise is that by mining a discrete refinement of the hidden action space from undirected state transition data, one can apply Q-learning to these "latent actions" and still recover the optimal value function, thereby enabling subsequent downstream applications such as accelerated policy learning, controller guidance, and embodiment transfer across agents and domains [2204.12458]. ## 1. Problem Setting and Motivation LAQ addresses the problem of learning value functions in finite Markov decision processes (MDPs) $(S, A, p, \gamma)$ from offline datasets comprised solely of state-transition-reward triplets $(s_t, s_{t+1}, r_t)$, with the intervening actions $a_t$ unlabeled and unknown. This setting generalizes scenarios such as: - Learning from video demonstrations - Random or suboptimal exploration data - Cross-embodiment offline experiences No assumption is made about optimality or intent in the collected data, and the sole goal is to learn a state-value function $V(s)$ that correlates with the optimal $V^*(s)$, despite missing true action labels. ## 2. Theoretical Foundation: Action-Space Refinement and Value Preservation A pivotal theoretical result supporting LAQ is the preservation of optimal value functions under refinement of the action space. Given a true MDP $M = (S, A, p, \gamma)$ and a refinement $\hat M = (S, \hat A, \hat p, \gamma)$, where each $\hat a \in \hat A$ exactly matches the dynamics and reward of some $a \in A$ and each $a \in A$ is covered by at least one $\hat a$, the theorem states: \[ \forall s \in S, \quad V^*_{\hat M}(s) = V^*_M(s) \] The proof leverages "fundamental action classes"—sets of actions with identical transition-reward kernels—and demonstrates that for any policy on $M$ there exists a mirrored policy on $\hat M$ (and vice versa), and thus Q-learning converges to $V^*$ in both spaces. This establishes that if one can assign any refinement of the hidden actions to observed $(s, s', r)$ transitions, running Q-learning on these pseudo-labels recovers true optimal value functions [2204.12458]. ## 3. Mining Latent Actions via Conditional State Prediction LAQ constructs a discrete latent action set $\hat A$ by fitting a conditional forward model: \[ f_\theta: S \times \hat A \to S \] This model predicts $s_{t+1}$ from $s_t$ and a candidate latent action $\hat a$. The objective is an EM-style reconstruction loss: \[ L(\theta) = \sum_{(s, s') \in D} \min_{\hat a \in \hat A} \| f_\theta(s, \hat a) - s' \|^2 \] Training alternates between: - Assigning to each $(s, s')$ pair the latent action $\hat a_t = \arg\min_{\hat a} \| f_\theta(s_t, \hat a) - s_{t+1} \|^2$ - Updating $\theta$ by gradient descent given the current assignments Architectures vary: MLPs for low-dimensional states; convolutional encoder-decoders for images. Each latent action learns to best "explain" observed state transitions, in effect yielding a data-driven refinement of the (hidden) ground-truth action space. ## 4. LAQ Algorithmic Workflow With the latent forward model fit, the dataset is relabeled as $\hat D = \{ (s_t, \hat a_t, s_{t+1}, r_t) \}$. Standard offline Q-learning is then applied: \[ Q(s_t, \hat a_t) \leftarrow Q(s_t, \hat a_t) + \alpha \left[ r_t + \gamma \max_{a' \in \hat A} Q(s_{t+1}, a') - Q(s_t, \hat a_t) \right] \] Value estimation follows $V(s) = \max_{a \in \hat A} Q(s, a)$. Variants with DQN or BCQ are directly compatible for high-dimensional or continuous state spaces. High-level pseudocode: | Step | Detail | Purpose | |------------------------------------------|-------------------------------------------------------|--------------------------| | Fit forward model $f_\theta$ | EM-style min reconstruction loss | Uncover latent actions | | Label latent actions | $\hat a_t = \arg\min_{\hat a} \| f_\theta(s_t, \hat a) - s_{t+1} \|^2$ | Assign to transitions | | Q-learning update | On $(s, \hat a, s', r)$ quadruples | Learn value function | | Value extraction | $V(s) = \max_{\hat a} Q(s, \hat a)$ | Output for downstream RL | ## 5. Empirical Evaluation and Recovery of Value Functions LAQ's recovered values are benchmarked via Spearman's rank correlation $\rho(V, V_{gt})$ with reference value functions trained on ground-truth actions. Results across various domains: | Environment | LAQ $\rho$ | Ground Truth $\rho$ | |----------------------------|--------------|---------------------| | 2D Grid World | ≈ 0.985 | ≈ 1.000 | | Atari Freeway (images) | ≈ 0.961 | ≈ 0.970 | | 3D Visual Navigation | ≈ 0.927 | ≈ 0.991 | | Maze2D (continuous) | ≈ 0.844 | ≈ 0.851 | | FrankaKitchen Manipulation | ≈ 0.905 | ≈ 0.901 | The data indicate that LAQ's value functions match the rank orderings given by the true action labels very closely. In all cases, LAQ outperforms clustering-based or "one-action" baselines, and approaches oracle performance [2204.12458]. ## 6. Downstream Applications LAQ enables several practical outcomes using value functions learned solely from state-only data: - **Reward Shaping for Sample Efficiency:** Using $F(s,s') = V(s') - V(s)$ as a potential-based shaping term densifies sparse reward RL (so-called "densified RL"), resulting in 3–10× faster learning compared to using the sparse reward alone. - **Low-Level Controller Guidance:** Value functions derived from LAQ allow selection among a small set of primitive controllers $C$ by choosing the one-step outcome with highest $V(\cdot)$. This provides zero-shot navigation in high-dimensional domains (e.g., SPL = 0.82 versus 0.53 for naive strategies). - **Cross-Embodiment Transfer:** Learned $V(s)$ functions can be mapped to the state-space of a new embodiment (e.g., quadruped agent, robot arm), accelerating RL in the new agent by 2–5× relative to training from scratch. ## 7. Experimental Design and Comparative Metrics Experiments span a wide range of high- and low-dimensional domains: - **2D GridWorld** (tabular, sparse rewards) - **Atari Freeway** (84×84 image inputs, discrete actions) - **Maze2D** (continuous navigation) - **3D Visual Navigation** (Habitat building scans) - **FrankaKitchen** (9-DOF manipulation, embodiment transfer with "hook" variant) Baselines include: - Single-action labeling with TD(0) - K-means clustering on concatenated state and next-state pairs, or on state transitions - D3G (state-only Q-learning via generative modeling) - Behavior cloning, with or without RL fine-tuning - Prior inverse model labeling methods Key metrics: - Spearman's rank correlation $\rho$ to $V_{gt}$ - Mean squared error to tabular $V^*$ (toy MDPs) - Sample complexity (return vs. interactions) - SPL (Success weighted by Path Length) for navigation LAQ consistently achieves value ranking within 98% of ground-truth Q-learning in grid worlds, 96% in Freeway, and 93% in 3D navigation. Densified RL using LAQ accelerates convergence by 3–10× compared to sparse reward RL, often outperforming behavior cloning with RL fine-tuning. Cross-embodiment transfer using LAQ values accelerates RL by factors of 2–5 compared to baseline agents. --- LAQ thus demonstrates that value function learning is not fundamentally dependent on access to true action labels. By extracting a data-driven, latent refinement of the underlying action space, it becomes feasible to recover value functions sufficient for a variety of high-level RL applications, even in settings involving high-dimensional observations and cross-embodiment transfer [2204.12458]. Source: https://www.emergentmind.com/topics/latent-action-q-learning-laq