Task-Agnostic Latent Action Space

• Task-agnostic latent action space is a compact, compositional, and minimal representation of an agent’s dynamics learned solely from visual observations.
• The approach employs a sequential stochastic video prediction model with VAE-inspired techniques to enforce minimality and composability in the latent space.
• Empirical results demonstrate that unsupervised planning in this latent space rivals supervised models, dramatically reducing labeling needs while ensuring robust transfer across environments.

A task-agnostic latent action space refers to a learned, compact, composable, and minimal latent variable representation that captures the structure of an agent’s action space purely from observation—distinct from representations engineered for a particular task, label, control interface, or robot embodiment. In this context, “task-agnostic” means that the latent space encodes agent dynamics and action effects independently of any specific downstream objective or required supervision, making it suitable for generalization, transfer, imitation, and efficient planning.

1. Core Methodology: Learning Latent Action Spaces from Visual Observation

The methodology addresses the problem of inferring an agent’s entire action space solely by passively observing its visual behavior, with no access to action labels or ground-truth control signals. The central approach utilizes a sequential, stochastic video prediction model inspired by a temporal extension of the conditional Variational Autoencoder (VAE). The model is structured to satisfy two overarching criteria:

1. Minimality: The latent variable at each time step, $z_t$, should encode only the dynamic aspects of the scene (i.e., those corresponding to actions), while remaining insensitive to static features or scene content.
2. Composability: Latent representations of individual actions must be composable, such that the latent representing a sequence of actions can be constructed by functionally composing the latents of their constituent parts.

Formally, at each time $t$:

• The latent variable $z_t$ is sampled as $z_t \sim \mathcal{N}(0, I)$. Given previous frames and all latent samples up to $t$, the generative model predicts the next frame as:

$$x_t \sim p_\theta(x_t | \mathbf{x}_{1:t-1}, \mathbf{z}_{2:t}) = \mathcal{N}(\mu_\theta(\mathbf{x}_{1:t-1}, \mathbf{z}_{2:t}), I)$$

An information bottleneck objective is used to optimize $z_t$ to capture as little information as possible beyond that required for accurate next-frame prediction:

$$\max_{p_\theta, q_\phi} I((z_t, x_{t-1}), x_t) - \beta_z I(z_t, x_{t-1:t})$$

2. Composability and Disentanglement of Action Structure

A defining feature is the explicit enforcement of composability through a novel loss. The method introduces a compositional latent variable $\nu_t$ which recursively aggregates the impact of sequence fragments:

$\nu_t \sim q_\zeta(\nu_t | \nu_{t-1}, z_t)$

A secondary information bottleneck across composed sequences further ensures that the compositional representation $\nu_t$ must:

• Allow the network to generate plausible futures from composed latents, but
• Exclude static scene content or irrelevant nuisance factors.

The total loss combines the prediction IB and composability IB objectives:

$$\mathcal{L}^{total}_{\theta,\phi,\zeta} = \mathcal{L}^{comp}_{\theta,\phi,\zeta} + \mathcal{L}^{pred}_{\theta,\phi}$$

with

$$\mathcal{L}^{comp}_{\theta,\phi,\zeta} = \sum_{t=1}^{T_C} \left[ \mathbb{E}_{q_{\phi, \zeta}(\nu_{1:t} | x_{1:T})} \log p_{\theta}(x_{tC} | x_{1:(t-1)C}, \nu_{1:t}) - \beta_\nu D_{KL}(q_{\phi,\zeta}(\nu_t | x_{(t-1)C:tC}) || p(\nu)) \right]$$

This composability is crucial for disentanglement: if two actions are performed in sequence, the model must represent their joint outcome by composing their individual dynamics, not by accumulating or referencing irrelevant static features. The only feasible solution becomes encoding the agent’s true action structure in the latent variable.

3. Empirical Validation and Comparison to Supervised Baselines

The framework is validated in both synthetic and real-world environments (single-DoF simulated reacher arm and the BAIR robot pushing dataset).

Performance metrics:

Method	Reacher Abs. Error (deg)	BAIR Rel. Error (px)
Denton & Fergus Baseline	22.6 ± 17.7	3.6 ± 4.0
CLASP	2.9 ± 2.1	3.0 ± 2.1
Supervised	2.6 ± 1.8	2.0 ± 1.3

• In both simulation and real-robot data, the task-agnostic latent action space recovers the true action structure—independent of variations in appearance, background, or agents.
• Visual servoing experiments demonstrate that planning in the unsupervised latent space achieves performance that is indistinguishable from fully supervised models given only 100 labeled sequences (vs. 10,000+ for supervised baselines).
• The learned space is robust to nuisance variables (background, morphology, lighting), confirming the disentanglement property.

4. Practical Applications: Efficiency, Robustness, and Transfer

Task-agnostic latent action spaces enable:

• Data-efficient planning: Downstream tasks such as action-conditioned prediction or servoing require only minimal action labeling, reducing annotation effort by 100–1000x.
• Generalization and adaptation: Since representations are not tied to a specific scene, morphology, or task, skills transferred between agents or environments with minor retraining.
• Internet-scale learning: The methodology is applicable on large, unlabeled video data—extracting action spaces from observation alone—paving the way for scalable imitation and learning from demonstration where explicit action annotation is infeasible.

Notably, the method’s robustness to static content and scene-specific artifacts ensures that learned representations remain stable when transferred to new environments.

5. Technical and Implementation Details

Architectural components:

• CNN Encoder $\text{CNN}_e$: Encodes each frame to a latent embedding.
• LSTM/MLP Generative Model: Predicts next frame from history and latent variables.
• MLP Inference Network: Estimates mean and variance for $z_t$.
• Compositional MLP $\text{MLP}_{\text{comp}}$: Aggregates latent variables into composition latent $\nu_t$.

Training:

• Unsupervised phase: Trains full model on unlabeled frame sequences, minimizing $\mathcal{L}^{total}$.
• Minimal supervised mapping: For planners or downstream tasks, learns a small decoder/MLP to map between latent $z$ and action $u$ using a small labeled subset.

Planning: Utilizes Model Predictive Control (MPC) in the latent space with the Cross Entropy Method (CEM), performing rollouts in latent space and using the post-hoc action mapping to translate these to physical controls.

Key hyperparameters include: latent dimension (10), bottleneck weights $\beta_z = 10^{-2}$, $\beta_\nu = 10^{-8}$, frame resolution $64\times 64$, Adam optimizer ($2\times 10^{-4}$ learning rate).

6. Broader Implications and Theoretical Significance

This framework establishes several fundamental properties for task-agnostic latent action spaces:

• Expressivity: They fully encode agent dynamics, invariant to static and contextual features.
• Composability: They support structured, sequence-level reasoning, enabling hierarchical plan synthesis.
• Sample Efficiency: They unlock “learning to act by watching” with orders-of-magnitude less supervision.
• Transferability and Robustness: The approach is applicable across agents, morphologies, and real-world variations.

A plausible implication is that as the availability of unlabeled behavior data (e.g., Internet video) grows, task-agnostic latent action space learning will form the backbone of scalable, efficient, and general robotics and embodied AI methodologies.

---

Table: Summary of CLASP Properties

Property	CLASP Approach	Practical Impact
Action encoding	Minimal latent variables $z_t$, compositional $\nu_t$	Data-efficient, robust action modeling
Disentanglement	Information bottleneck, composability loss	Separates action from background/content
Generalization	No task/agent-specific supervision	Cross-agent/environment/scene transfer
Supervision need	Orders-of-magnitude fewer labels	Feasible for large-scale, weakly-labeled data
Planning	MPC with generative rollouts in latent space	Safe, efficient, interpretable control

7. Conclusion

Task-agnostic latent action spaces, as operationalized through stochastic, compositional video prediction, enable agents to autonomously discover and exploit the true structure of actions from visual data alone. The approach achieves strong data efficiency, generality, and robustness—supporting efficient planning and control with minimal supervision and forming a scalable path toward universal embodied learning.