Latent Actions from Factorized Transition Effects under Agent Ambiguity

Abstract: Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed transition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse-forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zeroshot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity.

• The paper presents a novel factorized model that extracts reusable motion primitives from ambiguous visual transitions for latent-action learning.
• It leverages unsupervised patchwise vector quantization and state-aware aggregation to construct robust and transferable action-like latents.
• Empirical evaluations demonstrate competitive policy performance and superior zero-shot transfer across diverse agents and environments.

Latent Actions from Factorized Transition Effects under Agent Ambiguity

Introduction and Motivation

The framework of world models in visual planning and control has traditionally relied on explicit agent and action knowledge, facilitating the separation of agent-induced variations from those caused by distractors or autonomous environmental dynamics. Observation-only Latent Action Models (LAMs), designed for learning in the absence of explicit action annotations, instead infer action-like latents solely from transitions in the observation space. In realistic environments—where multiple motion sources, including distractors and camera motion, are superimposed in the observed visual stream—these models face inherent ambiguity in attributing observed changes to controllable versus uncontrollable sources.

Figure 1: Motivation for agent ambiguity in observation-only latent action learning: LAMs must infer action-relevant latents solely from visual transitions, without access to explicit action labels.

Previous work has attempted to address this ambiguity through segmentation-based and object-centric strategies. However, these assumptions encounter limitations due to the misalignment between objects and scene dynamics: the visual effects of actions may span or transcend object boundaries, and strict reliance on objects can hinder generalization across diverse embodiments and environments.

The work introduces a bottom-up approach, proposing that transferable, robust action-like latents arise from factorizing observed visual transitions into reusable primitives, which can be later abstracted into actionable representations.

Problem Formulation and Theoretical Perspective

The underlying challenge is formalized as an inverse problem, contrasting the generative pathway (action → physical state transition → visual transition) with the aim of inferring a representation of controllable influence from visual transitions alone.

Figure 2: Forward generative chain—actions and exogenous sources propagate through the environment dynamics and rendering into observed visual transitions. Inverse inference recovers action-like signals from ambiguous outcomes.

Analytically, the mapping from action to visual transition is not injective—many distinct sources (agent, distractors, background, camera) may contribute similarly to $\Delta x_t$. Thus, LAMs trained purely on observations can at best uncover equivalence classes of causes as reflected by their observed effects, never recovering true underlying control signals without additional supervision.

The paper posits that observed effects constitute a more transfer-friendly intermediate than actions or underlying state changes, as the latter are tied to embodiment and control interface, whereas patterns of visual change (e.g., translation, deformation) may recur independent of agent morphology or environment.

Observed Transition Factorization (OTF)

Observed Transition Factorization (OTF) is introduced as an unsupervised compositional autoencoding scheme for structuring ambiguous visual transitions into a set of localized, reusable primitives.

Figure 3: OTF’s vocabulary learning. Motion-centered transition signals are patchified and quantized via VQ-VAE into a codebook of discrete observed-transition primitives; a decoder reconstructs the motion signal conditioned on the reference frame.

Key properties of OTF:

• Patchwise vector quantization: The observed motion is partitioned into non-overlapping patches, each encoded and matched to a latent codebook entry.
• Sparse activation: Only the codebook entries that best reflect localized motion patterns within each patch become active for a given transition.
• Spatial support: For each transition, the set of activated codes, their locations, and activation strengths jointly define the factorized representation $E_t$.

This approach encodes what and where motion-like effects are expressed, but does not yet resolve controllability or action relevance.

From OTF to Latent-Action Abstraction

To bridge from observed effects to actionable latents, the OTF-LAM model constructs a state-aware aggregation of the extracted primitives:

Figure 4: OTF-LAM forms action-like latents by aggregating state-aware factor tokens, scoring each for relevance, and composing them into a compact latent vector used for prediction.

• State-aware tokens: Each primitive is contextually re-embedded according to the current frame.
• Gating: A learned mechanism scores which primitives are likely relevant for representing the effect of the underlying (possibly unobserved) action.
• Aggregation: Relevance-weighted combination yields an action-like latent, $z_t^{\mathrm{act}}$, which is optimized for one-step forward prediction.

In parallel, OTF-LAM-Dino replaces the explicit image decoder with prediction in a fixed DINOv2 feature space, improving transferability by anchoring both transition and state representations in reusable spaces.

Figure 5: OTF-LAM-Dino leverages a frozen DINOv2 encoder and the OTF-vocabulary, learning to predict future states in a robust latent representation space.

Policy Learning Interface

For the learned latent space to support control, the authors connect $z_t^{\mathrm{act}}$ to real actions using limited action-labeled data:

Figure 6: Policy training pipeline—behavior cloning in the latent space followed by supervised mapping from $z_t^{\mathrm{act}}$ (and observation) to ground-truth actions.

A compact downstream interface is thereby established, suitable for policy optimization even with sparse supervision.

Empirical Evaluation

Transferability of Observed-Transition Primitives

The OTF codebook exhibits strong reusability in cross-morphology and cross-carrier transfer settings, evidenced by zero-shot evaluation:

Figure 7: Example of learned OTF codes assigned to the motion input, visualizing spatially localized code activations.

Figure 8: Cross-morphology transfer—OTF codebook trained on walker-run, evaluated zero-shot on cheetah-run in the DCS suite.

Figure 9: Controlled carrier transfer—zero-shot application of OTF codebook from one digit class set to another in Moving MNIST.

Quantitatively, transfer degradation (increase in MSE) under significant embodiment or carrier shift is markedly lower for OTF compared to monolithic VQ-based baselines, supporting claims of codebook generality.

Downstream Control Performance

On policy learning benchmarks (e.g., Distracting Control Suite), the following key results are observed:

Figure 10: (a) Downstream policy returns for OTF-LAM and baselines on cheetah-run. (b) Influence of OTF vocabulary size on performance.

• OTF-LAM and OTF-LAM-Dino show competitive or superior performance relative to standard and factorized LAMs, particularly as transition ambiguity increases.
• OTF-LAM-Dino consistently outperforms pixel-space OTF-LAM, suggesting frozen representation spaces ease the burden on the prediction head by filtering nuisance variation.

Performance dependence on codebook size is observed to be non-monotonic and task-dependent, indicating a trade-off between expressivity and over-localization of transition effects.

Qualitative Visualizations

Additional visualizations highlight:

• The effect of different motion-centered input transforms on code activation patterns (Figure 11).
• t-SNE visualization of the codebook structure in Moving MNIST, illustrating cluster diversity and primitive reuse (Figure 12).
• Code assignments in individual transitions for both DCS and Moving MNIST scenarios (Figures 13, 16).
• Zero-shot transfer outcomes under novel carrier/environment modifications (Figure 13).
• Enlarged and nuanced views of code activations under challenging transfer (Figure 14).

Implications and Future Directions

This work demonstrates that factorized representations of observed visual transitions form a robust substrate for learning transferable, action-like latents in ambiguous, distractor-rich settings. By decoupling local transition effects from both explicit action and object-centric priors, the approach enables broader generalization across agents and environments.

The practical implication is a two-stage interface for action learning:

1. Unsupervised construction of a generalist motion primitive vocabulary;
2. Lightweight, partially supervised mapping to control-specific latents and actions.

Theoretically, this supports a move away from monolithic or object-bound abstraction layers toward compositional, locally quantized intermediates, which connect more directly to causally observable consequences of control.

Integration of pretrained visual representations (e.g., DINOv2 in OTF-LAM-Dino) can further regularize the latent space, suppressing nuisance variation and enhancing downstream control—an avenue aligned with the broader trend in offline model-based RL and representation learning.

Outstanding research questions include:

• Extending transfer to environments with complex occlusion, multi-agent interaction, or persistent non-local dependencies;
• Systematic investigation of codebook topology, patch granularity, and their control-theoretic implications;
• Unified, scalable policy interfaces that integrate observation-only LAMs with action-rich, embodied agents in open-ended domains.

Conclusion

Observed Transition Factorization (OTF) and its downstream abstraction, OTF-LAM, provide a principled, modular approach to latent action learning in contexts of agent ambiguity. Empirical results establish the robustness and transferability of structured motion-effect vocabularies across agents, carriers, and environments. The extension to frozen representation spaces via OTF-LAM-Dino further validates the benefits of decoupled, reusable latent-action learning. This work underscores the promise of compositional, observation-centered representations for robust policy learning in the absence—or scarcity—of explicit action supervision.

(2606.30544)