Learning from Observation (LfO): Overview

• LfO is a subfield of imitation learning where agents learn from expert state trajectories rather than action labels.
• It employs methods such as distribution matching, inverse dynamics, and reward engineering to infer policies in complex environments.
• Empirical results in robotics and control highlight LfO's potential when expert action data is unavailable or costly to obtain.

Learning from Observation (LfO) is a subfield of imitation learning and reinforcement learning in which an agent learns to imitate expert behaviors using only observed state sequences from expert demonstrations, without access to the expert’s action labels. LfO is motivated by practical scenarios where expert actions are unavailable or costly to obtain, such as learning from video data or deploying robots in environments where instrumentation is infeasible. Over the past several years, LfO has evolved into a rigorous and multi-faceted discipline, distinct from Learning from Demonstration (LfD), and has yielded numerous specialized algorithms, frameworks, and applications across robotics, control, and artificial intelligence.

1. Fundamental Principles and Mathematical Formulation

LfO operates within the Markov Decision Process (MDP) formalism, but—unlike LfD—it discards direct access to expert actions. Let ℳ = (𝒮, 𝒜, 𝒯, r, γ, ρ₀) denote the MDP with state space 𝒮, action space 𝒜, transition kernel 𝒯, reward r, discount γ, and initial distribution ρ₀. Expert data is provided as sequences {s₀, s₁, ..., s_T}, forming state or state-transition trajectories, rather than {(s₀, a₀), (s₁, a₁), ...} pairs.

LfO methods seek to recover a policy π: 𝒮 → Δ(𝒜) whose induced trajectory distribution matches that of the expert. Since actions are not observed, these algorithms rely on matching state occupancy measures or state–transition distributions, occasionally by introducing surrogate models or by inferring missing action information through auxiliary mechanisms.

Formally, for a stationary distribution μ^π(s) or state–transition distribution μ^π(s, s′) under policy π, the occupancy matching principle underpins many LfO algorithms:

$\min_\pi D(\mu^\pi(s, s') \parallel \mu^E(s, s'))$

where D is a divergence (KL, Wasserstein, or f-divergence), and μ^E(s, s′) is the expert distribution over consecutive states.

2. Canonical LfO Algorithmic Taxonomy

Recent surveys (Burnwal et al., 20 Sep 2025) have established a taxonomy of LfO algorithms along two axes: (A) demonstration dataset construction, and (B) algorithmic design choice.

• Dataset Construction Axis:
  • Expert Type: Same-agent, dynamically different, or morphologically different expert.
  • Viewpoint: First-person or third-person (exteroceptive: e.g. monocular video).
  • Composition: Strictly expert, ranked/expert-multiquality, multi-expert, or expert+non-expert with extra proxy data.
• Algorithmic Design Choice:

1. Supervised Approaches: Use inverse dynamics models to infer missing action labels from state transitions, enabling behavior cloning. Vulnerable to compounding errors (“test-time shift”).
2. Goal Extraction Approaches: Decompose policy into high-level goal selection from the expert trajectory and a low-level controller for goal-reaching.
3. Reward Engineering: Define imitation rewards via learned similarity metrics on states or transitions (e.g., via representation learning), sometimes leveraging optimal transport or deep metric learning.
4. Distribution Matching: Directly align occupancy measures via adversarial learning (GAIfO-style), stationary distribution correction (DICE variants), or optimal transport. Often employs discriminators or density ratio estimation to derive surrogate rewards.

The following table summarizes this classification:

Approach	Core Method	Typical Assumptions
Supervised	Inverse dynamics	Similar agent dynamics
Goal Extraction	Subgoal matching	High-level segmentation
Reward Engineering	Metric learning	Flexible metrics, latent spaces
Distribution Matching	Adversarial/DICE	Occupancy divergence estimation

Each class addresses the challenge of missing expert actions via differing inductive biases. Distribution matching approaches, especially those based on adversarial imitation (e.g., GAIfO/IDDM/DIFO) or distribution correction estimation (DICE, LobsDICE, PW-DICE), have gained substantial prominence in high-dimensional and offline settings.

3. Key Theoretical Insights and Divergence with LfD

Theoretical analysis has focused on quantifying the gap between LfO and LfD. A central insight is the role of inverse dynamics disagreement (Yang et al., 2019, Cheng et al., 2020). In LfD, the agent directly matches expert state–action occupancy, while in LfO, only state or state-transition distributions can be matched. The divergence between the agent’s and expert’s inverse dynamics—formally, the KL divergence $D_{KL}(\rho_\pi(a|s,s') || \rho_E(a|s,s'))$—quantifies this gap.

A crucial result is that, under deterministic dynamics, each state transition (s, s′) corresponds to a unique action, causing the inverse dynamics disagreement to vanish. In general, the divergence is upper-bounded by negative causal entropy, which can be minimized in a model-free fashion (Yang et al., 2019):

$$D_{KL}(\rho_\pi(a|s,s') || \rho_E(a|s,s')) \leq -\mathcal{H}_\pi(s,a) + \text{const}.$$

Thus, maximizing causal entropy narrows the LfO–LfD gap, and in deterministic or nearly deterministic domains, LfO and LfD yield almost equivalent performance (Cheng et al., 2020).

4. Algorithmic Advances and Modern Frameworks

Distribution Matching with Adversarial and DICE-style Methods

• GAIfO and Variants: Generalized adversarial imitation from observation (GAIfO) matches the distribution over agent and expert state–transitions using a discriminator. Subsequently, IDDM augments GAIfO with entropy and mutual information terms to reduce inverse dynamics disagreement (Yang et al., 2019).
• DICE Algorithms: LobsDICE and PW-DICE formulate offline LfO as occupancy measure matching with stationary distribution correction, optimizing convex objectives with correction ratios and allowing more general (Wasserstein, f-divergence) metrics (Kim et al., 2022, Yan et al., 2023). PW-DICE further leverages contrastively learned distance functions.
• Diffusion-based LfO: Diffusion Imitation from Observation (DIFO) replaces the standard MLP discriminator with a conditional diffusion model, using denoising error as a “realness” reward and reformulating the adversarial objective for smoother, more robust credit assignment (Huang et al., 7 Oct 2024).

Off-Policy and Offline Learning

• OPOLO: Enables off-policy optimization by upper-bounding state–transition KL divergence and leveraging saddle-point duality, with regularization from a learned inverse action model to improve mode covering (Zhu et al., 2021).
• LobsDICE/PW-DICE: Operate in fully offline settings, leveraging imperfect but action-labeled datasets for correction while matching expert (state-only) trajectories, offering improved stability and scalability in stochastic environments (Kim et al., 2022, Yan et al., 2023).

Distributional and Stabilization-Oriented LfO

• MODULE: Unites distributional RL (distributional soft actor-critic) with LfO, bringing stability and improved return variance estimation to adversarial occupancy matching (Zhou et al., 22 Jan 2025).
• LSO-LLPM: Leverages control-theoretic Lyapunov proxy models to guide the learning of stabilizing policies directly from observation trajectories, bypassing reward engineering (Ganai et al., 2023).

Multi-Modal and Robot-Centric LfO

• Semantic Constraints and Task Models: Recent work develops frameworks for encoding human “common sense” (semantic constraints) into LfO task models, introducing representations like Labanotation for postures, contact-webs for grasping, and explicit modeling of physical and semantic motion constraints (Ikeuchi et al., 2021, Ikeuchi et al., 2023).
• Interactive and Multimodal LfO: The ITES system combines vision, speech, and human-in-the-loop correction to robustly extract executable task models from human demonstrations, supporting household robot applications (Wake et al., 2022).
• Skill-agent Libraries: Pre-designed, hardware-agnostic “skill agents” form modular libraries for transforming task models into robot-specific commands, enabling high reusability and easy transfer across platforms (Takamatsu et al., 4 Mar 2024).

5. Applications, Empirical Results, and Benchmarking

LfO algorithms have been validated on a wide spectrum of benchmark domains:

• Classical/Continuous Control: Tasks include CartPole, Pendulum, Hopper, HalfCheetah, Walker2d, Ant, and Humanoid (OpenAI Gym and MuJoCo) (Yang et al., 2019, Zhu et al., 2021, Kim et al., 2022, Zhou et al., 22 Jan 2025). In many cases, algorithms such as IDDM, OPOLO, and LobsDICE achieve expert-level performance or consistently outperform baseline LfO methods.
• Robotic Manipulation and Grasping: Frameworks based on contact-webs, domain randomization, and reinforcement learning transfer learned grasping skills from demonstration to hardware, showing robustness to pose and shape uncertainty (Saito et al., 2022).
• Household Task Automation: LfO pipelines integrating multimodal perception, symbolic task models, and hardware-independent skill libraries reproduce complex manipulation sequences on diverse robotic platforms (Ikeuchi et al., 2023, Takamatsu et al., 4 Mar 2024).
• Collective and Group Behavior Analysis: Structural identification methods extract interaction kernels for swarming, flocking, and multi-agent systems using solely observational data, attaining optimal nonparametric convergence rates (Feng et al., 2023).
• Visual LfO and Video-Only Policy Recovery: State-to-Go Transformers and diffusion imitation methods enable policy learning from raw video, successfully matching or exceeding expert returns on Atari, Minecraft, and visual robotics domains without environment rewards or action labels (Zhou et al., 2023, Huang et al., 7 Oct 2024).

Empirical evaluations indicate that distribution matching (adversarial, DICE, and diffusion-based) algorithms typically exhibit the best scalability, robustness to stochasticity, and performance in modern high-dimensional or offline settings, especially when combined with auxiliary mechanisms such as entropy maximization, mutual information regularization, or contrastive metric learning.

6. Relationship to Other Areas and Open Challenges

LfO interfaces closely with offline RL, model-based RL, and hierarchical RL:

• Offline RL: DICE-based LfO methods resolve distributional mismatch by leveraging additional proxy data, and can function entirely without online exploration (Kim et al., 2022, Yan et al., 2023).
• Model-Based RL: Forward and inverse dynamics models have been used for action inference, policy regularization, or reward construction (Zhu et al., 2021).
• Hierarchical RL: Goal extraction and subgoal identification approaches naturally mesh with skill decomposition and options frameworks.

Key open challenges identified in (Burnwal et al., 20 Sep 2025) include:

• Scalably adapting LfO to third-person demonstrations, domain transfer, and morphologically dissimilar experts.
• Automating subgoal segmentation and hierarchical policy induction from raw observation logs.
• Integrating plan-based reasoning with LfO for long-horizon imitation.
• Guaranteeing safety and regulatory compliance in policy learning from observation.
• Developing evaluation metrics sensitive to imitation fidelity rather than merely cumulative task returns.

7. Future Research Directions

Research frontiers in LfO include:

• Foundation Models and Large-Scale Data: Integrating LfO with foundation models trained on massive observation-only datasets promises better generalization, especially in unstructured real-world scenarios.
• Unified Objective Formulations: Extending primal Wasserstein and f-divergence frameworks, as seen in PW-DICE, to encompass broader classes of divergence and metric learning for more adaptive policy and reward shaping (Yan et al., 2023).
• Robustness and Uncertainty Modeling: Emerging diffusion-based discriminators and distributional RL suggest new directions in stabilizing adversarial LfO training (DIFO, MODULE) (Huang et al., 7 Oct 2024, Zhou et al., 22 Jan 2025).
• Cross-Modal and Semantic Reasoning: Continued integration of language, vision, and symbolic reasoning for capturing common-sense constraints and enabling intuitive human–robot interaction (Wake et al., 2022, Ikeuchi et al., 2023).
• Open-World, Multi-Agent LfO: Inferring emergent, long-scale collective behavior and interaction laws in heterogeneous, partially observed, or non-stationary agent populations (Feng et al., 2023).

LfO is now a mature subfield with theoretically grounded, empirically validated methods offering practical pathways for agent learning in realistic, data-constrained environments. Ongoing development is expected to yield both more scalable algorithms and richer theoretical underpinnings, capable of spanning vision-centric, language-informed, and safety-critical deployment domains.