---
title: State-Only Imitation Learning (SOIL)
url: https://www.emergentmind.com/topics/state-only-imitation-learning-soil
type: topic
---

# State-Only Imitation Learning (SOIL)

State-Only Imitation Learning (SOIL), also termed Learning from Observation (LfO), is a paradigm in which an agent acquires task-solving policies by imitating expert-generated state trajectories, while explicitly lacking access to the underlying expert actions. This approach relaxes the conventional imitation learning constraint of acquiring both state and action information, thus enabling the utilization of demonstration data that exists solely as state trajectories (e.g., from videos or raw logs). This shift broadens the applicability of imitation learning to domains where expert action annotations are impractical or unavailable.

## 1. Foundational Frameworks and Problem Formalization

SOIL is addressed within the overall context of reinforcement learning (RL) and imitation learning (IL), both typically defined over a Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, r, \gamma, \rho_0)$, where $\mathcal{S}$ is the set of states, $\mathcal{A}$ the set of actions, $\mathcal{T}$ the transition dynamics, $r$ the reward function, $\gamma$ the discount factor, and $\rho_0$ the initial state distribution [2509.19379]. A stationary policy $\pi$ induces an occupancy measure $\mu^\pi(s, a)$ that satisfies the Bellman flow constraint:
$$
\sum_a \mu(s, a) = (1-\gamma)\rho_0(s) + \gamma \sum_{s', a'} \mathcal{T}(s | s', a') \mu(s', a'), \quad\mu(s, a) \geq 0\ \forall s, a.
$$
Traditional IL uses state-action pairs for behavioral cloning, inverse reinforcement learning (IRL), or adversarial IL such as GAIL. In SOIL, policies must be learned—using only state (or, in some cases, state transition) visitation information—by inferring or reconstructing action-level feedback or by aligning state(-transition) distributions.

SOIL methods generally fall into the following categories [1905.13566, 2509.19379]:
- **Model-based approaches**: Use learned system dynamics (inverse or forward models) to infer hidden action information from state transitions.
- **Model-free approaches**: Learn directly from state(-transition) distributions via adversarial or reward-engineering objectives without explicit action inference.
- **Hierarchical and Goal-directed decomposition**: Decompose imitation into sub-goal prediction and low-level control, often leveraging hierarchical RL techniques.

## 2. Core Methodologies and Algorithmic Taxonomy

The taxonomy of SOIL algorithms can be systematically organized as follows, reflecting both the composition of the expert dataset and the learning strategy [2509.19379]:

| Methodological Class         | Key Mechanism                                                        | Example Techniques                                                           |
|-----------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------|
| Inverse/Forward Dynamics    | Estimate hidden actions or latent controls from state-only sequences   | Inverse dynamics models, latent variable policies [1905.13566, 2004.04650]  |
| Reward Engineering          | Compute surrogate rewards from state similarity in embedding space     | Embedding-based rewards, temporal/contrastive losses [1905.13566, 2509.19379]|
| Adversarial Distribution-Matching | Minimax divergence minimization using discriminators over states or transitions | State-only GAIL, WAIfO, occupancy-based matching [1905.13566, 2202.04332, 2312.06348, 2410.05429] |
| Goal-from-Observation (Hierarchical)   | Meta-policy selects achievable demonstration goals, low-level policy solves reaching | Hierarchical frameworks (e.g., SILO) [1912.07670]                        |

**Model-based Approaches:**  
Inverse dynamics models seek a function $f_{\rm inv}: (s_t, s_{t+1}) \mapsto a_t$, trained from environment samples $(s_t, a_t, s_{t+1})$ to minimize prediction error on actions, and then used to generate surrogate actions for expert sequences [1905.13566, 2004.04650]. Forward dynamics models $f_{\rm fwd}: (s_t, z) \mapsto s_{t+1}$ with latent (“action-like”) variables can first train a latent policy $\pi(z|s_t)$ such that predicted $s_{t+1}$ matches observed expert transitions, with a mapping later established between the latent actions $z$ and real actions $a_t$.

**Model-Free Approaches:**  
In adversarial variants, the minimax objective becomes
$$
\min_{\pi} \max_D \mathbb{E}_{(s_t, s_{t+1})\sim\tau_e} [\log D(s_t, s_{t+1})] + \mathbb{E}_{(s_t, s_{t+1})\sim\pi} [\log(1 - D(s_t, s_{t+1}))]
$$
with the goal of aligning the state-transition occupancy measure between imitator and expert [1905.13566, 2312.06348, 2410.05429].

Reward engineering can involve defining
$$
r(s_t) = - \| \phi(s_t) - \phi(s^{\text{expert}}_t)\|_2
$$
where $\phi$ is a learned or fixed state embedding, potentially leveraging contrastive or temporal constraints for robustness [1905.13566].

**Hierarchical and Goal-based Decomposition:**  
Meta policies select temporally feasible or reachable demonstration states (“sub-goals”) given the current agent’s capabilities, while low-level controllers attempt to reach those sub-goals [1912.07670]. This enables effective handling of differences in dynamics, embodiment, or workspace relative to the demonstrator.

## 3. Theoretical Properties and Divergence-minimization Objectives

A central theme in SOIL is the minimization of divergence between the expert’s and the imitator’s visitation measures:
$$
\min_\pi D(\mu^\pi(\cdot) \Vert \mu^E(\cdot))
$$
where $\mu^\pi$ and $\mu^E$ are, e.g., state or state–transition distributions under the imitator and expert, respectively [2202.04332, 2305.00303, 2509.19379].  
- **Adversarial methods** implement this by training a state(-transition)-level discriminator via GAN or diffusion-driven losses [2312.06348, 2410.05429].
- **Non-adversarial (density-matching) methods** optimize, for example, the $\mathrm{KL}$ divergence between μ^π and μ^E, rewriting the objective into maximum entropy RL or SAC-style updates with rewards computed from conditional state transition densities [2202.04332].

Several works propose further refinements: robust approaches handle transition dynamics mismatch by optimizing over families of plausible transition models or integrating worst-case action perturbations [2002.11879, 2202.06003]; hierarchical and decoupled schemes construct planners operating at the state or sub-goal level, abstracting policy optimization away from direct state-to-action mappings [2203.02214].

## 4. Practical Applications, Empirical Results, and Performance Characteristics

SOIL has demonstrated efficacy in a range of domains, with notable performance in:
- **Robotics**: real-world robot arms performing reaching or manipulation tasks using only state-based demonstrations or third-person videos, benefitting from sample-efficient adversarial methods and hierarchical goal decomposition [1906.07374, 1912.07670].
- **Dexterous Manipulation**: providing state-only demonstration enables learning in high-dimensional control setups where collecting action data is prohibitive; joint inverse model/policy training yields competitive performance with state-action baselines [2004.04650].
- **Continuous Control**: methods such as conditional density estimation, diffusion-based adversarial discriminators, or coupled normalizing flows achieve superior or state-of-the-art performance across benchmarks, even with as little as a single demonstration trajectory [2305.00303, 2312.06348, 2410.05429, 2408.09125, 2308.12573].
- **Zero-shot Transfer and Domain Adaptation**: robust SOIL methods generalize favorably across transition dynamics and morphology mismatches, supporting sim-to-real transfer and heterogeneous embodiment scenarios [1911.10947, 2002.11879, 2202.06003, 2203.02214].
- **Interactive and Human-guided Learning**: frameworks such as TIPS allow non-expert users to teach policies via state-space corrections, with lower cognitive burden and empirically improved learning curves relative to classic IL techniques [2008.00524].

Empirical results across these settings consistently show that, while SOIL policies may converge more slowly or suffer modest performance loss outside the demonstrated distribution, appropriately designed algorithms (e.g., those incorporating optimistic exploration, robust loss formulations, or diffusion-model discriminators) can achieve performance competitive with conventional action-aware methods.

## 5. Connections to Related Learning Paradigms

SOIL methodologies overlap with and draw from several adjacent fields:
- **Offline RL**: Algorithms such as DICE-based methods exploit Bellman flow constraints and leverage offline imitator data to reduce distribution shift and enable safe deployment [2509.19379].
- **Model-based RL**: The use of learned forward and inverse models for prediction, planning, and reward estimation in SOIL closely relates to model-based exploration and planning in RL.
- **Hierarchical RL**: Meta/controller decomposition in goal-driven SOIL methods parallels the structure seen in many hierarchical RL architectures, enhancing temporal abstraction and generalization across environments [1912.07670, 2203.02214].
- **Computer Vision and Perception**: Advances in pose estimation, keypoint detection, and domain adaptation directly support robust SOIL, especially for visual demonstration data [1905.13566].
- **Diffusion and Flow-based Generative Modeling**: Modern discriminators in SOIL now employ diffusion models or coupled normalizing flows for improved stability and accurate density estimation, improving reward signal fidelity and policy training robustness [2312.06348, 2410.05429, 2305.00303].

## 6. Open Challenges and Future Research Directions

Ongoing challenges and open avenues for research in SOIL, as articulated in recent surveys and position papers [2509.19379], include:
- **Robust Perceptual Front-ends**: Addressing challenges in extracting precise state information from observational data, particularly in the context of noise, occlusion, and viewpoint/embodiment mismatch. Potential avenues involve advanced CV techniques and representation learning (e.g., using CycleGAN for domain adaptation).
- **Sample Efficiency and Data Utilization**: Although significant progress has been made (e.g., LQR+GAIfO, diffusion-based reward estimation), further improvements in sample efficiency—especially in real-world or safety-critical applications—require continued development of control-theoretic and model-based approaches.
- **Integration with Reinforcement Learning**: Combining SOIL techniques with RL (e.g., via bootstrapping, mixing imitation-based reward signals with sparse true rewards) may enhance learning robustness and policy generalization.
- **Third-person and Unaligned Demonstrations**: Leveraging demonstrations from different viewpoints, domains, or embodiments (e.g., human-to-robot transfer, YouTube videos) requires sophisticated domain alignment and viewpoint-invariant representation learning, as well as hierarchical or sub-goal decomposition.
- **Automatic Discovery of Sub-goals and Hierarchies**: Enabling agents to automatically identify and exploit subgoal structures within unannotated demonstration data remains an open problem for scaling SOIL to complex, long-horizon tasks.
- **Safe Imitation**: Developing evaluation metrics beyond accumulated return, emphasizing safety and alignment with demonstrator intent, particularly important for SOIL in critical domains.

## 7. Summary Table: Classes of SOIL Algorithms

| Algorithm Class           | Summary Description                                                                    | Cited Works                      |
|---------------------------|----------------------------------------------------------------------------------------|----------------------------------|
| Inverse/Forward Dynamics  | Infer actions or latent controls from state transitions; joint policy/model learning   | [1905.13566, 2004.04650]         |
| Adversarial Distribution  | Minimax discrimination over state(-transition) distributions between agent and expert  | [1905.13566, 2312.06348, 2410.05429]     |
| Reward Engineering        | Construct reward signals from state (or embedding) similarity metrics                  | [1905.13566, 2509.19379]         |
| Goal-from-Observation     | Hierarchical selection of achievable expert goals with low-level control               | [1912.07670]                     |
| Conditional Density Estimation | Markov or balance-equation satisfaction loss with (kernel or flow-based) estimators     | [2308.12573, 2408.09125]         |
| Diffusion/Flow-based      | Diffusion/model-based discriminators, robust density matching                          | [2312.06348, 2410.05429, 2305.00303]     |

## References to Specific Papers

- [1905.13566]: Literature review and methodological summary of SOIL, introducing the distinction between model-based and model-free approaches and outlining open problems (perception, sample efficiency, viewpoint mismatch).
- [1912.07670]: SILO—Meta-policy framework for selective imitation of reachable states, supporting hierarchical skill transfer and robust adaptation.
- [1911.10947]: State alignment methods that leverage both local (β-VAE next-state prediction + inverse dynamics) and global (Wasserstein distribution distances) perspectives for improved stability under dynamics mismatch.
- [2004.04650]: Joint training of inverse dynamics and policy for dexterous manipulation from pure state observations in high-dimensional action spaces.
- [2202.04332]: KL divergence minimization for non-adversarial SOIL, achieving robust policy learning with interpretable convergence metrics.
- [2203.02214]: Decoupled policy optimization, learning transferable high-level planners separately from low-level skill modules.
- [2305.00303]: Coupled flow-based density estimation for flexible state-only or state-transition occupancy matching.
- [2312.06348, 2410.05429]: Diffusion-based discriminators in SOIL, providing enhanced distribution modeling capacity and more reliable reward signals.
- [2408.09125, 2308.12573]: Markov balance and conditional density estimation frameworks for fully batch, offline SOIL settings, with theoretical consistency guarantees and competitive empirical results.
- [2509.19379]: Comprehensive survey, taxonomy, and roadmap for SOIL research, emphasizing connections to hierarchical, model-based, and offline RL, and outstanding challenges: third-person LfO, subgoal discovery, safety, and evaluation metrics.

---

In summary, SOIL encompasses a broad class of methods—spanning model-based inference, adversarial distribution alignment, reward engineering, and hierarchical decomposition—that enable effective imitation from state-only trajectories. The field continues to advance with innovations in generative modeling, robust transfer, and integration with vision and planning, while addressing practical constraints in demonstration collection, environmental mismatch, and evaluation.

Source: https://www.emergentmind.com/topics/state-only-imitation-learning-soil