Imitation Learning from Observation

• IfO is a paradigm of imitation learning that reproduces expert behavior using only state or observation sequences, bypassing the need for explicit action labels.
• It employs techniques such as inverse dynamics, adversarial distribution matching, and reward shaping to infer the control signals underlying observed transitions.
• IfO applies to various domains including robotics and autonomous driving, demonstrating effective policy transfer from low-dimensional state data to raw visual observations.

Imitation Learning from Observation (IfO), also termed ILfO, ILFO, or learning from observations (LfO) in parts of the literature, is the variant of imitation learning in which a learner must reproduce expert behavior from state or observation sequences alone, without access to demonstrator actions. Demonstrations may consist of low-dimensional state trajectories or raw visual observations such as video frames, and the central difficulty is that the learner must recover useful control information from what happened rather than from what the expert did. This makes direct behavioral cloning inapplicable and shifts the problem toward transition-level reasoning, representation learning, reward or value construction, dynamics estimation, and reinforcement-learning-based policy optimization (Torabi et al., 2019, Torabi et al., 2018).

1. Formal problem setting

A common formalization places IfO in an MDP $M=\{S,A,P,r,\gamma\}$ or, in episodic analyses, in a finite-horizon MDP $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$. What distinguishes the setting is the demonstration interface: instead of expert trajectories of state-action pairs, the learner receives only state or observation sequences such as $\tau_e=\{o_t\}$ or $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$. The learner interacts with the environment using its own actions, but the demonstrator’s control signals are absent (Torabi et al., 2021, Kidambi et al., 2021).

This difference is not merely notational. Standard imitation learning can supervise a policy directly on $(s_t,a_t)$, whereas IfO must infer behavior from state evolution alone. Several papers explicitly contrast this with behavior cloning and with offline RL: behavior cloning requires action-labeled expert demonstrations, and offline RL typically assumes reward signals are available or can be defined, while IfO must operate with action-free expert data and often no external reward (Bloesch et al., 9 Jul 2025).

A recurrent technical point is that state-only matching can be ambiguous. Transition-based formulations argue that the correct object of imitation is often the behavioral effect of actions on the environment, not the demonstrator’s motor commands themselves. This is the motivation for state-transition occupancies such as

$$p_\pi(s_i,s_j)=\sum_a P(s_j\mid s_i,a)\,\pi(a\mid s_i)\,p_\pi(s_i),$$

which appear in transition-matching approaches and make explicit that IfO is fundamentally about matching induced dynamics in observation space (Torabi et al., 2018).

2. Methodological taxonomy

The literature review on IfO organizes methods into perception and control, and, on the control side, into model-based and model-free approaches. Subsequent work has refined that decomposition into a more granular landscape including inverse-dynamics recovery, adversarial transition matching, non-adversarial density matching, likelihood-based reward learning, optimal-transport rewards, and direct value learning from demonstrations (Torabi et al., 2019).

Family	Core object	Representative papers
Inverse/forward dynamics and action recovery	Infer actions from $(s_t,s_{t+1})$ or fit dynamics/latent-action models	(Monteiro et al., 2020, Pavse et al., 2019, Wang et al., 2024)
Adversarial distribution matching	Match expert and learner state-transition distributions with a discriminator	(Torabi et al., 2018, Torabi et al., 2019, Torabi et al., 2021)
Offline occupancy or KL matching	Match expert state or transition densities without a min-max game	(Ma et al., 2022, Boborzi et al., 2022)
Reward, value, or transport-based imitation	Construct rewards, distances, or values directly from observations	(Edwards et al., 2019, Jaegle et al., 2021, Chang et al., 2023, Sonwa et al., 2023)
Context-, viewpoint-, and progress-aware visual alignment	Translate demonstrations, use keypoint similarity, or schedule temporal credit	(Liu et al., 2017, Karnan et al., 2021, Liu et al., 2023)

This taxonomy also clarifies a frequent misconception: IfO is not a single algorithmic trick for “behavioral cloning without actions.” In practice, different methods target different missing structures. Some reconstruct latent actions, some match distributions over transitions, some learn dense rewards, and some avoid reward learning altogether by regressing state values or by using optimal transport directly.

3. Action inference and model-based control

One major IfO lineage reduces the problem to pseudo-labeled imitation by learning an inverse dynamics model from the learner’s own interaction data and then applying it to expert state transitions. Augmented Behavioral Cloning from Observation (ABCO) is an explicit improvement of the BCO line: it trains an inverse dynamics model $p_\theta(a_t\mid s_t,s_{t+1})$, uses that model to infer expert actions, and trains a policy by behavioral cloning on the inferred labels. Its two distinctive augmentations are self-attention in both the inverse dynamics model and policy model, and a sampling strategy that mixes successful post-demonstrations with random pre-demonstrations to prevent collapse into sub-optimal local minima and loss of minority actions (Monteiro et al., 2020).

RIDM takes a different model-based route. It assumes only one state-only demonstration trajectory and treats the demonstration as a sequence of set points. Its task-specific inverse dynamics model $\mathcal{M}_\theta:(s_t,s_{t+1})\mapsto a_t$ is queried as

$\widetilde{a_t}=\mathcal{M}_\theta(s_t,s^e_{t+1}),$

after which the learner executes the resulting action and updates $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$0 to maximize environment return rather than imitation accuracy alone. This design explicitly permits deviation from the demonstration when that increases task reward, and the reported evaluations include MuJoCo domains, SimSpark RoboCup 3D tasks, and a physical UR5 robot arm (Pavse et al., 2019).

A related but more perception-heavy model-based formulation appears in context translation. “Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation” learns a context translation model on top of video prediction, built from encoders $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$1, $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$2, a translation module $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$3, and a decoder $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$4. The translated expert rollout is then compared with the learner’s behavior to define dense imitation rewards for deep RL. The method is explicitly designed for viewpoint changes, environment configuration changes, object position variations, and real-world robotic chores such as sweeping, pushing, and ladling almonds (Liu et al., 2017).

A more recent model-based formulation assumes known system dynamics and uses them to estimate the missing control sequence. “Imitation Learning from Observations: An Autoregressive Mixture of Experts Approach” assumes discrete-time dynamics $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$5, estimates controls by

$$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$6

and then fits a stochastic switching autoregressive mixture-of-experts policy by regularized maximum likelihood. It further imposes a Lyapunov stability constraint, via an LMI reformulation, to ensure asymptotic stability of the identified model under recursive rollout. The reported demonstrations come from autonomous driving datasets collected from human drivers (Wang et al., 2024).

4. Adversarial, occupancy, and distribution matching

GAIfO is the canonical adversarial formulation of IfO. Its central move is to replace state-action costs with transition costs $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$7 and to adversarially match expert and imitator transition occupancies. The discriminator $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$8 is trained on state transitions, and the policy is optimized with reward $$\mathcal{M}=\{\mathcal{S},\mathcal{A},P^\star,H,c,s_0\}$$9. The paper’s conceptual claim is that IfO should match the demonstrator in terms of behavioral effect, not identical actions, and it explicitly motivates transition matching by examples where state distributions alone are ambiguous (Torabi et al., 2018).

A practical criticism of adversarial IfO is sample inefficiency. LQR+GAIfO addresses that by replacing model-free policy optimization with trajectory-centric reinforcement learning. The method learns local dynamics, trains a discriminator on state transitions using a Wasserstein loss with gradient penalty, composes the discriminator with the learned dynamics to obtain a surrogate cost $\tau_e=\{o_t\}$0, approximates that cost locally by a second-order Taylor expansion, and improves a time-varying linear-Gaussian controller with LQR or iLQR under a KL-divergence trust region. The empirical target is real-robot feasibility, and the reported experiments involve a Universal Robotics UR5 six-degree-of-freedom robot arm and Gazebo simulation (Torabi et al., 2019).

DEALIO follows a closely related strategy but integrates PILQR into adversarial IfO. It defines the discriminator as a quadratic form over state transitions,

$\tau_e=\{o_t\}$1

then substitutes a learned linear dynamics model $\tau_e=\{o_t\}$2 to rewrite the transition-space discriminator as a state-action cost compatible with PILQR. The reported benefit is substantially improved data efficiency over GAIfO across Disc, PegInsertion, GripperPusher, and DoorOpening (Torabi et al., 2021).

Non-adversarial distribution matching has become an important alternative. SOIL-TDM minimizes

$\tau_e=\{o_t\}$3

where the KL is over state-only trajectory distributions. By decomposing $\tau_e=\{o_t\}$4 using forward dynamics and a backward or inverse action density, it rewrites the objective as a maximum-entropy RL problem optimized with SAC under a modified reward computed from three learned density models: expert transition density, forward dynamics, and backward action density. A stated advantage is an interpretable convergence metric derived from the same KL objective (Boborzi et al., 2022).

SMODICE offers a related offline perspective. It targets state-occupancy matching,

$\tau_e=\{o_t\}$5

derives a regularized objective using the offline dataset occupancy $\tau_e=\{o_t\}$6, solves the dual via Fenchel conjugacy, estimates occupancy correction weights

$\tau_e=\{o_t\}$7

and then trains the policy by weighted behavior cloning. Because it avoids action matching altogether, the same framework is used for offline IfO, IfO with morphology or dynamics mismatch, and example-based reinforcement learning (Ma et al., 2022).

Model-based distribution matching with explicit exploration appears in MobILE. MobILE learns a forward model, then solves an optimistic min-max objective

$\tau_e=\{o_t\}$8

where the bonus $\tau_e=\{o_t\}$9 weakens penalties in uncertain regions and thereby trades off imitation against strategic exploration. The paper also proves an exponential sample-complexity separation between standard imitation learning with expert actions and ILFO, formalizing that action-free imitation is strictly harder (Kidambi et al., 2021).

5. Rewards, values, trajectory distances, and temporal progress

A distinct strand of work learns scalar guidance directly from observation trajectories rather than inferring actions. PVO is the clearest value-based formulation. It assumes goal-directed demonstrations and treats the final state of each trajectory as a surrogate terminal goal, assigning reward $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$0 to the last state and $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$1 elsewhere. The resulting self-supervised value target is

$D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$2

learned by regression with

$D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$3

The learned value is then used either to replace Q-learning bootstrapping via $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$4 under the sparse-reward assumption, or as a potential-based shaping reward $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$5 (Edwards et al., 2019).

FORM derives reward from a KL-divergence objective over observation-sequence distributions, but replaces adversarial discrimination with generative effect models. Its per-step reward is

$D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$6

where $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$7 models demonstrator transitions and $D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$8 models imitator transitions. The formulation is explicitly non-adversarial and is motivated as more robust to task-irrelevant distractor features because it compares conditional transitions rather than global occupancy alone. The reported benchmark spans 13 tasks from six domains in the DeepMind Control Suite (Jaegle et al., 2021).

BootIfOL extends reward learning to raw-visual continuous control by constructing a trajectory-level distance in latent space,

$D^e=\{\tau_i^e\}, \tau_i^e=\{(s^e)_i\}$9

Its novelty lies in bootstrapped contrastive learning: negatives are initially random-policy rollouts, then progressively replaced by current-policy rollouts as the agent improves, making the reward model increasingly discriminative around near-expert behavior (Sonwa et al., 2023).

Optimal transport provides another non-adversarial route. OOPS treats expert and learner trajectories as empirical measures over transition space and minimizes a Sinkhorn-regularized Wasserstein distance. Given the transport plan $(s_t,a_t)$0, it defines shaped reward

$(s_t,a_t)$1

Because the reward is computed directly from transport cost, the method dispenses with learned dynamics models and adversarial training, and can be paired with off-policy RL algorithms such as DDPG and TD3 (Chang et al., 2023).

Temporal structure itself can become the bottleneck. ADS identifies “progress dependency” as a failure mode of proxy-reward ILfO: later-step rewards can prevent the learner from mastering earlier prerequisite behaviors. It introduces a progress recognizer based on the Longest Increasing Subsequence of nearest-neighbor matches and sets the discount factor by

$(s_t,a_t)$2

so that early in training the agent is myopic and later rewards are heavily downweighted. As progress increases, the effective horizon expands, producing a curriculum over temporal prefixes. The reported evaluations cover nine Meta-World tasks (Liu et al., 2023).

6. Empirical regimes, assumptions, and emerging directions

The empirical literature spans a wide range of observation regimes. Some methods operate on low-dimensional state features, some on proprioceptive state for policy learning with visual discrimination, and others on raw video. A notable hybrid is “Imitation Learning from Video by Leveraging Proprioception,” where the policy acts on proprioceptive state $(s_t,a_t)$3 via $(s_t,a_t)$4 while a CNN discriminator compares stacked grayscale frames $(s_t,a_t)$5. The reported result is that using vision to define the imitation objective but proprioception to choose actions improves both learning speed and final performance on several MuJoCo continuous-control tasks (Torabi et al., 2019).

Real-world deployment remains comparatively rare but has become more visible. Context translation reports real-world robotic chores and pushing with real videos; LQR+GAIfO reports learning on a physical UR5 arm; RIDM includes a real UR5 reaching task; and VOILA shows that a Clearpath Jackal can imitate a human walking route from a single mobile-phone video under egocentric viewpoint mismatch. VOILA defines imitation reward from SuperPoint keypoint match density and a shaping term

$(s_t,a_t)$6

and the paper also reports successful generalization to perturbed real-world layouts, with degradation under more substantial structural changes (Karnan et al., 2021).

The field’s assumptions are method-dependent and often strong. PVO assumes goal-directed demonstrations and that the last observed state can be treated as a terminal goal state; FORM notes that its current experiments use proprioceptive state rather than pixels; SMODICE requires expert state coverage by the offline dataset; LQR+GAIfO and DEALIO rely on local linear dynamics and linear-Gaussian controllers; ADS depends on the quality of the visual cost $(s_t,a_t)$7 and on reliable progress recognition; context translation assumes enough videos to learn a useful translation model (Edwards et al., 2019, Ma et al., 2022, Liu et al., 2023, Liu et al., 2017).

A broader methodological controversy concerns evaluation protocols. Several papers argue that much of earlier IfO work studied idealized settings, often with bimodal expert-versus-random data or with low-dimensional state observations, and that these regimes can overstate performance. “Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvement” explicitly targets more nuanced data distributions by combining action-free expert trajectories $(s_t,a_t)$8 with an action-labeled background dataset $(s_t,a_t)$9. Its key object is a state value trained on mixed expert and background data,

$$p_\pi(s_i,s_j)=\sum_a P(s_j\mid s_i,a)\,\pi(a\mid s_i)\,p_\pi(s_i),$$0

followed by advantage-weighted regression on the background actions. The paper’s central claim is that self-improvement-like data distributions are more informative than simple expert-versus-random benchmarks for scalable IfO research (Bloesch et al., 9 Jul 2025).

Taken together, the literature suggests a stable conceptual core. IfO is best understood not as a single missing-label variant of imitation learning, but as a family of methods for transferring task structure from observation-only demonstrations into control. The transfer object may be inferred actions, transition occupancies, state occupancies, generative likelihood ratios, Wasserstein couplings, potential-based shaping rewards, or state values. Which formulation is appropriate depends on the observation modality, availability of environment interaction, dynamics knowledge, need for sample efficiency, and the extent of context, embodiment, or temporal mismatch between demonstrator and learner.