---
title: Generalized Adversarial Imitation from Observation
url: https://www.emergentmind.com/topics/generalized-adversarial-imitation-from-observation-gaifo
type: topic
---

# Generalized Adversarial Imitation from Observation

GAIfO, introduced under the title "Generative Adversarial Imitation from Observation," is an imitation-from-observation method in which the learner receives expert demonstrations as state-only trajectories rather than state-action trajectories, and learns by adversarially matching expert and learner state-transition behavior [1807.06158]. The method addresses a weaker and more practical setting than standard imitation learning: the learner does not observe the demonstrator’s actions, so standard behavioral cloning and most inverse reinforcement learning formulations are not directly applicable. Its central move is to replace costs over state-action pairs with costs over state transitions, thereby shifting the imitation target from the demonstrator’s controls to the demonstrator’s effect on the environment [1807.06158].

## 1. Problem setting and motivation

In classical imitation learning, the learner usually observes expert state-action trajectories, so it can either clone actions directly or infer a reward and then solve a reinforcement learning problem. In imitation from observation (IfO), by contrast, the learner only gets state trajectories—demonstrations such as videos or logged states—without access to the expert’s actions [1807.06158]. This difference is central because the absence of action information removes the direct applicability of standard behavioral cloning and most IRL formulations.

GAIfO was motivated by the observation that many real-world data sources, especially internet videos, contain only observations. In that sense, the lack of action labels is not only a restriction but also an opportunity: it expands imitation learning beyond carefully instrumented demonstrations toward abundant observation-only data [1807.06158]. The 2019 survey "Recent Advances in Imitation Learning from Observation" places GAIfO in the model-free adversarial branch of IfO and emphasizes the same contrast with conventional imitation learning, where expert trajectories are typically written as $\tau_e=\{(s_t,a_t)\}$ rather than state-only or observation-only sequences [1905.13566].

A recurring misconception is that state-only imitation can be solved by matching state marginals alone. GAIfO was explicitly motivated against that view. The original paper argues that state distribution matching does not uniquely characterize the expert’s policy, especially in cyclic tasks, and the survey reiterates the same point with the observation that different behaviors can induce the same state distribution [1807.06158; 1905.13566].

## 2. Transition-based inverse reinforcement learning formulation

The general framework proposed with GAIfO modifies the usual IRL formulation from costs over state-action pairs to costs over state transitions. Instead of learning a cost $c:S\times A\to\mathbb{R}$, the method defines a transition-based cost $c:S\times S\to\mathbb{R}$, with the intuition that the learner should match the expert’s effect on the environment, not necessarily the exact controls used [1807.06158].

The IfO inverse RL objective is written as
$$
IRL^{IfO}(\pi_E)=\arg\max_{c\in R^{S\times S}}\left(-\psi(c)+\left(\min_{\pi\in\Pi} \mathbb{E}_{\pi}[c(s,s')]\right)-\mathbb{E}_{\pi_E}[c(s,s')]\right),
$$
and the corresponding RL step is
$$
RL(c)=\arg\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,s')].
$$
This formulation makes the target of imitation explicit: recover a transition cost whose induced policy matches expert state-transition behavior [1807.06158].

GAIfO then defines the state-transition occupancy measure as
$$
p_\pi(s_i,s_j)=\sum_{a} P(s_j\mid s_i,a)\,\pi(a\mid s_i)\,\rho_\pi(s_i),
$$
where $\rho_\pi(s_i)$ is the state visitation distribution under policy $\pi$ [1807.06158]. The key proposition in the paper states that solving the transition-based IRL/RL problem is equivalent to finding a policy whose state-transition occupancy matches the expert’s. This is the formal basis for treating transition matching, rather than action recovery, as the core objective.

This transition-based formulation also explains why GAIfO is presented as a better fit for IfO than state distribution matching alone. Repeated states, cyclic trajectories, and phase ambiguity can make state marginals behaviorally uninformative, whereas transition pairs retain motion information [1807.06158].

## 3. Adversarial objective and practical algorithm

GAIfO instantiates the transition-based framework with a generative adversarial regularizer, adapting the GAIL idea from state-action occupancy matching to state-transition occupancy matching [1807.06158]. The resulting adversarial objective is
$$
V_{GA}(p_\pi-p_{\pi_E}) = \max_{D\in(0,1)^{S\times S}} \mathbb{E}_{p_\pi}[\log D(s,s')] + \mathbb{E}_{p_{\pi_E}}[\log(1-D(s,s'))],
$$
which leads to the optimization
$$
\min_{\pi\in\Pi}\max_{D\in(0,1)^{S\times S}} \mathbb{E}_{\pi}[\log D(s,s')] + \mathbb{E}_{\pi_E}[\log(1-D(s,s'))].
$$

The roles are GAN-like. The discriminator $D$ receives pairs $(s,s')$ and tries to tell expert transitions from agent transitions, while the policy tries to generate transitions that fool the discriminator into classifying them as expert-like [1807.06158]. In the practical algorithm, the discriminator is trained with cross-entropy, and the policy is updated with TRPO using a reward derived from the discriminator:
$$
r(s,s')=-\log D(s,s').
$$

Two implementation points are central. First, expert demonstrations are required only as sequences of states, so expert transitions $(s,s')$ can be formed directly and no action inference is needed. This is explicitly contrasted with BCO, which first learns an inverse dynamics model from self-supervised interaction, uses that model to infer missing actions from demonstrations, and then reduces the problem to standard behavioral cloning [1807.06158]. Second, the method does not require demonstrations to be time-aligned. The paper identifies this as a practical advantage over TCN-based approaches, which often depend on temporal synchronization or a fixed correspondence between demonstration and learner time steps [1807.06158].

In the visual setting, the demonstrations are raw grayscale image sequences. Because images alone may be partially observable, the visual implementation uses stacks of frames from multiple time steps: the policy takes a stack of recent images as input, and the discriminator also operates on short image sequences rather than single images. The paper notes that this is important because velocity or motion information is not available from a single frame [1807.06158].

## 4. Experimental evaluation in low-dimensional and visual domains

The original evaluation covers two settings: demonstrations consisting of low-dimensional, manually defined state features, and demonstrations consisting of high-dimensional, raw visual data [1807.06158]. In the low-dimensional setting, expert policies are trained with TRPO from low-level state features, and demonstrations are recorded as state trajectories. The reported domains are InvertedDoublePendulum, InvertedPendulumSwingup, Hopper, and Walker2D. In the visual setting, the reported domains are Reacher, InvertedPendulum, InvertedPendulumSwingup, and Hopper [1807.06158].

The baseline set includes BCO, TCN, and GAIL. GAIL is not a direct IfO method because it uses expert actions, but it serves as a strong upper baseline. TCN learns a representation with a triplet loss and then defines reward as distance in embedding space. BCO infers actions and then applies behavioral cloning. In the visual experiments, the paper also compares against a TRPO agent trained directly on visual input, shown as a dotted line, to indicate what performance can be achieved if reward is available despite partial observability [1807.06158].

Empirically, the paper reports that GAIfO consistently outperforms the prior IfO methods. In the low-dimensional experiments, it often matches or comes very close to GAIL despite having no access to expert actions, and it substantially outperforms TCN and BCO in most domains. In tasks such as InvertedPendulumSwingup, Hopper, and Walker2D, the paper attributes the gap relative to BCO to BCO’s compounding error and GAIfO’s on-policy learning. TCN is reported to struggle especially on cyclic or unsynchronized tasks because its time-dependent representation does not handle repeated states or phase ambiguity well [1807.06158].

In the visual setting, GAIfO again clearly outperforms BCO and TCN. The paper stresses that its performance is sometimes comparable to a TRPO agent trained with access to reward on visual observations, though still below the expert trained on low-level state features. The authors interpret the remaining gap as arising at least partly from the inherent difficulty of learning from partial visual observations, not only from the imitation algorithm itself [1807.06158].

## 5. Modular reinterpretations and optimization-focused extensions

Subsequent work has repeatedly treated GAIfO as a canonical observation-only adversarial imitation algorithm. "RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning" explicitly states that GAIfO is a RAIL algorithm: in its taxonomy, GAIL corresponds to an on-policy RL backbone with discriminator input $(s_t,a_t)$, whereas GAIfO corresponds to an on-policy RL backbone with discriminator input $(s_t,s_{t+1})$ [2105.03756]. On that basis, the paper introduces SAIfO by keeping the GAIfO-style observation-only discriminator and replacing the TRPO backbone with SAC, reporting better empirical performance on locomotion tasks.

Several later papers target GAIfO’s sample complexity. DEALIO integrates PILQR into the GAIfO framework, keeping the adversarial transition-matching discriminator but replacing the model-free TRPO policy step with a model-based controller update based on time-varying linear Gaussian policies and locally fitted linear Gaussian dynamics [2104.00163]. "Sample-efficient Adversarial Imitation Learning from Observation" follows a similar optimization-centric direction: it retains the GAIfO-style adversarial objective but uses a trajectory-centric, model-based LQR/iLQR controller update and reports improved learning rate and efficiency on a reaching task with a physical UR5 arm and its simulated version in Gazebo [1906.07374].

More recently, "On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration" replaces the built-in RL approach in GAIfO with DSAC, yielding MODULE, which combines a GAIfO-style adversarial reward-learning structure with maximum-entropy exploration and distributional value estimation [2501.12785]. Across these papers, the recurring claim is not that GAIfO’s transition-based discriminator is discarded, but that its RL backbone can be changed to improve sample efficiency or stability.

## 6. Visual, latent, non-adversarial, and application-level descendants

GAIfO has also served as a reference point for methods that keep the state-transition perspective while altering perception, reward learning, or deployment setting.

A first line of work addresses video-only demonstrations. VGAIfO-SO inserts a self-supervised state observer that predicts lower-dimensional proprioceptive state representations from high-dimensional images, and then performs the adversarial imitation objective on predicted state transitions rather than raw pixels [2202.00243]. LAIfO extends the same general idea to partially observable visual imitation by learning a latent representation from sequences of observations and matching latent state-transition distributions with an off-policy adversarial discriminator [2309.17371].

A second line decouples reward modeling from policy learning. IL-flOw models expert state-to-state transitions with a conditional normalizing flow and uses transition log-probability density as reward for forward RL, explicitly presenting itself as an alternative to the instability of adversarial methods [2205.09251]. NEAR likewise replaces the online adversarial discriminator with a pretrained energy model learned from expert state-transition data by denoising score matching, and presents itself as sidestepping the optimization challenges of adversarial imitation learning techniques while remaining in the state-only imitation-from-observation regime [2501.14856].

A third line repurposes GAIfO-like transition matching outside direct policy imitation. GARAT applies an adversarial IfO-style objective to transfer learning with dynamics mismatch by learning an action transformation policy that makes grounded-source transitions resemble target transitions [2008.01594]. DARAIL uses a GAIfO-style state-only adversarial imitation stage to transfer source-domain DARC behavior into a target domain under dynamics shift, with a reward-augmented estimator for policy optimization [2411.09891]. MPAIL goes further by replacing the policy generator with an MPC agent based on MPPI and derives an observation-only adversarial imitation objective over state transitions for planning from observation [2507.21533].

This suggests that the durable part of GAIfO is not only its original TRPO-based implementation, but also its reformulation of observation-only imitation as state-transition occupancy matching.

## 7. Limitations, misconceptions, and significance

GAIfO removes the need for expert actions and for time alignment, but it does not remove the need for environment interaction. The original method is an on-policy adversarial RL algorithm, so the learner must still act in the environment and generate its own transitions during training [1807.06158]. It also assumes that expert and agent trajectories are available as sequences of states, so state transitions $(s,s')$ can be formed.

Another limitation is observability. In the visual setting, the original implementation relies on frame stacks as a workaround for partial observability rather than a full solution, and the paper explicitly notes that entropy regularization is omitted from the main derivation for theoretical simplicity, though it could be useful in practice [1807.06158]. Later visual papers preserve the transition-matching intuition but add latent-state or observer modules precisely because raw visual observations make the original GAIfO formulation sample inefficient or only partially observable [2202.00243; 2309.17371].

It is also important not to conflate GAIfO with action inference methods. BCO first learns an inverse dynamics model and infers missing actions; GAIfO avoids that extra inference stage entirely and instead learns through adversarial matching of transition behavior [1807.06158]. Likewise, GAIfO should not be reduced to single-state distribution matching. The survey literature identifies transition-based discrimination as the key step that distinguishes GAIfO from earlier adversarial IfO methods that match single states and can fail when identical states recur under different behaviors [1905.13566].

Within the IfO literature, GAIfO is therefore best understood as the method that established state-transition occupancy matching as a principled adversarial objective for action-free imitation. Its original claim was that matching expert and learner state-transition distributions with an adversarial objective gives an effective way to do imitation from observation, preserving much of the strength of GAIL while removing the need for expert actions [1807.06158]. Subsequent work has largely developed that claim rather than replaced it: later methods typically modify the optimizer, the representation, or the reward model, but continue to treat transition matching as the central observation-only imitation primitive.

Source: https://www.emergentmind.com/topics/generalized-adversarial-imitation-from-observation-gaifo