Generalized Adversarial Imitation from Observation

• GAIfO is an imitation learning framework that leverages adversarial state-transition matching to learn policies solely from state trajectories.
• It reformulates inverse reinforcement learning by shifting from state-action pair costs to state-transition costs, effectively handling cyclic behaviors and partial observability.
• Empirical results demonstrate that GAIfO often matches or outperforms other observation-based imitation methods in both low-dimensional and visual domains.

GAIfO, introduced under the title "Generative Adversarial Imitation from Observation," is an imitation-from-observation method in which the learner receives expert demonstrations as state-only trajectories rather than state-action trajectories, and learns by adversarially matching expert and learner state-transition behavior (Torabi et al., 2018). The method addresses a weaker and more practical setting than standard imitation learning: the learner does not observe the demonstrator’s actions, so standard behavioral cloning and most inverse reinforcement learning formulations are not directly applicable. Its central move is to replace costs over state-action pairs with costs over state transitions, thereby shifting the imitation target from the demonstrator’s controls to the demonstrator’s effect on the environment (Torabi et al., 2018).

1. Problem setting and motivation

In classical imitation learning, the learner usually observes expert state-action trajectories, so it can either clone actions directly or infer a reward and then solve a reinforcement learning problem. In imitation from observation (IfO), by contrast, the learner only gets state trajectories—demonstrations such as videos or logged states—without access to the expert’s actions (Torabi et al., 2018). This difference is central because the absence of action information removes the direct applicability of standard behavioral cloning and most IRL formulations.

GAIfO was motivated by the observation that many real-world data sources, especially internet videos, contain only observations. In that sense, the lack of action labels is not only a restriction but also an opportunity: it expands imitation learning beyond carefully instrumented demonstrations toward abundant observation-only data (Torabi et al., 2018). The 2019 survey "Recent Advances in Imitation Learning from Observation" places GAIfO in the model-free adversarial branch of IfO and emphasizes the same contrast with conventional imitation learning, where expert trajectories are typically written as $\tau_e=\{(s_t,a_t)\}$ rather than state-only or observation-only sequences (Torabi et al., 2019).

A recurring misconception is that state-only imitation can be solved by matching state marginals alone. GAIfO was explicitly motivated against that view. The original paper argues that state distribution matching does not uniquely characterize the expert’s policy, especially in cyclic tasks, and the survey reiterates the same point with the observation that different behaviors can induce the same state distribution (Torabi et al., 2018, Torabi et al., 2019).

2. Transition-based inverse reinforcement learning formulation

The general framework proposed with GAIfO modifies the usual IRL formulation from costs over state-action pairs to costs over state transitions. Instead of learning a cost $c:S\times A\to\mathbb{R}$, the method defines a transition-based cost $c:S\times S\to\mathbb{R}$, with the intuition that the learner should match the expert’s effect on the environment, not necessarily the exact controls used (Torabi et al., 2018).

The IfO inverse RL objective is written as

$$IRL^{IfO}(\pi_E)=\arg\max_{c\in R^{S\times S}}\left(-\psi(c)+\left(\min_{\pi\in\Pi} \mathbb{E}_{\pi}[c(s,s')]\right)-\mathbb{E}_{\pi_E}[c(s,s')]\right),$$

and the corresponding RL step is

$$RL(c)=\arg\min_{\pi\in\Pi}\mathbb{E}_{\pi}[c(s,s')].$$

This formulation makes the target of imitation explicit: recover a transition cost whose induced policy matches expert state-transition behavior (Torabi et al., 2018).

GAIfO then defines the state-transition occupancy measure as

$$p_\pi(s_i,s_j)=\sum_{a} P(s_j\mid s_i,a)\,\pi(a\mid s_i)\,\rho_\pi(s_i),$$

where $\rho_\pi(s_i)$ is the state visitation distribution under policy $\pi$ (Torabi et al., 2018). The key proposition in the paper states that solving the transition-based IRL/RL problem is equivalent to finding a policy whose state-transition occupancy matches the expert’s. This is the formal basis for treating transition matching, rather than action recovery, as the core objective.

This transition-based formulation also explains why GAIfO is presented as a better fit for IfO than state distribution matching alone. Repeated states, cyclic trajectories, and phase ambiguity can make state marginals behaviorally uninformative, whereas transition pairs retain motion information (Torabi et al., 2018).

3. Adversarial objective and practical algorithm

GAIfO instantiates the transition-based framework with a generative adversarial regularizer, adapting the GAIL idea from state-action occupancy matching to state-transition occupancy matching (Torabi et al., 2018). The resulting adversarial objective is

$$V_{GA}(p_\pi-p_{\pi_E}) = \max_{D\in(0,1)^{S\times S}} \mathbb{E}_{p_\pi}[\log D(s,s')] + \mathbb{E}_{p_{\pi_E}}[\log(1-D(s,s'))],$$

which leads to the optimization

$$\min_{\pi\in\Pi}\max_{D\in(0,1)^{S\times S}} \mathbb{E}_{\pi}[\log D(s,s')] + \mathbb{E}_{\pi_E}[\log(1-D(s,s'))].$$

The roles are GAN-like. The discriminator $c:S\times A\to\mathbb{R}$0 receives pairs $c:S\times A\to\mathbb{R}$1 and tries to tell expert transitions from agent transitions, while the policy tries to generate transitions that fool the discriminator into classifying them as expert-like (Torabi et al., 2018). In the practical algorithm, the discriminator is trained with cross-entropy, and the policy is updated with TRPO using a reward derived from the discriminator:

$c:S\times A\to\mathbb{R}$2

Two implementation points are central. First, expert demonstrations are required only as sequences of states, so expert transitions $c:S\times A\to\mathbb{R}$3 can be formed directly and no action inference is needed. This is explicitly contrasted with BCO, which first learns an inverse dynamics model from self-supervised interaction, uses that model to infer missing actions from demonstrations, and then reduces the problem to standard behavioral cloning (Torabi et al., 2018). Second, the method does not require demonstrations to be time-aligned. The paper identifies this as a practical advantage over TCN-based approaches, which often depend on temporal synchronization or a fixed correspondence between demonstration and learner time steps (Torabi et al., 2018).

In the visual setting, the demonstrations are raw grayscale image sequences. Because images alone may be partially observable, the visual implementation uses stacks of frames from multiple time steps: the policy takes a stack of recent images as input, and the discriminator also operates on short image sequences rather than single images. The paper notes that this is important because velocity or motion information is not available from a single frame (Torabi et al., 2018).

4. Experimental evaluation in low-dimensional and visual domains

The original evaluation covers two settings: demonstrations consisting of low-dimensional, manually defined state features, and demonstrations consisting of high-dimensional, raw visual data (Torabi et al., 2018). In the low-dimensional setting, expert policies are trained with TRPO from low-level state features, and demonstrations are recorded as state trajectories. The reported domains are InvertedDoublePendulum, InvertedPendulumSwingup, Hopper, and Walker2D. In the visual setting, the reported domains are Reacher, InvertedPendulum, InvertedPendulumSwingup, and Hopper (Torabi et al., 2018).

The baseline set includes BCO, TCN, and GAIL. GAIL is not a direct IfO method because it uses expert actions, but it serves as a strong upper baseline. TCN learns a representation with a triplet loss and then defines reward as distance in embedding space. BCO infers actions and then applies behavioral cloning. In the visual experiments, the paper also compares against a TRPO agent trained directly on visual input, shown as a dotted line, to indicate what performance can be achieved if reward is available despite partial observability (Torabi et al., 2018).

Empirically, the paper reports that GAIfO consistently outperforms the prior IfO methods. In the low-dimensional experiments, it often matches or comes very close to GAIL despite having no access to expert actions, and it substantially outperforms TCN and BCO in most domains. In tasks such as InvertedPendulumSwingup, Hopper, and Walker2D, the paper attributes the gap relative to BCO to BCO’s compounding error and GAIfO’s on-policy learning. TCN is reported to struggle especially on cyclic or unsynchronized tasks because its time-dependent representation does not handle repeated states or phase ambiguity well (Torabi et al., 2018).

In the visual setting, GAIfO again clearly outperforms BCO and TCN. The paper stresses that its performance is sometimes comparable to a TRPO agent trained with access to reward on visual observations, though still below the expert trained on low-level state features. The authors interpret the remaining gap as arising at least partly from the inherent difficulty of learning from partial visual observations, not only from the imitation algorithm itself (Torabi et al., 2018).

5. Modular reinterpretations and optimization-focused extensions

Subsequent work has repeatedly treated GAIfO as a canonical observation-only adversarial imitation algorithm. "RAIL: A modular framework for Reinforcement-learning-based Adversarial Imitation Learning" explicitly states that GAIfO is a RAIL algorithm: in its taxonomy, GAIL corresponds to an on-policy RL backbone with discriminator input $c:S\times A\to\mathbb{R}$4, whereas GAIfO corresponds to an on-policy RL backbone with discriminator input $c:S\times A\to\mathbb{R}$5 (Hudson et al., 2021). On that basis, the paper introduces SAIfO by keeping the GAIfO-style observation-only discriminator and replacing the TRPO backbone with SAC, reporting better empirical performance on locomotion tasks.

Several later papers target GAIfO’s sample complexity. DEALIO integrates PILQR into the GAIfO framework, keeping the adversarial transition-matching discriminator but replacing the model-free TRPO policy step with a model-based controller update based on time-varying linear Gaussian policies and locally fitted linear Gaussian dynamics (Torabi et al., 2021). "Sample-efficient Adversarial Imitation Learning from Observation" follows a similar optimization-centric direction: it retains the GAIfO-style adversarial objective but uses a trajectory-centric, model-based LQR/iLQR controller update and reports improved learning rate and efficiency on a reaching task with a physical UR5 arm and its simulated version in Gazebo (Torabi et al., 2019).

More recently, "On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration" replaces the built-in RL approach in GAIfO with DSAC, yielding MODULE, which combines a GAIfO-style adversarial reward-learning structure with maximum-entropy exploration and distributional value estimation (Zhou et al., 22 Jan 2025). Across these papers, the recurring claim is not that GAIfO’s transition-based discriminator is discarded, but that its RL backbone can be changed to improve sample efficiency or stability.

6. Visual, latent, non-adversarial, and application-level descendants

GAIfO has also served as a reference point for methods that keep the state-transition perspective while altering perception, reward learning, or deployment setting.

A first line of work addresses video-only demonstrations. VGAIfO-SO inserts a self-supervised state observer that predicts lower-dimensional proprioceptive state representations from high-dimensional images, and then performs the adversarial imitation objective on predicted state transitions rather than raw pixels (Karnan et al., 2022). LAIfO extends the same general idea to partially observable visual imitation by learning a latent representation from sequences of observations and matching latent state-transition distributions with an off-policy adversarial discriminator (Giammarino et al., 2023).

A second line decouples reward modeling from policy learning. IL-flOw models expert state-to-state transitions with a conditional normalizing flow and uses transition log-probability density as reward for forward RL, explicitly presenting itself as an alternative to the instability of adversarial methods (Chang et al., 2022). NEAR likewise replaces the online adversarial discriminator with a pretrained energy model learned from expert state-transition data by denoising score matching, and presents itself as sidestepping the optimization challenges of adversarial imitation learning techniques while remaining in the state-only imitation-from-observation regime (Diwan et al., 24 Jan 2025).

A third line repurposes GAIfO-like transition matching outside direct policy imitation. GARAT applies an adversarial IfO-style objective to transfer learning with dynamics mismatch by learning an action transformation policy that makes grounded-source transitions resemble target transitions (Desai et al., 2020). DARAIL uses a GAIfO-style state-only adversarial imitation stage to transfer source-domain DARC behavior into a target domain under dynamics shift, with a reward-augmented estimator for policy optimization (Guo et al., 2024). MPAIL goes further by replacing the policy generator with an MPC agent based on MPPI and derives an observation-only adversarial imitation objective over state transitions for planning from observation (Han et al., 29 Jul 2025).

This suggests that the durable part of GAIfO is not only its original TRPO-based implementation, but also its reformulation of observation-only imitation as state-transition occupancy matching.

7. Limitations, misconceptions, and significance

GAIfO removes the need for expert actions and for time alignment, but it does not remove the need for environment interaction. The original method is an on-policy adversarial RL algorithm, so the learner must still act in the environment and generate its own transitions during training (Torabi et al., 2018). It also assumes that expert and agent trajectories are available as sequences of states, so state transitions $c:S\times A\to\mathbb{R}$6 can be formed.

Another limitation is observability. In the visual setting, the original implementation relies on frame stacks as a workaround for partial observability rather than a full solution, and the paper explicitly notes that entropy regularization is omitted from the main derivation for theoretical simplicity, though it could be useful in practice (Torabi et al., 2018). Later visual papers preserve the transition-matching intuition but add latent-state or observer modules precisely because raw visual observations make the original GAIfO formulation sample inefficient or only partially observable (Karnan et al., 2022, Giammarino et al., 2023).

It is also important not to conflate GAIfO with action inference methods. BCO first learns an inverse dynamics model and infers missing actions; GAIfO avoids that extra inference stage entirely and instead learns through adversarial matching of transition behavior (Torabi et al., 2018). Likewise, GAIfO should not be reduced to single-state distribution matching. The survey literature identifies transition-based discrimination as the key step that distinguishes GAIfO from earlier adversarial IfO methods that match single states and can fail when identical states recur under different behaviors (Torabi et al., 2019).

Within the IfO literature, GAIfO is therefore best understood as the method that established state-transition occupancy matching as a principled adversarial objective for action-free imitation. Its original claim was that matching expert and learner state-transition distributions with an adversarial objective gives an effective way to do imitation from observation, preserving much of the strength of GAIL while removing the need for expert actions (Torabi et al., 2018). Subsequent work has largely developed that claim rather than replaced it: later methods typically modify the optimizer, the representation, or the reward model, but continue to treat transition matching as the central observation-only imitation primitive.