HILONet: Hierarchical Imitation Learning Framework

• HILONet is a hierarchical imitation learning framework that overcomes the need for temporal alignment by dynamically selecting sub-goals from multiple expert observation trajectories.
• It employs a two-level architecture where a high-level policy selects demonstration sub-goals and a low-level policy uses goal-conditioned control with engineered observation-based rewards.
• Experimental results in diverse environments demonstrate HILONet’s efficiency in handling non-aligned demonstration data while mitigating hierarchical non-stationarity via techniques like hindsight transitions and asynchronous updates.

HILONet, short for Hierarchical Imitation Learning from Observation, is an imitation learning framework for settings in which demonstration trajectories contain only observations and are not temporally aligned with the learner’s execution. It addresses Imitation Learning from Observation (ILfO) under the assumption that the agent observes its own state $o_t$, while demonstrations $D=\{\tau_1,\dots,\tau_N\}$ contain only observation sequences $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$, with no expert actions and no external reward used for training. Its central design choice is a two-level hierarchical architecture that replaces step-by-step temporal alignment with dynamic sub-goal selection from multiple expert trajectories, allowing the agent to choose feasible demonstration frames and learn to achieve them over a fixed horizon (Liu et al., 2020).

1. Problem formulation and motivation

HILONet is defined for non-time-aligned imitation from observation. In this setting, the demonstrations are observation-only trajectories of varying lengths, and different trajectories cannot be matched step-by-step. The framework is motivated by the observation that many imitation learning methods presuppose temporal alignment, or at least a consistent step index, between the learner and the expert. According to the formulation in the paper, this assumption becomes problematic when episode lengths vary, when the agent has different dynamics or embodiment, or when multiple demonstrations provide multiple candidate observations per time step (Liu et al., 2020).

The method therefore departs from alignment-based ILfO. Rather than attempting to reproduce the exact expert frame at time $t$, it chooses sub-goal observations directly from a set of demonstrations and treats imitation as a sequence of feasible observation-reaching problems. This suggests a shift from strict trajectory replay to hierarchical progress through the demonstrated observation manifold. A plausible implication is that the method is particularly suited to scenarios in which demonstrations are informative at the level of key observations but unreliable as synchronized temporal scripts.

The paper characterizes the training signals as engineered observation-based rewards. No expert actions or environment rewards are used. The resulting learning problem is thus neither standard behavior cloning nor adversarial ILfO in the usual sense, but a hierarchical control problem induced entirely by observation proximity and trajectory-progress structure.

2. Hierarchical architecture and sub-goal representation

HILONet uses a two-level architecture consisting of a high-level policy for sub-goal selection and a low-level policy for goal-conditioned control (Liu et al., 2020). The relevant variables are:

• $o_t$: the agent’s current observation at time $t$
• $D=\{\tau_i\}$: the set of $N$ expert trajectories, with $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$
• $o_g$: the sub-goal observation chosen from $D=\{\tau_1,\dots,\tau_N\}$0
• $D=\{\tau_1,\dots,\tau_N\}$1: the environment action produced by the low-level policy

The high-level policy outputs a 2D continuous action $D=\{\tau_1,\dots,\tau_N\}$2 that indexes into the demonstration set through the normalized trajectory index

$D=\{\tau_1,\dots,\tau_N\}$3

The selection rule is written as

$D=\{\tau_1,\dots,\tau_N\}$4

so the first coordinate selects a demonstration trajectory and the second selects a frame within that trajectory. The high-level policy is trained with DDPG.

The low-level policy is a goal-conditioned controller: $D=\{\tau_1,\dots,\tau_N\}$5 It receives the current observation and the selected sub-goal observation, and attempts to reach $D=\{\tau_1,\dots,\tau_N\}$6 within a fixed horizon. In the experiments, the option duration is five steps, and the high-level emits a new sub-goal every $D=\{\tau_1,\dots,\tau_N\}$7 steps, with $D=\{\tau_1,\dots,\tau_N\}$8.

Sub-goal completion is defined in observation space through a threshold criterion: $D=\{\tau_1,\dots,\tau_N\}$9 If this condition holds, the sub-goal is considered achieved; otherwise it is not. This explicit completion rule is also the basis for the reward structure at both levels. The architecture thereby couples a discrete-like selection process over demonstrated observations with continuous control over the environment action space.

3. Learning objectives and alignment-free reward design

Both levels are trained using observation-driven engineered rewards and off-policy DDPG updates (Liu et al., 2020). The low-level reward is designed to encourage progress toward the selected sub-goal while also providing a sparse completion bonus: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$0 where $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$1 is a positive constant sparse reward.

The high-level reward is defined over sub-goal transitions separated by $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$2 steps. It rewards sub-goals that are achieved by the low-level and that advance the agent along a demonstration sequence: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$3 with the convention that $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$4 if $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$5 is not in expert trajectories. The high-level reward is

$\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$6

where $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$7 controls the behavior mode.

This reward design is the mechanism by which HILONet handles non-aligned demonstrations without explicit time matching. The paper identifies three ingredients in this alignment-free handling: the high level selects $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$8 directly from any frame in any trajectory, sparse achievement-based rewards enforce feasibility, and the normalized index $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$9 makes progress comparable across trajectories of different lengths. The use of $t$0 is critical here, because it converts raw frame position into a length-normalized progress signal.

For optimization, the paper recalls both REINFORCE and deterministic policy gradient forms: $t$1

$t$2

In HILONet, the practical learning rule is off-policy DDPG at both levels, with separate replay buffers and target networks.

4. Behavior-mode control through $t$3

A distinctive aspect of HILONet is the explicit role of $t$4 as a behavior-mode dial (Liu et al., 2020). The paper analyzes two optimal policies, $t$5 and $t$6, both of which reach the final goal with the same low-level ability, but with different high-level behavior: $t$7 follows the expert trajectory closely, whereas $t$8 reaches the same final goal through a novel path.

The optimal value at a state is written as

$t$9

and, after substituting the high-level reward, the values of the two policies are expressed as

$o_t$0

$o_t$1

The difference is then

$o_t$2

with the paper further noting that

$o_t$3

From this, it concludes that $o_t$4 increases monotonically with $o_t$5.

The practical interpretation given in the paper is direct: higher $o_t$6 magnifies return differences favoring policies that follow demonstration sequences closely, while lower $o_t$7 reduces this pressure and encourages exploration and novel paths. In the reported implementation, $o_t$8 is used for single-goal tasks and $o_t$9 for sequenced-goal tasks. This suggests that HILONet is not only an alignment-free imitation learner, but also a controllable one whose imitation-exploration trade-off is embedded in the reward design rather than imposed externally.

5. Sample-efficiency mechanisms and training procedure

To address hierarchical non-stationarity, HILONet introduces three sample-efficiency methods (Liu et al., 2020). The source of non-stationarity is that the low-level policy changes during training, which alters the transition distribution experienced by the high level.

The first method is Hindsight Transitions, applied at both levels. At the high level, the original transition

$t$0

is replaced, when $t$1 is in the demonstrations, by

$t$2

At the low level, the original transition

$t$3

is relabeled as

$t$4

when $t$5 is a demonstration observation. The paper describes this as reducing non-stationarity in $t$6 and increasing dense rewards in $t$7 by relabeling failed attempts as success cases for the achieved observation.

The second method is Asynchronous Delayed Update (ADU). The low-level policy is trained every step, whereas the high-level policy is trained with a time delay, described as every other step or, more generally, “double steps.” The paper states that this is analogous to, but distinct from, TD3’s delayed actor updates, and that it is particularly suitable because the high-level does not act directly in the environment but selects sub-goals.

The third method is a Smaller High-level Replay Buffer. The high-level buffer is intentionally kept smaller so that stale transitions reflecting outdated low-level behavior are not overused. The exact sizes are not specified.

The training procedure initializes $t$8 and $t$9, collects $D=\{\tau_i\}$0 using an expert policy, initializes separate replay buffers $D=\{\tau_i\}$1 and $D=\{\tau_i\}$2, and constructs high- and low-level Q-functions with target networks. At each step, the high level selects a sub-goal, the low level executes $D=\{\tau_i\}$3, rewards are computed, hindsight replacements are applied when appropriate, transitions are stored, mini-batches are sampled, and the two levels are updated with different schedules. The implementation uses two-layer fully connected policies with 64 units per layer and ReLU activation, with Q-networks mirroring the policy style. All tasks use standard environment state vectors rather than visual observations.

6. Experimental evaluation and empirical findings

The evaluation covers five non-time-aligned environments: MountainCar, LunarLander, and Swimmer from Gym, and Reacher and 3Dball from Unity ML-Agents (Liu et al., 2020). MountainCar and LunarLander are described as single-goal tasks; Swimmer and 3Dball as sequenced-goals tasks; and Reacher as a mixing task that requires reaching and maintaining. Demonstrations comprise 20 trajectories for MountainCar and 30 trajectories for LunarLander, Swimmer, 3Dball, and Reacher. TSRE uses only one trajectory due to its step-by-step structure, whereas HILONet uses all available trajectories.

The baselines are GAIL, GAILfO, and TSRE. The paper evaluates environment return, including success rate and cumulative reward proxies as standard RL evaluation metrics, with training curves averaged over three seeds.

Method	Environments/results
GAIL	MountainCar: $D=\{\tau_i\}$4; LunarLander: $D=\{\tau_i\}$5; Reacher: $D=\{\tau_i\}$6; 3Dball: $D=\{\tau_i\}$7; Swimmer: $D=\{\tau_i\}$8
GAILfO / TSRE	GAILfO — MountainCar: $D=\{\tau_i\}$9; LunarLander: $N$0; Reacher: $N$1; 3Dball: $N$2; Swimmer: $N$3. TSRE — MountainCar: $N$4; LunarLander: $N$5; Reacher: $N$6; 3Dball: $N$7; Swimmer: $N$8
HILONet	MountainCar: $N$9; LunarLander: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$0; Reacher: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$1; 3Dball: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$2; Swimmer: $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$3

The reported findings are structured by task type. On single-goal tasks, HILONet outperforms GAILfO and TSRE, and GAIL remains strongest overall. On sequenced-goal tasks, large $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$4 induces closer adherence to demonstration sequences and required action patterns, allowing HILONet to outperform GAILfO and TSRE; in 3Dball it also surpasses GAIL. On the mixing task Reacher, HILONet again exceeds the ILfO baselines and approaches GAIL.

The paper also reports a trajectory visualization for LunarLander: over training, the high-level sub-goal selections diversify beyond a single trajectory and the low-level achieves them, demonstrating exploration of novel successful paths. This is consistent with the intended behavior of dynamic multi-trajectory sub-goal composition.

Ablation results indicate that removing hindsight transitions or delayed update causes severe performance degradation, while larger high-level replay buffers also degrade performance but less drastically. The stated conclusion is that combating hierarchical non-stationarity is crucial for sample-efficient learning.

7. Robustness, limitations, and relation to neighboring methods

The empirical claims in the paper distinguish between robustness across task structures and sensitivity to design choices (Liu et al., 2020). For single-goal tasks, lower $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$5 encourages exploration of new paths to a terminal observation; for sequenced-goal tasks, higher $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$6 promotes faithful following of demonstration-frame order and supports the learning of complex action patterns necessary for maintaining desired states. The normalized progress reward $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$7 is presented as the mechanism that enables training on multiple demonstrations of varying lengths without time alignment.

At the same time, the paper states several limitations. HILONet relies on an observation-space proximity metric, $\tau_i=\{d_1^i,\dots,d_{T_i}^i\}$8, and on a Euclidean distance-based low-level reward; these may be inadequate if raw observations are not aligned with task semantics. Sub-goals are restricted to frames present in demonstrations, so discovering unseen but beneficial sub-goals is not covered. The mapping from continuous indices to discrete trajectory and frame selection may require careful handling such as rounding or clipping, though this is not elaborated. The method also assumes access to multiple high-quality demonstration trajectories, and performance may depend on demonstration diversity and coverage.

The future-work directions named in the paper are extension to multi-level hierarchies and applications in robotics and autonomous driving with richer observation modalities. A plausible implication is that the present formulation is best viewed as a hierarchical ILfO template whose current instantiation is strongest when observation geometry is already informative and demonstrations span the relevant task progression.

The paper situates HILONet relative to several neighboring lines of work: ILfO baselines such as GAILfO, model-based ILfO such as BCO and ILPO, reward-engineering approaches such as TCN and related frame-order methods including shuffle-and-learn, hierarchical RL approaches including the Options framework, HIRO, Option-Critic, FeUdal Networks, and HAC with hindsight, and generative adversarial imitation learning through GAIL. Within that landscape, HILONet is characterized by its combination of hierarchical sub-goal selection from multiple demonstration trajectories, observation-based rewards for feasibility and progress, and explicit mechanisms for mitigating hierarchical non-stationarity.