---
title: 'HG-DAgger: Human-Gated Imitation Learning'
url: https://www.emergentmind.com/topics/hg-dagger
type: topic
---

# HG-DAgger: Human-Gated Imitation Learning

HG-DAgger is an interactive imitation learning algorithm specifically designed to accommodate human experts in real-world systems. It addresses inherent deficiencies in classical behavioral cloning (BC) and Dataset Aggregation (DAgger), especially in the context of high-stakes domains like autonomous driving, where compounding errors, safety, and human label quality are central concerns. HG-DAgger introduces a principled, human-gated control interface and uncertainty-aware risk estimation, yielding superior safety, sample efficiency, and human-likeness in learned policies [1810.02890].

## 1. Background: Imitation Learning and DAgger

Imitation learning seeks a novice policy $\pi_N$ that closely mimics an expert $\pi_H$, usually through supervised learning on expert demonstrations $\{(x,a_H = \pi_H(x))\}$:
$$
\min_{\theta}\sum_{(x,a_H)\in\mathcal D_{\rm BC}} \|\pi_N^\theta(x) - a_H\|^2.
$$
Behavioral cloning trains $\pi_N$ solely on states encountered by $\pi_H$, causing distributional shift—$\pi_N$ may experience unfamiliar states at test time, leading to compounding errors. DAgger counters this by stochastically interleaving expert and novice actions during data collection, with a Bernoulli gate (probability $\beta$ for expert), aggregating states from both policies and thereby reducing the mismatch.

However, traditional DAgger exposes two limitations in human-in-the-loop scenarios: (1) the expert must provide corrective actions in real time without full system control, which degrades safety and label quality; (2) actuator lag exacerbates issues in dynamic settings.

## 2. Algorithmic Structure of HG-DAgger

HG-DAgger modifies DAgger to ensure that the human expert can gate control at will, fully intervening in real time when the novice policy enters perceived unsafe regions. The components are:

- **Gating Rule**: The human defines on-the-fly a permitted set of states $\mathcal P \subseteq \mathcal X$; the gating function operates as
  $$
  g(x_t) = \begin{cases}
    1, & x_t \notin \mathcal P \\
    0, & x_t \in \mathcal P
  \end{cases}
  $$
  where $g(x_t) = 1$ denotes expert control, $g(x_t) = 0$ denotes novice control.

- **Policy Rollout**: The joint control policy during data gathering is
  $$
  \pi_i(x_t) = g(x_t)\pi_H(x_t) + [1-g(x_t)]\pi_{N_i}(o_t),
  $$
  where $o_t = \mathcal O(x_t)$ is the observation available to the novice.

- **Uncertainty (‘Doubt’) Metric**: The novice $\pi_N$ is an ensemble of $M$ neural networks, each producing action $a_t^{(k)}$ for input $o_t$. Empirical covariance is computed as
  $$
  C_t = \frac{1}{M-1}\sum_{k=1}^M (a_t^{(k)} - \bar a_t)(a_t^{(k)} - \bar a_t)^\top,\quad \bar a_t = \frac{1}{M}\sum_k a_t^{(k)}.
  $$
  The scalar doubt is defined as
  $$
  d_N(o_t) = \|\mathrm{diag}(C_t)\|_2,
  $$
  where higher $d_N$ indicates regions of epistemic uncertainty—states with a risk of poor novice performance.

- **Learning the Safety Threshold $\tau$**: Doubt values at human interventions are accumulated in a log $\mathcal I$. After all data collection epochs, the safety threshold is set to the average of the top quartile of intervention-time doubts:
  $$
  \tau = \frac{1}{N/4} \sum_{i = \lfloor 0.75N \rfloor}^{N} \mathcal I[i].
  $$
  This threshold acts as a data-driven risk certificate for the trained novice.

## 3. Training Loop and Data Aggregation

HG-DAgger employs an iterative learning procedure integrating supervised updates, human gating, uncertainty logging, and risk threshold inference. The core loop is:

1. Start from an initial behavioral cloning dataset $\mathcal D_{\rm BC}$, and initialize policy $\pi_{N_1}$.
2. For each epoch $i$ over $K$ epochs and $M$ rollouts:
    - At each timestep $t$:
        - The gating function $g(x_t)$ determines whether the expert or novice acts.
        - When expert intervenes ($g(x_t)=1$): execute expert action, augment dataset and log current doubt in $\mathcal I$.
        - Otherwise, apply novice action.
    - Retrain the novice on the enlarged dataset.
3. After training, compute $\tau$ from $\mathcal I$.

This procedure preserves uninterrupted human control during interventions, capturing high-integrity expert labels and enabling the system to learn both safe state distributions and actionable risk certificates.

## 4. Novice Policy and Uncertainty Estimation Architecture

The novice $\pi_N$ is instantiated as an ensemble of feed-forward neural networks. Each subnetwork processes the concatenated observation vector
$$
o = [y, \theta, s, l_l, l_r, d_l, d_r]
$$
where the components represent lateral/heading/yaw, lane distances, and obstacle proximities. The architecture comprises two hidden layers (128–256 units, ReLU activation), with a continuous action output of (steering, speed). The ensemble mean prescribes the novice action; ensemble covariance estimates the epistemic uncertainty, closely approximating a scalable Gaussian process for risk estimation.

## 5. Experimental Setup and Evaluation Metrics

HG-DAgger was evaluated on both simulated and real-world autonomous driving tasks:

- **Simulation**: Two-lane road with static obstacle cars; novice navigates randomly lane-blocked sequences.
- **Real vehicle**: MG-GS car equipped with LiDAR, precision localization, onboard safety driver, and an off-board human expert for HG-DAgger interventions.

Baselines included:
- Behavioral Cloning (BC) with $N_0=10^4$ demonstration labels.
- Standard DAgger with $\beta_0=0.85$, decayed by $0.85$ per epoch.

Metrics:
- Collision rate (collisions per meter traveled)
- Road departure rate and mean duration
- Steering-angle distribution this relative to human reference (Bhattacharyya distance)

## 6. Empirical Results and Analysis

In simulation, after a total of $2 \times 10^4$ sampled states (BC + DAgger/HG-DAgger), the following rates were observed:

| Method        | Road Departure (m⁻¹) | Collision (m⁻¹) | Bhattacharyya Distance |
|---------------|----------------------|-----------------|-----------------------|
| BC            | $\approx 1.2\times10^{-3}$   | $\approx 0.9\times10^{-3}$    | 0.1173                |
| DAgger        | $\approx 1.5\times10^{-3}$   | $\approx 1.2\times10^{-3}$    | 0.1057                |
| HG-DAgger     | $\approx 0.5\times10^{-3}$   | $\approx 0.3\times10^{-3}$    | 0.0834                |

DAgger suffered from late-epoch instability, plausibly due to degraded label quality under stochastic action gating, whereas HG-DAgger achieved more stable learning. Risk-threshold validation—partitioning states into estimated safe $\hat{\mathcal P} = \{x: d_N(o) \leq \tau\}$ and unsafe regions—demonstrated that $\tau$ robustly separates high- and low-risk outcomes: inside $\hat{\mathcal P}$, collision/road-departure rates were $0.607\times 10^{-3}$; outside, rates rose to $7.53\times 10^{-3}$ and $12.09\times 10^{-3}$ per meter, respectively.

On-vehicle, HG-DAgger achieved zero collisions and zero departures, outperforming both BC and DAgger.

## 7. Discussion, Limitations, and Prospective Directions

HG-DAgger presents a practical solution for real-world imitation learning with human experts. By preserving the expert’s ability to intervene at will, the algorithm assures higher-quality expert demonstrations and safer data collection. The risk threshold $\tau$, learned directly from intervention-time uncertainty, operates as an actionable certificate for downstream safety filtering.

Limitations include reliance on the expert’s real-time discrimination of unsafe states and absence of formal regret or safety proofs. The ensemble-based uncertainty metric, while tractable, may undersample epistemic uncertainty. Prospective directions include automating the gating process using the learned $d_N(o) > \tau$ rule, employing richer uncertainty estimators such as Bayesian neural networks or MC-dropout, and extending the methodology to multi-modal or formally verifiable safety-critical domains [1810.02890].

Source: https://www.emergentmind.com/topics/hg-dagger