Introspective Imitation Learning via NEIL

• Introspective imitation learning is a human-in-the-loop method that uses a model’s uncertainty to selectively query user corrections, ensuring annotation efficiency.
• The NEIL algorithm integrates threshold-based uncertainty filtering with iterative data aggregation, reducing annotation costs while nearly matching supervised performance.
• Empirical evaluations on WikiSQL and Spider datasets show that NEIL achieves near full-supervised accuracy with significantly fewer annotations.

Introspective @@@@1@@@@ is a human-in-the-loop methodology for training semantic parsers, in which the model actively introspects its own uncertainty and queries the user for demonstration only at uncertain decision points. By carefully mixing high-confidence self-predictions with user-provided corrections, this approach achieves significant annotation efficiency while maintaining near-supervised performance. The paradigm is instantiated through the NEIL (Non-Exhaustive Imitation Learning) algorithm, with formal analysis and empirical results demonstrating its practical benefits in data-sparse semantic parsing scenarios (Yao et al., 2020).

1. Interactive Human-in-the-Loop Framework and Uncertainty Thresholding

A semantic parser is embedded within an interactive loop, such as the MISP framework. At each timestep $t$, the parser executes its policy $\hat\pi$ (a conditional distribution over actions $a$ given state $s_t$):

• The top action is proposed:

$a_t = \arg\max_a\;\hat\pi(a\mid s_t)$

• The model computes its confidence via the posterior probability:

$p_{\hat\pi}(a_t\mid s_t)$

When $p_{\hat\pi}(a_t\mid s_t)$ falls below a fixed threshold $\mu$, e.g., $\mu = 0.95$ for SQLova or $\mu = 0.995$ for EditSQL, the parser designates itself as "uncertain" and queries the user for the correct action $a_t^*$. Otherwise, it executes $a_t$ autonomously. This decision rule ensures user annotation effort is concentrated only where the model lacks confidence:

$$\begin{cases} \text{if }p_{\hat\pi}(a_t\mid s_t)\ge\mu: & \text{execute }a_t\ (\text{confident}) \ \text{if }p_{\hat\pi}(a_t\mid s_t)<\mu: & \text{query user, execute }a_t^*\ (\text{uncertain}) \end{cases}$$

This introspective mechanism enables annotation-efficient bootstrapping and online improvement.

2. NEIL: Annotation-Efficient Imitation Learning Algorithm

The NEIL algorithm operates in alternating phases of interaction and retraining, following a DAgger-style data aggregation loop but querying the user only on uncertain states. The procedure is as follows:

• Initialization: Start from a seed dataset $D_0$, threshold $\mu$, and an initial policy $\hat\pi_1$ trained on $D_0$.
• Data Collection: For each of $N$ iterations, interact with $m$ live user queries using policy $\hat\pi_i$. For each state $s_t$:
  • If $p_{\hat\pi_i}(a_t\mid s_t)\ge\mu$, accept $a_t$ as a pseudo-gold demonstration ($w_t = 1$).
  • If $p_{\hat\pi_i}(a_t\mid s_t)<\mu$, query the user for $a_t^*$: if valid, add as demonstration ($w_t=1$); if invalid (user chooses "None of the above"), discard ($w_t=0$).
• Aggregation and Retraining: Aggregate $D \leftarrow D \cup D_i$ and retrain:

$$\hat\pi_{i+1} = \arg\min_\pi\; \sum_{(s,\tilde a, w)\in D} -w\;\log\pi(\tilde a\mid s)$$

• Selection: The best $\hat\pi_i$ on validation is returned.

Pseudo-code summarizes the above workflow and experimental details confirm all parameters and thresholds (Yao et al., 2020).

3. Theoretical Guarantees: Annotation Cost Bound

A rigorous analysis provides cost bounds on NEIL relative to full supervised imitation:

• The one-step loss is defined as:

$$\ell(s,\hat\pi) = 1 - p_{\hat\pi}(a^*\mid s),\quad a^* \sim \pi^*(s)$$

• The total cost for $\hat\pi$ is:

$$J(\hat\pi) = T\;\mathbb{E}_{s\sim d_{\pi^*}}[\ell(s,\hat\pi)]$$

where $d_{\pi^*}$ is the expert state distribution, $T$ is trajectory length.

• Supervised imitation achieves cost $J(\hat\pi_{sup}) = T\,\epsilon_N$, where $\epsilon_N$ is average error.
• NEIL, under $N=O(T)$ DAgger-style rounds and defining $e_i =$ probability of "confident but wrong," achieves:

$$J(\hat\pi) \le T\left[\epsilon_N + \frac{2T\,\ell_{max}}{N}\sum_{i=1}^N e_i\right] + O(1)$$

The gap to full supervision is thus controlled by the mean "confident-wrong" probability $e_i$, which can be minimized via high $\mu$ and good seeds.

This theoretical framework shows that annotation complexity is nearly linear with "confident-wrong" rate and can be tuned for practical efficiency.

4. Uncertainty Filtering and Demonstration Handling

The NEIL approach uses the parser's own posterior $p_{\hat\pi}(a_t\mid s_t)$ to estimate uncertainty, with a user-defined threshold $\mu$ dictating when to request human input. Practical deployment enables multi-choice feedback when uncertain, for example, prompting users with targeted column choices or a "None of the above" option. Only steps with high confidence or valid user demonstrations are stored as effective training data; invalid demonstrations (user says "None of the above") are discarded ($w_t=0$). This enables robust annotation efficiency and minimizes label noise, while conferring theoretical soundness by aligning training data with introspected uncertainty.

5. Experimental Protocol for Text-to-SQL Semantic Parsing

The evaluation uses two prominent datasets and models:

• WikiSQL: 56K train, 8K dev, 16K test questions; base model is SQLova.
• Spider: 7.3K train, 1K dev (after cleanup); base model is EditSQL.

Initialization uses fractions of seed annotated data (10%, 5%, 1%), with the remainder comprising "live" user interaction. Models are retrained every 1,000 queries and all experiments are repeated three times with independently shuffled query orders. Baselines include:

• Full Expert: complete gold supervision.
• Binary User: accept/reject feedback only.
• Binary User + Expert: expert annotation only on rejection.
• Self Train: pseudo-gold for self-predictions above $p\ge0.5$.

Performance is measured by logical form exact match accuracy and total annotation count.

6. Empirical Findings and Comparative Analysis

NEIL produces annotation-efficient learning across all settings:

WikiSQL (10% seed)

• Full Expert: 79.4% accuracy, 56,000 annotations
• NEIL: 77.6% accuracy, $\approx$10,000 annotations (under 10% of full expert)
• NEIL* (idealized perfect confidence): 79.2% accuracy, $\approx$3,400 annotations (6% of expert)
• Binary User: 71.2% (negatively impacted by spurious programs)
• Binary User + Expert: $\approx$78.8% at $\approx$50,000 annotations
• Self Train: 53.4%

WikiSQL (5% seed)

• NEIL: 76.1% accuracy, $\approx$6,000 annotations

WikiSQL (1% seed)

• All methods experience $>$20% accuracy drops, indicating poor initialization.

Spider (10% seed)

• Full Expert: 36.7% dev accuracy, 8,000 annotations
• NEIL: 31.5%, $\approx$1,200 annotations
• NEIL*: 36.1% at $\approx$600 annotations
• Self Train: 20.4%

Key observations:

• NEIL attains nearly fully-supervised accuracy (within 2%) using an order of magnitude fewer annotations.
• The remaining gap to supervised is explained by the "confident-wrong" probability $e_i$.
• With perfect introspective calibration (NEIL*), NEIL matches supervised accuracy at only 6% of the label cost.

Summary Table: Annotated Examples and Accuracy (WikiSQL, 10% Init)

Method	Accuracy (%)	# Annotations
Full Expert	79.4	56,000
NEIL	77.6	10,000
NEIL* (ideal conf)	79.2	3,400

A plausible implication is that introspective imitation learning, by rejecting overconfident errors and focusing human labeling effort, provides a scalable paradigm for annotation-constrained semantic parsing applications.

7. Broader Significance and Directions

Introspective imitation learning, exemplified by NEIL, enables efficient, theoretically grounded, and empirically effective semantic parser training directly from user interaction. By combining introspective uncertainty estimation, targeted human correction, and DAgger-style self-improving loops, it sharply reduces annotation needs—paving the way for privacy-preserving user-adaptive systems and rapid domain adaptation in low-data regimes. The empirical results on WikiSQL and Spider datasets establish NEIL as a compelling annotation-efficient alternative to traditional full-supervised regimes (Yao et al., 2020).