Interactive Imitation Learning (IIL)

• Interactive Imitation Learning (IIL) combines traditional imitation learning with interactive machine learning, involving sequential decision-making tasks where a teacher provides real-time feedback as an agent executes its tasks.
• IIL addresses issues such as covariate shift and compounding errors by gathering data based on the learner’s evolving policy, utilizing feedback modalities, and involving methods such as DAgger and AggreVaTe for iterative improvements.
• Key applications of IIL include autonomous driving, robotics, and manipulation tasks, showcasing its potential for human-in-the-loop systems and safer sequential decision-making processes.

Interactive Imitation Learning (IIL) is a branch of imitation learning in which a teacher provides intermittent feedback while an agent executes a sequential decision-making task. Unlike offline behavioral cloning, which learns from states visited by an expert, IIL gathers information on states induced by the learner’s evolving policy. The teacher may provide action labels, corrective adjustments, evaluative signals, preferences, interventions, natural-language instructions, desired state transitions, or safety judgments. The resulting interaction addresses covariate shift and compounding errors while introducing new problems involving human workload, safety, feedback quality, uncertainty estimation, multimodal behavior, and online policy adaptation (Celemin et al., 2022).

1. Definition and problem formulation

IIL combines imitation learning with interactive machine learning. The defining elements are sequential decision-making, teacher-provided knowledge during execution, learner-induced data distributions, and iterative updates in which the teacher observes the consequences of previous feedback. A generic trajectory is

$\tau=(s_0,a_0,\ldots,s_T,a_T),$

where $s_t$ and $a_t$ denote states and actions. A policy may be deterministic, $\pi:S\mapsto A$, or stochastic, $\pi(a\mid s)$.

In conventional behavioral cloning, demonstrations are sampled from the expert-induced distribution. If $\pi^h$ is the human policy and $\pi$ the learner policy, the objective can be written as

$$\pi^{l*}=\arg\max_{\pi\in\Pi} \mathbb{E}_{\tau\sim p_{\pi^h}(\tau)} \left[\sum_{t=0}^{T}\log \pi(a_t\mid s_t)\right].$$

At deployment, however, the learner induces a different distribution. A small prediction error can move the system into an unfamiliar state, where subsequent predictions are less reliable and errors compound. IIL instead supplies feedback on states sampled from distributions such as $p_{\pi_i^l}(s)$, where $\pi_i^l$ is the learner policy at iteration $s_t$0.

The teacher’s feedback can be represented by

$s_t$1

where $s_t$2 indicates whether feedback was provided and $s_t$3 contains its content. Unlike a conventional Markov reward, human feedback may depend on the complete history,

$s_t$4

and may be policy-dependent, delayed, stochastic, inconsistent, or nonstationary.

The central distinction is therefore temporal rather than architectural: offline imitation fixes the data before learning, whereas IIL repeatedly executes, receives feedback, updates, and executes again. The teacher may be a human, another agent, a planner, an expert controller, or—in recent work—an LLM-generated executable policy (Werner et al., 30 Apr 2025).

A useful abstract objective is

$s_t$5

where $s_t$6 is a feedback-derived surrogate for an often-unavailable task objective. The learner’s state distribution depends on its own policy, so the resulting optimization is not ordinary i.i.d. supervised learning.

2. Interactive data collection and no-regret reductions

The foundational reduction treats IIL as interactive no-regret online learning. In a finite-horizon control problem with horizon $s_t$7, stage cost $s_t$8, and policy-induced state distributions $s_t$9, the average state distribution is

$a_t$0

and total expected cost is

$a_t$1

The essential difficulty is that $a_t$2 depends on $a_t$3. The policy is simultaneously a predictor and a generator of future inputs. A performance-difference identity expresses the cost difference between policies as an expectation of one-step deviations under an executing-policy state distribution:

$a_t$4

This identity connects interactive state collection with online cost-sensitive learning.

DAgger and interactive aggregation

DAgger repeatedly executes the current learner, queries the expert on learner-induced states, aggregates the resulting labels, and retrains. Its principal contribution is correction of distribution shift. Its basic loss, however, is usually immediate action disagreement:

$a_t$5

DAgger can be interpreted as an online-learning game in which the current policy determines the loss distribution. A standard mixture uses

$a_t$6

where $a_t$7 controls expert participation. The mixture reduces the difference between the state distribution used for data collection and that induced by the learner alone.

Subsequent variants modify the interaction protocol. HG-DAgger replaces stochastic per-step switching with human-controlled gating: the novice acts while the human considers the state safe; the human then takes control, performs recovery under uninterrupted control, and manually returns control to the novice (Kelly et al., 2018). Robot-gated methods such as EnsembleDAgger, ThriftyDAgger, LazyDAgger, and AIM attempt to determine when expert assistance is necessary.

AggreVaTe and cost-sensitive imitation

AggreVaTe—“Aggregate Values to Imitate”—extends DAgger by replacing immediate action agreement with expert cost-to-go. Its action loss is

$a_t$8

rather than

$a_t$9

This distinction matters when errors have different downstream consequences. Steering slightly toward a cliff and immediately driving over it may both be classification errors, while their future costs differ substantially. AggreVaTe collects states under the current learner or a learner–expert mixture, explores an action, and obtains an estimate of the cost-to-go after following the expert.

The online loss is

$\pi:S\mapsto A$0

The method separates online regret from policy-class approximation error. Online regret measures performance relative to the best fixed policy in the class on the sequence of interactive losses; class regret measures the irreducible cost-sensitive approximation error of the policy class.

Under bounded expert cost-to-go, decaying expert mixing, infinite data per iteration, and no-regret updates, AggreVaTe provides a guarantee of the form

$\pi:S\mapsto A$1

The horizon dependence is linear rather than exponential. Cost-to-go estimates can be obtained by rollout, an analytic oracle, or a demonstrator. Full-information cost vectors and partial-information feedback for only the explored action are both supported (Ross et al., 2014).

Agnostic interactive imitation

Agnostic IIL does not assume that the expert belongs to the learner’s policy class. The learner competes with the best policy available in $\pi:S\mapsto A$2. MFTPL-P—Mixed Follow the Perturbed Leader with Poisson perturbations—combines DAgger, FTPL, Poisson perturbations, and repeated calls to a classification oracle. Its perturbations are sampled from an explorative state distribution and paired with uniformly random actions. The resulting policy ensemble induces a mixed policy that improves exploration.

Bootstrap-Dagger replaces explicit sampling from an explorative distribution with bootstrap resampling of the accumulated interactive dataset. Its ensemble is used to generate more informative learner states rather than only to average final predictions. In continuous-control experiments, Bootstrap-Dagger outperformed BC and standard DAgger under realizable and non-realizable expert settings (Li et al., 2023).

3. Feedback modalities and learned representations

IIL is commonly organized by the information content and timing of teacher feedback. Corrective feedback specifies what action or state transition should occur; evaluative feedback indicates whether behavior is acceptable; preferences compare alternatives; interventions transfer control; and natural-language or multimodal feedback communicates task-level intent.

Corrective action feedback

In action-space methods, the teacher directly supplies or modifies an action. DCOACH uses a corrective action target

$\pi:S\mapsto A$3

where $\pi:S\mapsto A$4 is the current policy action, $\pi:S\mapsto A$5 is a directional feedback signal, and $\pi:S\mapsto A$6 is a correction constant. The corrected pair is used for immediate policy updates and replay-based minibatch training.

Action-space feedback is direct and informative, but it can be difficult when the action consists of joint torques, joint velocities, or coupled actuator commands. TIPS addresses this problem by allowing the teacher to specify a desired state change rather than a low-level action (Jauhri et al., 2020).

State-space feedback

TIPS represents human feedback as

$\pi:S\mapsto A$7

where each component indicates whether a selected state dimension should decrease, remain unchanged, or increase. A desired state is constructed as

$\pi:S\mapsto A$8

A learned forward dynamics model,

$\pi:S\mapsto A$9

evaluates candidate actions and selects

$\pi(a\mid s)$0

The selected action is executed immediately and stored as a policy-training target. TIPS therefore separates the teacher’s specification of what should change from the system’s inference of how to produce that change. It was evaluated on CartPole, Reacher, LunarLanderContinuous, Robot Fishing, and Laser Drawing, including continuous and discrete action spaces (Jauhri et al., 2020).

Evaluative feedback and preferences

Evaluative methods provide scalar feedback, acceptability labels, or preferences. TAMER learns a human-evaluation model; COACH treats corrections as advantage-like feedback; preference methods learn a reward or utility model from trajectory comparisons and then optimize it with reinforcement learning.

Evaluative feedback reduces the need for precise action knowledge but has weaker credit assignment. A negative signal does not necessarily identify the action that should have been taken. Preference feedback is comparatively accessible to non-experts, but it requires autonomous candidate generation and can be unsafe or expensive in physical systems.

RLIF treats intervention itself as a reward signal. A learner transition receives reward $\pi(a\mid s)$1 if the expert intervenes at the next step and $\pi(a\mid s)$2 otherwise:

$\pi(a\mid s)$3

Unlike DAgger, RLIF does not require copying a potentially suboptimal takeover action. It learns actions that avoid states leading to intervention. Consequently, under an informative intervention rule, RLIF can exceed the takeover expert’s performance (Luo et al., 2023).

Learned policies, dynamics, values, and risk models

IIL systems may learn:

• Direct policies mapping states or observations to actions.
• State-transition models mapping desired state changes to actions.
• Reward, cost, value, or utility models optimized with planning or RL.
• Uncertainty and risk models used to decide when to request assistance.
• Human models describing feedback delay, policy dependence, reliability, or changing preferences.
• Affordance models representing state-action pairs likely to produce task effects.

The selected representation depends on the action interface, state dimensionality, safety requirements, teacher expertise, and whether the teacher can provide absolute or relative information.

4. Robot-gated intervention, uncertainty, and safety

Human-gated IIL requires continuous monitoring. Robot-gated IIL attempts to reduce this burden by requesting assistance selectively. Its central problem is deciding when uncertainty corresponds to meaningful task risk.

Uncertainty-based gating

EnsembleDAgger estimates epistemic uncertainty from disagreement among policy ensemble members. This can identify underrepresented states, but disagreement may also reflect legitimate multimodality. Conversely, agreement may mean that all models share the same extrapolation error. IBC and IIFL address multimodality using energy-based policies:

$\pi(a\mid s)$4

with

$\pi(a\mid s)$5

Multiple low-energy modes can represent distinct valid actions produced by heterogeneous supervisors. IIFL quantifies disagreement between bootstrapped EBMs using Jeffreys divergence, avoiding direct partition-function estimation. It allocates scarce supervisors across a fleet and optimizes return on human effort (Datta et al., 2023).

Runtime monitoring

Model-Based Runtime Monitoring with IIL learns an encoder, latent dynamics model, policy, and failure classifier. The monitor rolls the policy forward in latent space, samples possible futures, detects out-of-distribution states, and classifies predicted failure states. It uses both OOD detection and intervention-informed failure prediction. The system is trained from deployment transitions and human intervention indicators, allowing the monitor to improve as the human–robot team operates (Liu et al., 2023).

The method distinguishes failure states, normal rollout states, and intervention states. It does not require deliberately collecting catastrophic failures, but it does not provide a formal safety guarantee or autonomous recovery controller.

Precision-aware intervention

DPIIL estimates environmental precision from the demonstrator’s speed–accuracy trade-off. Humans are assumed to move quickly in open regions and slowly in narrow, clearance-limited regions. A probabilistic model estimates speed conditional on state:

$\pi(a\mid s)$6

Perceived precision is modeled using either inverse mean speed,

$\pi(a\mid s)$7

or inverse mean-plus-standard-deviation speed,

$\pi(a\mid s)$8

The intervention risk combines perceived precision and policy-ensemble variance:

$\pi(a\mid s)$9

This causes the same policy uncertainty to trigger intervention more readily in narrow regions than in open space. DPIIL was evaluated on aperture passing, ring threading, shaft reaching, and real UR5e assembly tasks (Oh et al., 2024).

Conformal uncertainty

ConformalDAgger calibrates policy prediction intervals from intermittent expert labels. Intermittent Quantile Tracking updates a quantile only when a label is observed:

$\pi^h$0

where $\pi^h$1 is the probability that feedback is observed. Importance weighting makes the update unbiased with respect to missing feedback under the stated observation model.

The resulting interval is used both to quantify uncertainty and to determine whether the robot should query the expert. This allows the uncertainty mechanism to adapt to deployment-time expert-policy shift, unlike fixed ensembles that may remain confidently biased. ConformalDAgger increased interventions when the expert changed from straight-line wiping to zigzag wiping and learned the shifted behavior more rapidly (Zhao et al., 2024).

Adaptive intervention mechanisms

AIM learns a proxy intervention $\pi^h$2-function rather than using a fixed uncertainty threshold. Its value estimates whether an expert would intervene if the robot took action $\pi^h$3 in state $\pi^h$4. Alignment is measured by action discrepancy:

$\pi^h$5

for continuous actions, or by action inequality for discrete actions. The proxy values increase when robot actions deviate from the expert and decrease as the robot becomes proficient. AIM therefore adapts the intervention criterion over training and reduces expert take-over cost relative to Thrifty-DAgger in simulated expert-in-the-loop experiments (Cai et al., 10 Jun 2025).

5. Applications and representative systems

IIL has been applied to sequential control, autonomous driving, manipulation, human–robot interaction, physics-based character control, multimodal embodied agents, and robot fleets.

Human-supervised autonomous systems

HG-DAgger was designed for human-supervised autonomous driving. Its human-controlled gating avoids stochastic shared control and perceived actuator lag. An ensemble doubt measure is recorded during interventions and used to estimate a permitted set. In the reported vehicle experiment, HG-DAgger had no collisions or road departures in the listed test samples, although the physical evaluation was small and did not establish a formal safety guarantee (Kelly et al., 2018).

Interactive social robotics

Interactive imitation can require prediction of a partner rather than reproduction of an isolated trajectory. A deep generative framework for human–robot interaction learns recurrent histories, shared task dynamics, agent-specific motion latents, and robot-specific joint trajectories. Human–human data provide interaction dynamics, while smaller human–robot datasets provide embodiment-specific mappings. The system was evaluated on hand-shaking, hand-waving, parachute fist-bumping, and rocket fist-bumping (Bütepage et al., 2019).

Multimodal embodied agents

MIA learns from human–human interactions in the Playhouse environment. It combines behavioral cloning with recurrent multimodal representations, cross-modal contrastive learning, and hierarchical action selection. The agent receives visual and language observations, manipulates objects, emits language, maintains memory, and acts over extended interactions. The reported system achieved over 70% success in human-rated online interactions and approximately 75% of the performance achieved by humans in comparable solver positions (Team et al., 2021).

Physics-based character control

A GAN-like approach trains physics-based motor policies using discriminator-derived imitation rewards. Each behavior has a policy and discriminator ensemble. At runtime, discriminator scores determine whether another policy can safely take over, without explicit reference-pose tracking, motion phase, or runtime motion matching. The system supports walking, running, jumping, punching, kicking, recovery, and perturbation handling (Xu et al., 2021).

Interactive fleet learning

IIFL extends IIL to multiple robots and heterogeneous remote supervisors. Robots share an implicit energy-based policy and request assistance through robot-gated allocation. In simulation and physical block pushing, the method preserved multiple supervisor-specific modes while collecting labels on robot-induced edge cases. Its central objective is return on human effort rather than task return alone (Datta et al., 2023).

Dexterous manipulation

Dexterous manipulation introduces high-dimensional control, contact-rich dynamics, multimodal solutions, partial observability, tactile requirements, and expensive data collection. Surveyed approaches include tactile policy correction, fingertip-pressure correction, diffusion policies with DAgger interventions, residual-action correction, mocap-based teleoperation, preference learning, and hybrid imitation–RL methods (Welte et al., 30 May 2025).

Interactive correction is particularly relevant when demonstrations do not cover contact failures, unstable grasps, object variation, or recovery maneuvers. However, the application remains underexplored relative to standard imitation learning and RL.

LLM-based teaching

LLM-iTeach compiles a task description into an executable Python CodePolicy through hierarchical prompting. During training, the CodePolicy proposes an action, compares it with the learned agent’s action using angular similarity, and supplies either evaluative feedback or a corrective action. The method was evaluated in simulated RLBench manipulation tasks and exceeded BC on the reported tasks, achieving performance similar to the CEILing human-teacher baseline under the selected state API and action representation (Werner et al., 30 Apr 2025).

6. Theoretical, practical, and human limitations

IIL does not eliminate the assumptions of learning-based control; it changes where supervision is obtained and how it is used.

Expert quality and policy dependence

DAgger and corrective imitation generally assume that expert actions are near-optimal. If the expert is suboptimal, supervised imitation can reproduce that suboptimality. RLIF relaxes this requirement by using intervention decisions rather than takeover actions, but then assumes that intervention timing correlates with action quality (Luo et al., 2023).

Human feedback may also be policy-dependent. A teacher may judge an action relative to the current policy’s competence, making old feedback potentially inconsistent after the learner improves. Replay is useful for reducing forgetting but can conflict with policy-dependent feedback.

Recoverability and approximation

State-wise guarantees often require recoverability: a mistaken action must have bounded downstream cost. Without recoverability, a small local action error can cause catastrophic consequences. Trajectory-wise first-step mixing and decoupled Hellinger analysis provide guarantees without recoverability in a realizable deterministic setting.

Recent sample-complexity analysis shows that offline BC is optimal in full-trajectory annotation complexity under realizable deterministic experts and log loss, whereas one-sample-per-round DAgger can improve state-wise annotation complexity. Thus, interaction is not universally statistically superior or inferior; its benefit depends on whether expert effort is counted by trajectories or individual state labels (Li et al., 2024).

Human workload and interface constraints

Interactive learning transfers information efficiently but requires time, attention, availability, and sometimes physical control. Human-gated systems impose continuous monitoring; robot-gated systems risk false positives and false negatives. The feedback interface affects cognitive load, label quality, safety, and the information transferred per interaction.

Action-space correction is precise but can be difficult for non-experts. State-space correction is often more intuitive but depends on a sufficiently accurate forward dynamics model. Preference and evaluative feedback reduce action-level expertise requirements but complicate credit assignment and may require unsafe exploration.

Safety

IIL can reduce risk by allowing intervention before failure, but intervention is not a formal safety guarantee. Safety depends on reaction time, takeover authority, model accuracy, threshold calibration, human vigilance, and whether the robot can be stopped or recovered. Runtime monitors may anticipate future failures, but learned latent dynamics can be inaccurate under distribution shift. Risk proxies such as ensemble disagreement, conformal interval size, speed-derived precision, and Jeffreys divergence should not be interpreted as calibrated failure probabilities without additional assumptions.

Computational cost

Interactive learning reduces expert annotation in some settings by trading human effort for computation. Ensembles, energy-based inference, latent rollouts, candidate-action searches, LLM inference, diffusion-policy training, and repeated online updates can be expensive. IIFL, for example, is substantially slower than explicit fleet learning because energy-based policies require sampling and uncertainty estimation.

Evaluation and benchmark limitations

IIL studies use heterogeneous tasks, interfaces, teacher models, intervention budgets, and metrics. Robot metrics include success, reward, collision rate, intervention count, imitation error, recovery rate, and task completion time. Human metrics include interaction time, intervention burden, NASA-TLX workload, frustration, effort, trust, and perceived usability. Standard RL benchmarks do not reproduce the human component of IIL, and physical studies frequently use small numbers of participants, tasks, and configurations.

Expert shift and nonstationarity

A teacher may change strategy, preferences, or standards over time. Standard DAgger assumes a stationary expert, while ConformalDAgger specifically addresses intermittent feedback under expert shift. A reliable system must distinguish learner uncertainty, environmental novelty, multimodal valid behavior, and genuine changes in expert intent.

Interactive Imitation Learning is therefore best understood not as a single algorithm but as a design space spanning feedback modalities, intervention timing, learned representations, uncertainty estimation, policy optimization, interfaces, and human factors. Its central principle is to align supervision with the states and failures generated by the learner itself. The most active directions combine on-policy corrective feedback with multimodal policies, adaptive robot-gated intervention, runtime monitoring, state-space teaching, self-supervised representation learning, human-aware uncertainty calibration, and hybrid imitation–reinforcement learning.