Imitation Bootstrapped RL

• IBRL is a reinforcement learning framework that uses expert demonstrations to bootstrap both exploration and value estimation through dual imitation and RL policies.
• It employs a two-stage architecture, beginning with behavior cloning and advancing to actor-critic updates where imitation and RL proposals are compared via Q-values.
• IBRL has inspired variants that adjust bootstrap mechanisms for exploration and target updates, yielding significant gains in sample efficiency on robotic control tasks.

Imitation Bootstrapped Reinforcement Learning (IBRL) denotes a family of methods in which expert demonstrations are used not merely to define an imitation objective, but to directly accelerate reinforcement learning itself. In its canonical formulation, IBRL first trains an imitation policy on demonstrations and then uses that policy during off-policy RL to propose alternative actions for both online exploration and bootstrapping target values, so that demonstrations continue to shape control long after the initial behavior-cloning stage (Hu et al., 2023). More broadly, the same principle appears in related systems that use expert data to seed replay, alter Bellman targets, shape exploration noise, or initialize reward-learning pipelines, suggesting that IBRL is best understood as a design pattern for using imitation data as a persistent bootstrap signal inside RL rather than as a one-shot supervised pretraining step (Wu et al., 2024, Dong et al., 9 Jun 2025).

1. Definition, scope, and historical precursors

In the narrow sense associated with the method explicitly titled “Imitation Bootstrapped Reinforcement Learning,” IBRL is a demonstration-augmented, off-policy reinforcement learning framework for robotic control that first trains an imitation learning policy on a demonstration dataset $\mathcal D$, then uses the learned imitation policy $\mu_\psi$ inside RL both for online action selection and for critic target construction (Hu et al., 2023). The motivating claim is that demonstrations should not be treated only as static replay or as a regularizer on the RL actor; instead, a strong standalone imitation policy should remain available throughout learning as a source of high-quality candidate actions.

In a broader encyclopedic sense, the term also covers adjacent formulations in which demonstrations bootstrap RL without direct action cloning of the final policy. A particularly early precursor is Price and Boutilier’s “implicit imitation,” which uses mentor-induced state transitions to improve model-based RL through model extraction, augmented Bellman backups, confidence testing, and focused prioritized sweeping rather than through policy cloning (Boutilier et al., 2011). This suggests that the conceptual core of IBRL predates deep RL: the essential idea is to exploit privileged trajectories to reduce unguided exploration and accelerate value propagation.

A useful distinction follows from the supplied literature. Some methods use imitation to bootstrap actions directly, as in reference-policy IBRL (Hu et al., 2023); some bootstrap RL updates inside an imitation or inverse-RL procedure, as in expert replay bootstrapping and expert Q bootstrapping (Wu et al., 2024); some bootstrap exploration rather than the policy objective, as in Data-Guided Noise (Dong et al., 9 Jun 2025); and some bootstrap RL by first solving an easier privileged-observation problem, distilling it, and then fine-tuning in the harder deployment observation space (Xing et al., 2024). This suggests that “IBRL” is both a specific algorithm family and a broader organizing principle.

2. Canonical architecture of reference-policy IBRL

The canonical deep-control formulation has two stages. First, one trains an imitation policy $\mu_\psi$ from demonstrations $\mathcal D$. With standard behavior cloning, the loss is

$$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$

which under an isotropic Gaussian action model reduces to

$$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$

Second, one runs actor-critic RL, but allows both the RL actor $\pi_\theta$ and the fixed imitation policy $\mu_\psi$ to propose actions at each state (Hu et al., 2023).

During online interaction, the learner forms two candidates: an imitation action $a^{\text{IL}} \sim \mu_\psi(s)$ and an RL action $a^{\text{RL}} \sim \pi_\theta(s)$. The executed action is the one with the higher target-Q value: $\mu_\psi$0 This is the actor-proposal mechanism. Its intended effect is to avoid wasting early interaction on very poor RL actions when the imitation policy already supplies a meaningful lower bound on action quality.

IBRL also modifies critic targets. Instead of bootstrapping only through the target actor’s next action, it compares the next-state imitation proposal and the next-state RL proposal: $\mu_\psi$1 This is the bootstrap-proposal mechanism. The critic therefore learns under a hybrid next-action rule that selects whichever of the two proposals appears better under the target critic.

The paper instantiates this design with TD3 and, in state-based settings, with RED-Q-style critic ensembles. The actor update itself remains the usual actor-critic update; the distinctive change is that the imitation policy participates in behavior generation and target construction, not in the actor loss. The replay buffer is initialized with demonstrations, but the method’s main mechanism is not demonstration oversampling. The fixed imitation policy is a persistent action proposer, and the balance between imitation and RL is mediated implicitly by critic comparisons rather than by a manually tuned imitation penalty (Hu et al., 2023).

A soft variant is also defined. To avoid the masking effect of a hard $\mu_\psi$2, IBRL can replace deterministic selection by a Boltzmann distribution over the two candidate actions: $\mu_\psi$3 with $\mu_\psi$4. In the reported experiments, soft IBRL was particularly useful in state-based settings, with $\mu_\psi$5, whereas pixel-based settings used the simpler hard-max rule (Hu et al., 2023).

3. Relation to imitation pretraining, replay seeding, and adaptive IL–RL hybrids

A recurrent misconception is that IBRL is equivalent to behavior-cloning warm start or replay seeding. The original framework is explicit that it differs from both. Replay-buffer initialization or oversampling, exemplified by RLPD-style baselines, helps critic learning by reusing successful demonstration transitions, but it does not exploit the fact that a trained imitation policy can itself output useful actions online. Pretrain-and-regularize methods such as regularized fine-tuning keep the RL actor close to demonstrations by adding an imitation penalty, but this requires balancing the RL objective and the imitation objective and forces IL and RL to share a policy parameterization (Hu et al., 2023). Canonical IBRL instead keeps the imitation policy separate and fixed, and lets a critic arbitrate between RL and IL proposals.

A second misconception is that imitation bootstrapping necessarily means “imitate first, then abandon imitation.” LOKI is a clean example of that staged strategy: it performs a short randomized number of imitation iterations and then switches permanently to policy-gradient RL, with the theoretical claim that the switched policy is close to expert performance and the later RL phase behaves almost as if policy gradient had started from the expert (Cheng et al., 2018). IBRL takes a different position: the imitation policy remains an active source of candidate actions throughout training.

A third misconception is that stronger experts always imply better imitation bootstraps. ADVISOR shows that this can fail when the teacher uses privileged information unavailable to the student. In that setting, standard imitation learns the conditional average

$\mu_\psi$6

which can be substantially worse than both the teacher and the best student policy under the student’s information constraints. ADVISOR therefore replaces fixed IL$\mu_\psi$7RL schedules with state-dependent mixing,

$\mu_\psi$8

where

$\mu_\psi$9

This establishes an important conceptual boundary for IBRL: demonstrations are useful only insofar as they are reproducible or informative under the learner’s observation structure (Weihs et al., 2020).

4. Empirical behavior in robotic control and online fine-tuning

The original IBRL paper evaluates on 6 simulation and 3 real-world tasks. In simulation it uses four Meta-World tasks and two Robomimic tasks, including PickPlaceCan and NutAssemblySquare; in the real world it studies Lift, Drawer, and Hang on Franka robots with sparse $\mu_\psi$0 success rewards (Hu et al., 2023). The strongest gains appear on harder tasks. On the hardest simulation task, the paper reports that IBRL “nearly doubles” the performance over the second-best method. In real-world Hang, it performs $\mu_\psi$1 better than the second-best RL method. On real-world Drawer, at $\mu_\psi$2 interaction steps, IBRL reaches $\mu_\psi$3, whereas baselines are $\mu_\psi$4, which the authors interpret as at least $\mu_\psi$5 greater sample efficiency.

The real-world results also show how the reference-policy mechanism behaves under distribution shift. On Lift, trained with 10 demonstrations and 8K environment steps, the reported final success rates are BC $\mu_\psi$6, RLPD $\mu_\psi$7, RFT $\mu_\psi$8, and IBRL $\mu_\psi$9. In a harder evaluation with the block initialized at the visual boundary, BC drops to $\mathcal D$0, while IBRL remains at $\mathcal D$1. On Drawer, trained with 30 demonstrations and 16K environment steps, the reported final success rates are BC $\mathcal D$2, RLPD $\mathcal D$3, RFT $\mathcal D$4, and IBRL $\mathcal D$5. On Hang, trained with 30 demonstrations and 30K environment steps, BC reaches $\mathcal D$6, RLPD $\mathcal D$7, RFT $\mathcal D$8, and IBRL $\mathcal D$9 (Hu et al., 2023).

The dexterous-hand variant IBORL preserves the same high-level template—behavior-cloning initialization followed by real-world online RL—but adds an adaptive anti-forgetting regularizer to prevent the pretrained policy from collapsing under immature critic estimates. Its BC loss is

$$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$0

and its RL objective combines critic maximization with an imitation penalty weighted by

$$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$1

In real dexterous tasks, pure RL via SERL achieves $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$2 success on all reported tasks, whereas IBORL reaches $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$3 on Cup Taking, $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$4 on Cube Taking, $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$5 on Scanner Taking, and $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$6 on Loopy Taking; the paper also reports an almost $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$7 success rate and a $$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$8 improvement in cycle time overall (Huang et al., 6 Mar 2025). This supports a broader IBRL claim: when exploration is prohibitively hard, imitation bootstrapping can be the difference between learnability and complete failure.

5. Variants, extensions, and adjacent formulations

The literature supplied with the topic shows that the reference-policy formulation is only one member of a broader family. A concise summary is given below.

Formulation	Bootstrap mechanism	Representative paper
Reference-policy IBRL	IL policy proposes actions for acting and target bootstrapping	(Hu et al., 2023)
DRLR	Reference-policy IBRL with calibrated Q-based action selection and SAC backend	(Shen et al., 4 Sep 2025)
Expert bootstrapping in IRL	Expert replay and expert-action Q bootstrapping inside the inner RL loop	(Wu et al., 2024)
Exploration-guided imitation bootstrap	Demonstrations shape state-dependent exploration noise rather than actor loss	(Dong et al., 9 Jun 2025)
Privileged-teacher RL$$L(\psi) = -E_{(s,a)\sim\mathcal D}[\log \mu_\psi(a \mid s)],$$9IL$$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$0RL	RL teacher, IL student distillation, then RL fine-tuning in deployment observation space	(Xing et al., 2024)

DRLR is explicitly developed “based on an algorithm called Imitation Bootstrapped Reinforcement Learning.” It retains the two-policy architecture—reference policy plus RL policy—but modifies the action-selection module to use a calibrated reference Q-value computed on states sampled from the demonstration dataset, and replaces TD3 with SAC. The selector is written as

$$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$1

with $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$2 and $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$3. Empirically, the paper reports that on the harder Open Drawer task, the modified selector reduces BC loss by about $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$4 relative to IBRL and reaches roughly $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$5 higher mean reward during interaction; on open drawer sparse reward, full DRLR improves average reward by around $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$6 (Shen et al., 4 Sep 2025).

A different but closely related line appears in inverse reinforcement learning. “Accelerating Inverse Reinforcement Learning with Expert Bootstrapping” argues that demonstrations should not only define the imitation objective; they should also accelerate the inner RL machinery. It introduces expert replay bootstrapping, which mixes expert transitions into the replay buffer,

$$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$7

and expert Q bootstrapping, which replaces the next-state value on expert transitions by a two-action soft maximum over the expert next action and the actor-sampled next action: $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$8 On benchmark MuJoCo tasks, the reported number of iterations to reach $$L(\psi) = E_{(s,a)\sim\mathcal D}\left\lVert \mu_\psi(s) - a \right\rVert_2^2.$$9 of deterministic expert performance drops from $\pi_\theta$0 on Hopper-v2, $\pi_\theta$1 on HalfCheetah-v2, $\pi_\theta$2 on Walker2d-v2, and $\pi_\theta$3 on Ant-v2 (Wu et al., 2024). Although this paper is about IRL rather than standalone control, it is explicitly described as an IBRL-style contribution.

Data-Guided Noise instantiates a more exploration-centric variant. Instead of cloning or constraining the policy, it learns a state-dependent covariance model $\pi_\theta$4 from demonstrations and samples training actions from

$\pi_\theta$5

The covariance is trained by Gaussian negative log-likelihood on demonstration actions, while the RL actor mean is optimized by standard RL. The method achieves up to $\pi_\theta$6 improvement over prior RL-from-offline-data methods across seven continuous-control tasks and is explicitly positioned as an exploration-based imitation bootstrapping method rather than an imitation-regularized RL method (Dong et al., 9 Jun 2025).

Finally, the RL$\pi_\theta$7IL$\pi_\theta$8RL template for vision-based agile flight offers a different bootstrap channel. A privileged teacher policy is first trained with PPO on low-dimensional simulator state, then distilled into a student visual policy by imitation, and finally fine-tuned again with RL in the visual observation space. In simulation, direct RL from pixels or corners yields $\pi_\theta$9 success on all three race tracks, whereas the distilled-and-fine-tuned method reaches $\mu_\psi$0 success and improves lap times beyond DAgger-only policies (Xing et al., 2024). This suggests that some IBRL problems are best solved not by persistent reference policies, but by moving competence across observation spaces.

6. Limitations, assumptions, and open questions

Several limitations recur across the IBRL literature. The most basic is dependence on demonstration quality. The original IBRL paper reports that when the imitation policy is trained on poorer “Multi-Human” demonstrations, early performance drops because the proposed IL actions are weaker, even though the method can eventually escape the worse BC policy (Hu et al., 2023). IBORL is more explicit: performance is highly dependent on the quality of the pretrained policy, because if BC is too poor, the online RL search space remains too large (Huang et al., 6 Mar 2025).

A second limitation concerns the need for action-labeled demonstrations. This is especially clear for reference-policy IBRL, DRLR, and EQB-style methods, which require an imitation policy or expert next action. The expert-bootstrapping IRL paper notes explicitly that EQB assumes access to expert demonstrations with actions, and that if demonstrations are poor one would expect target estimates to become biased in the wrong direction (Wu et al., 2024). Similar caveats apply to reference-policy methods that compare $\mu_\psi$1 and $\mu_\psi$2.

A third issue is critic reliability. Canonical IBRL, DRLR, and related methods all rely on Q-values to arbitrate between imitation and RL actions. The original IBRL paper partly mitigates this with target critics, double-Q-style minima, and demonstration-prefilled replay, but the selection mechanism still depends on value estimation quality (Hu et al., 2023). DRLR is motivated precisely by a failure mode in which out-of-distribution RL actions receive overestimated Q-values, leading to inefficient exploration and state distribution drift (Shen et al., 4 Sep 2025).

A fourth issue is that some forms of imitation bootstrapping can be actively harmful when the teacher uses privileged information unavailable to the student. ADVISOR formalizes this as the imitation gap and shows that policy-averaging under hidden variables can make naive IL$\mu_\psi$3RL staging worse than either adaptive mixing or, in some cases, no imitation at all (Weihs et al., 2020). This complicates any simplistic identification of IBRL with “stronger expert implies better bootstrap.”

A fifth issue concerns theoretical scope. “Imitation Learning by Reinforcement Learning” shows that, for deterministic experts, imitation can be reduced to a single RL problem with stationary intrinsic reward

$\mu_\psi$4

and proves occupancy and reward recovery guarantees under ergodicity and mixing assumptions. But the paper states explicitly that the deterministic-expert assumption is critical and cannot be relaxed for the single-call stationary-reward reduction (Ciosek, 2021). This indicates that some elegant theoretical justifications for imitation-to-RL reductions do not transfer directly to general stochastic-expert settings.

More generally, the literature suggests an unresolved design question: what exactly should demonstrations bootstrap? The supplied papers provide at least four answers—actions, Bellman targets, exploration noise, and reward models—and no single choice dominates all regimes. This suggests that future work on IBRL will likely remain heterogeneous, with architecture, task hardness, reward sparsity, observation mismatch, and demonstration quality determining which bootstrap channel is most effective.