When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

Abstract: Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/

• The paper introduces Q2RL, which analytically estimates a behavior-cloning policy’s Q-function from action likelihoods, entropy, and a small number of online rollouts without requiring the original demonstrations.
• Q2RL gates between behavior-cloning and reinforcement-learning actions using their respective Q-values, achieving up to 0.91 simulation success and 1.00 on-robot success while reducing unsafe exploration.
• The method improves adaptation to distribution shifts and difficult manipulation tasks, but currently depends on policies exposing likelihoods and entropy and on an approximate soft-optimality assumption.

Motivation and problem setting

Behavior cloning (BC) policies are effective when demonstrations cover the deployment distribution, but they lack a mechanism for self-guided online improvement. When training data is insufficient or the environment shifts—lighting changes, visual distractors, modified hardware or controllers—BC performance collapses under compounding errors from covariate shift. Existing offline-to-online methods address this incompletely: offline RL approaches such as CQL require large datasets of positive and negative samples and introduce pessimistic value bias; offline-to-online methods such as CalQL and WSRL can unlearn good BC actions during the transition; residual-policy methods require task-specific tuning of residual scales; and IBRL, the closest prior work, selects between BC and RL actions using a randomly initialized critic, which misprices BC actions early in training and depends on seeding the replay buffer with demonstration data.

The paper's setting assumes a pretrained BC policy $\pi_{BC}$ trained on successful demonstrations for a task class $\mathfrak{M}$ sharing a success condition, a test environment available for online interaction, and a sparse reward. The goal is to improve beyond $\pi_{BC}$ on $\mathcal{M}_{\text{test}}$ while preserving its competent behavior.

Method: Q-Estimation and Q-Gating

Q-Estimation derives an analytic estimate of the BC policy's Q-function from a soft-optimality assumption: the BC action distribution is modeled as Boltzmann-rational with respect to $Q_{BC}$,

$$\pi_{BC}(a \mid s) \approx \frac{\exp(Q_{BC}(s,a)/\alpha)}{\int \exp(Q_{BC}(s,a')/\alpha)\, da'}.$$

Under this assumption, with finite partition function $Z(s)$, the paper proves that

$$Q_{BC}(s,a) = V_{BC}(s) + \alpha \log \pi_{BC}(a \mid s) + \alpha\, \mathcal{H}[\pi_{BC}(\cdot \mid s)].$$

The value function $V_{BC}$ is estimated from Monte Carlo returns over $N$ initial online rollouts of the BC policy; the log-likelihood and entropy terms come directly from the policy head. The method is agnostic to the BC policy class provided it exposes action likelihoods and entropy—the paper instantiates closed forms for Gaussian heads and uses an entropy upper bound for Gaussian mixture models. This is a notable practical property: no access to the original training data is required, only a few environment rollouts. The assumption of soft optimality is a genuine restriction; the authors test it by injecting Gaussian and uniform noise into deterministic BC policies and observe an initial performance drop followed by recovery during online RL, suggesting some robustness but not equivalence to the exact case.

Q-Gating then guides online RL. Two Q-functions are initialized from $\mathfrak{M}$0: one frozen copy serving as a stable reference for BC actions, and a learnable $\mathfrak{M}$1 used as the SAC critic. At each state, both policies propose actions; the action with the higher respective Q-value is executed:

$\mathfrak{M}$2

Because $\mathfrak{M}$3 scores actions under "take this action, then follow BC," high-value BC proposals are preserved, while $\mathfrak{M}$4 improves on data from both sources, allowing the RL policy to surpass BC where exploration finds better behavior. An auxiliary BC loss regularizes the RL actor toward the BC state-action distribution; the authors note this regularization can occasionally prevent perfect success rates but is retained because it produces safe, smooth behaviors on hardware. The full system runs SAC with asynchronous actor/learner processes at 10 Hz control.

Simulation results

Across D4RL (Kitchen-Complete, Adroit-Pen, Adroit-Door) and robomimic (Lift, Can, Square; state and image variants), Q2RL outperforms CQL, CalQL, WSRL, RLPD, and IBRL in success rate and time to convergence in most settings. Representative results:

Task	BC	Best baseline	Q2RL
Kitchen (300k steps)	0.69	0.64 (WSRL)	0.91 ± 0.01
Can-State (no data, 550k)	0.60	0.0 (all)	0.82 ± 0.10
Square-State (no data, 550k)	0.58	0.0 (all)	0.76 ± 0.06
Can-Image (no data, 550k)	0.45	0.0 (IBRL)	0.63 ± 0.05

Two findings stand out. First, without access to BC training data, every baseline fails entirely on the harder tasks, whereas Q2RL continues to improve—attributed to Q-Estimation calibrating Q-values from BC rollouts alone, so the randomly initialized RL critic cannot assign spuriously high values to arbitrary actions. Second, ablations show Q-Gating is critical for high performance, while initializing $\mathfrak{M}$5 from $\mathfrak{M}$6 yields comparable final success to random initialization but a safer starting distribution closer to BC behavior—an important distinction for hardware deployment. Additional ablations show competitive performance with as few as 25 Q-Estimation rollouts and consistent improvement across BC checkpoints with initial success rates from 10% to 75%.

One caveat: on Adroit-Door, WSRL converges to a higher success rate than Q2RL, but qualitative rollouts show the WSRL policy exploits the simulator with actions infeasible on real hardware. This is a fair observation but also means Q2RL's advantage there rests on a qualitative realism judgment rather than the benchmark metric.

Real-world results

On-robot experiments use a Franka arm with wrist and workspace cameras, Cartesian impedance control, and GMM-RNN BC policies trained on 50–100 teleoperated demos. Three tasks of increasing difficulty were evaluated over 20 trials each, reporting best checkpoints within roughly 1–2 hours of interaction (at most ~80k environment steps):

Task	BC	IBRL	Q2RL
Peg Insertion	0.70	0.95 (with data); 0.0 (without)	1.00
Pipe Assembly	0.20	0.0	0.75
Kitting-Modified	0.35	0.0	0.70

These are strong claims: up to 3.75× improvement over the original BC policy (Pipe Assembly), 100% success on Peg Insertion, and adaptation to a distribution shift the BC policy never saw (Kitting-Modified, where bins contain two parts in new locations versus one centered part during training). IBRL achieves zero success whenever demonstrations are not seeded into the replay buffer, and zero even with seeding on the long-horizon, large-action-space tasks. Without replay seeding, CalQL reaches only 0.10–0.20 on Peg Insertion and produced four safety violations from high-jerk actions during evaluation—evidence that offline RL cannot learn usable Q-functions from small datasets of successful demonstrations alone.

Safety results favor Q2RL concretely: during IBRL training on Peg Insertion, two safety violations occurred (excessive force against the board causing robot faults) despite fixed controller gains, whereas Q2RL caused none. Qualitative analysis of annotated rollouts shows meaningful division of labor: BC actions handle grasping, alignment, and inter-bin transit; RL actions take over for contact-rich insertion and for task segments shifted from the training distribution. The authors candidly note not all rollouts exhibit this switching pattern exactly as described.

Limitations and open questions

The principal limitation, acknowledged by the authors, is that Q-Estimation requires action likelihoods and entropy from the BC policy. Diffusion and flow-matching policies—increasingly dominant in robot learning—do not expose these quantities by default, and extending the method to them is left open. The soft-optimality/Boltzmann assumption on human demonstration distributions is an approximation whose failure mode (initial performance drop) was characterized only empirically. The temperature $\mathfrak{M}$7 is fixed at 1 without sensitivity analysis. Real-world evaluation relies on human-provided sparse rewards, termination signals, and resets, so full autonomy of the learning loop remains untested. Finally, the Kitting experiment seeds the replay buffer with 30 episodes from the modified setting, so the distribution-shift result is not achieved strictly zero-shot.

Conclusion

Q2RL combines an analytic extraction of a Q-function from a black-box BC policy with a dual-critic gating mechanism for online RL. The approach requires neither the BC training data nor reward engineering or human interventions during learning, and it demonstrates state-of-the-art offline-to-online performance on standard benchmarks together with practical on-robot learning of contact-rich manipulation in 1–2 hours, including recovery from task distribution shift. Its scope is bounded by the likelihood-and-entropy requirement on the policy class and by the soft-optimality modeling assumption, both of which define clear open problems for extending the method to modern generative policy architectures.