Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning

Abstract: Learning from human demonstrations is a reliable way to teach robots new tasks, but the gains from each additional demonstration shrink as the policy improves. Continued improvement can instead come from supervised deployment, where an operator places the objects and intervenes when the policy fails. We ask how to maximize improvement from a fixed budget of supervised episodes on high-precision manipulation tasks with wide ranges of object placements. We observe that failures can concentrate in a small subset of initial states, so uniform collection spends much of the operator's time on states the policy already handles. Mulligan makes the initial-state distribution a decision, starting each round's episodes at observed failures and untried states. To further improve data efficiency, we augment interactive imitation learning with a value function trained on all data, including failures that imitation discards. Across three real-world tasks evaluated on 2,550 held-out, blinded episodes and two simulated tasks, Mulligan outperforms uniform initial-state sampling at matched collection budgets, and combined with value-based action selection, HiL-IDQL+Mulligan, improves final real-task success by 10-34 percentage points. With operator interventions, the human-robot team completes 98% of collection episodes, remaining productive while the policy learns. Videos, code, and data are available at https://mulligan.page/.

• The paper proposes Oursink performance-guided data collection strategy that leverages low-discrepancy Sobol initialization and counterfactual demos. The new approach improves policy learning by targeting initial-state space failures then filling gaps in coverage.
• This approach re-introduces observed failures, maximizing robust state coverage and retains task proficiency during sampling rounds; increasing task completion and resilience.
• Research demonstrates success in three real-world tasks and two simulated ones with significant improvements such as Thread Nut success rate +10 points and Insert Marker success rate +34 points.

The paper studies how to allocate a fixed budget of supervised robot episodes when the policy is already competent on much of its deployment distribution. Its central claim is that data collection should be treated as an adaptive initial-state design problem rather than as passive sampling from a fixed distribution. The proposed strategy, named oursink in the paper and associated with the title “Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning” (2610.05882), promotes previously observed failures and uses space-filling exploration to discover new failure regions. The strategy is evaluated both with interactive imitation learning and with a batch-online value-learning system that exploits autonomous failures without including them in the actor’s behavior-cloning dataset.

Problem formulation and central observation

The setting is supervised deployment for manipulation. An operator places objects at the start of each episode, allows the learned policy to act, and intervenes when necessary. Data are collected in rounds: the policy is trained, deployed for evaluation and supervised collection, and then retrained between rounds. The actor remains fixed during each round, making the episode outcomes interpretable as measurements of one policy. Crucially, the operator can choose the initial object configuration, so the initial-state distribution is a controllable experimental variable rather than an immutable property of the environment.

The paper’s motivating empirical observation is that failures are highly nonuniform over the initial-state space. In the Square-Narrow simulation, a policy with approximately $86\%$ overall success has 94.8% of its failures concentrated in the hardest 25% of initial states. Uniform collection therefore spends most episodes on states that the policy already handles reliably. The implication is direct: when the objective is improvement per supervised episode, the useful sampling distribution should be concentrated around failure regions while retaining sufficient coverage to detect failures that have not yet been observed.

Figure 2: Success is highly nonuniform across initial states; the hardest quarter of the states accounts for 94.8% of failures.

This claim differs from standard DAgger-style collection, in which the expert corrects trajectories generated from a fixed or uniformly randomized initial-state distribution. DAgger supplies recovery data under the policy’s state distribution, but it does not decide which initial configurations should be visited next. Oursink exploits the additional control available in supervised deployment: the operator can reset directly to an informative state and can repeat a failure after the policy has been updated.

Oursink: failure promotion with coverage preservation

Oursink operates at round boundaries. The initial state set for the next collection round is divided into two components.

Failure promotion selects initial states at which the preceding policy failed. These states are replayed verbatim, rather than being resampled approximately. In the real-world experiments, failures are ordered by the earliest stage at which the task broke down, so failures at grasp or transport are prioritized over failures that occur only during final insertion. A promotion cap reserves part of the collection budget for exploration.

Coverage fill uses farthest-point selection over the normalized initial-state space. After promoted failures have been selected, each remaining state is chosen to maximize its nearest-neighbor distance from all previously collected states and the promoted states. The initial demonstrations use scrambled Sobol sequences, which provide low-discrepancy coverage at small sample sizes. Later rounds use the accumulated design history to fill gaps left by prior collection.

The algorithm can optionally apply a hardness weighting to the coverage fill. However, the method imposes coverage guardrails: hardness weighting cannot produce a substantially worse cumulative nearest-neighbor coverage radius than a fresh Sobol block, and previously used collection and evaluation states are deduplicated. Thus, the method does not simply perform unrestricted hard-example mining. It combines exploitation of observed failures with continued exploration of unvisited regions.

Figure 4: Oursink alternates between replaying observed failures and selecting unvisited states that preserve coverage of the initial-state space.

The approach is especially appropriate for tasks with parameterized resets. Insert Marker and Thread Nut vary object and target placement over regions spanning tens of centimeters and the full object yaw range; Route Cable varies the free cable endpoint and the configurations of two clips. These distributions are substantially wider than the narrow reset ranges commonly used in real-world RL experiments. The method therefore depends on a capability that is available in supervised deployment but not necessarily in autonomous deployment: controlled placement of the task objects at arbitrary parameterized initial states.

Counterfactual demonstrations

A second component addresses a weakness of ordinary intervention data. A correction begins from the state reached by the policy, so it often teaches recovery from an error rather than the behavior that would have avoided the error. If a policy drops an object during transport, the correction may demonstrate how to reacquire it, but it does not demonstrate the correct transport trajectory from the original initial state.

Oursink therefore permits a counterfactual demonstration. After a policy failure and operator rescue, the operator resets the scene to the original initial state and teleoperates the complete task. The counterfactual trajectory is counted against the same episode budget as an ordinary supervised episode. It supplies the actor with a successful trajectory from the difficult initial configuration, placing the missing behavior directly in the support of the behavior-cloning distribution.

The counterfactual ablation shows that the effect is strongest in early rounds. On Insert Marker, round-1 success is 38% with counterfactual demonstrations versus 14% without them. On Thread Nut, the corresponding values are 34% versus 28%. At later rounds the difference diminishes as ordinary corrections accumulate. This result implies that counterfactual demonstrations are most valuable when the policy has not yet acquired the behavior needed to reach the relevant recovery state; they are less necessary once the policy can generate useful corrections from those states itself.

Figure 6: Counterfactual demonstrations produce the largest improvement during early collection rounds, particularly on Insert Marker.

The paper also reports that counterfactual demonstrations increase the number of human-controlled training frames at matched episode budgets. Relative to uniform collection, the oursink campaign contains 19% more human frames on Insert Marker, 28% more on Thread Nut, and 8% more on Route Cable. This is not merely a change in the distribution of states: the operator uses part of the episode budget for complete demonstrations rather than corrections, increasing the quantity of actor-relevant supervision.

Actor-critic architecture

The paper evaluates oursink with two learners. HG-DAgger trains a diffusion-policy actor by behavior cloning on human-controlled frames from successful episodes. HiL-IDQL retains this actor and adds a critic trained on all collected transitions, including autonomous successes, autonomous failures, and policy-controlled portions of intervened episodes.

The actor and critic are deliberately decoupled. The actor is trained only on $\mathcal{D}_h$, the human-controlled frames from successful demonstrations, corrections, and counterfactual demonstrations. The critic is trained on $\mathcal{D}_{\mathrm{all}}$, which contains every valid transition regardless of source or outcome. At deployment, the actor samples $N$ action chunks and the critic selects the highest-valued candidate. This best-of-$N$ operation improves action selection without backpropagating through the diffusion sampling process or modifying the actor’s policy class.

The critic uses action-chunked IQL with several adaptations. Targets bootstrap over the executed chunk, and a $\lambda$-return curriculum accelerates reward propagation over long manipulation horizons. The value function is distributional, using a categorical return representation and entropy-adaptive expectile selection. The critic also uses state-conditioned affine scaling of action inputs, intended to distinguish large transport actions from the much smaller corrections required during insertion and seating.

The actor-data ablation provides an important negative result. Adding successful autonomous segments to the actor’s human data reduces Square-Narrow success by 3.4 percentage points, with a 95% Welch interval of $[-6.5,-0.2]$. The paper therefore does not treat all successful trajectories as interchangeable. Autonomous successes may have different state-action distributions or lower-quality control than operator demonstrations, and indiscriminately mixing them into the actor objective can degrade performance. Autonomous data remain useful to the critic, where failures provide negative evidence for action selection.

The value-based component helps primarily after sufficient on-policy data have accumulated. Early in training, reranking provides little benefit because the critic has inadequate coverage of the relevant action-outcome relationships. Later gains concentrate on precision bottlenecks rather than on initial grasping. Stage-wise analyses show that performance-guided collection improves grasp and transport earlier, while critic reranking contributes more strongly to conditional insertion success.

Figure 8: Supervised deployment forms a trajectory funnel in which human corrections, failure data, counterfactual demonstrations, and critic-based action selection address different failure modes.

Experimental design

The real-world experiments use a Franka Panda on the DROID platform and three tasks:

• Insert Marker: grasp a pen and insert it into a holder with approximately 1 mm clearance.
• Thread Nut: grasp a square nut and seat it on a peg.
• Route Cable: seat a cable into two clips in sequence.

The simulation suite contains Square-Narrow and Square-Broad, which differ primarily in the width and dimensionality of the initial-state distribution. Square-Broad randomizes both nut and peg placement and is substantially more difficult.

The evaluation protocol is designed to reduce confounding from hardware drift and operator behavior. Compared policies are interleaved under anonymous labels on shared held-out Sobol initial states. Real-world evaluation uses 50 held-out states per round, with a separate state block from collection. Comparisons are paired by initial state and use McNemar tests and paired bootstrap intervals. The paper reports 2,550 blinded real-world evaluation episodes across the headline methods.

A qualification is that real-world curves generally represent one training run per method. The reported Wilson intervals quantify episode-level evaluation variability, not variability over independent training runs. The simulation experiments use five seeds and therefore provide stronger evidence about training variability than the robot experiments.

Main performance results

Oursink improves over uniform initial-state sampling across all three real-world tasks. When combined with HiL-IDQL, the final full-task success rate exceeds uniform HG-DAgger by:

Task	Final improvement of HiL-IDQL+oursink over HG-DAgger
Insert Marker	+34 percentage points
Thread Nut	+10 percentage points
Route Cable	+16 percentage points

The corresponding final success counts are 38/50 versus 21/50 on Insert Marker, 36/50 versus 31/50 on Thread Nut, and 17/50 versus 9/50 on Route Cable. The Insert Marker comparison is statistically significant with $p = 0.0015$. The Thread Nut comparison is not individually significant at the final checkpoint, with $p = 0.359$, and the Route Cable comparison has $p = 0.0768$. Thus, the headline range of 10–34 percentage points should not be interpreted as uniformly decisive final-round evidence for every task.

The strongest statistical support concerns the pooled effect of performance-guided collection itself. Across matched rounds, HG-DAgger+oursink improves over uniform HG-DAgger by 11.7 percentage points on Insert Marker, 10.9 points on Thread Nut, and 6.0 points on Route Cable. The paired McNemar $\mathcal{D}_h$0-values are 0.0019, 0.0007, and 0.0356, respectively. These pooled comparisons isolate collection strategy more effectively than single final-round contrasts because they aggregate multiple contemporaneous policy comparisons.

Figure 9: Held-out success improves more rapidly under performance-guided collection than under uniform initial-state sampling, with additional later gains from value-based reranking.

The simulation results are more consistent. On Square-Narrow, HG-DAgger+oursink exceeds uniform HG-DAgger by 8.4, 8.5, 3.6, and 1.8 percentage points across rounds 0–3. On Square-Broad, the corresponding differences are 3.3, 8.5, 11.1, and 7.6 points. HiL-IDQL+oursink improves over HiL-IDQL by similar margins. The largest performance-guided advantage occurs on Square-Broad, where the initial-state distribution is wider and the policy has greater difficulty covering it.

The paper compares against autonomous self-improvement methods in simulation. Without human supervision, only the best-of-$\mathcal{D}_h$1 value-reranked approaches improve reliably. Human-in-the-loop methods improve substantially faster, particularly on the broad distribution. On Square-Narrow, HiL-SERL requires more than ten times the frames needed by HiL-IDQL+oursink to exceed 90% success, while RLPD does not reach that threshold. On Square-Broad, the HiL-SERL variant remains below 10% success after 500,000 steps, whereas HiL-IDQL+oursink reaches 86% with 68,000 collected frames.

These comparisons support the paper’s narrower claim that controlled initial-state selection and supervised corrections are effective for wide initial-state distributions. They do not establish that oursink dominates all online RL methods under their preferred reset distributions or interaction budgets.

Productivity during collection

A central practical result is that performance-guided collection does not require sacrificing task completion during supervised deployment. Across 1,009 fresh real-world collection episodes, the human-robot team completes 98% of episodes, with no round below 90%. The operator can rescue failures while the policy is learning, so the collection process remains operationally productive even when autonomous held-out success is substantially lower.

Figure 11: Operator intervention preserves approximately 98% collection success across the three real-world tasks while autonomous policy performance improves.

The raw fraction of human-controlled frames generally declines as the policy improves. However, the failure-adjusted burden can be higher for oursink because the method intentionally directs episodes toward difficult states. This distinction is important: lower raw intervention rates do not necessarily imply that the sampler is exposing easier states, and higher intervention rates relative to the policy’s failure rate may be the expected consequence of targeted supervision.

Throughput results also favor HiL-IDQL+oursink. The method completes more tasks per robot minute than HG-DAgger on all three real-world tasks when failed-episode time is included. Since execution times for successful trajectories change little across rounds, the throughput improvement is primarily attributable to increased reliability rather than faster motion. This is a relevant operational result because a policy that improves success while becoming substantially slower may not improve deployment productivity.

Figure 12: Completed tasks per robot minute increases with performance-guided collection and value-based action selection, including time spent on failed episodes.

Interpretation of the ablations

The ablations separate several mechanisms that are otherwise confounded in the full method.

First, Sobol coverage alone improves the round-0 policy relative to i.i.d. uniform sampling. On Square-Narrow, Sobol initialization increases success from 70.80% to 79.03%, an 8.23-point gain. On Square-Broad, it increases success from 49.03% to 52.56%, a 3.53-point gain. Thus, part of the benefit attributed to oursink arises from better finite-sample coverage rather than failure promotion alone.

Second, performance-guided collection provides an additional improvement beyond Sobol coverage. On Square-Narrow round 1, the sequence of interventions yields 85.73% success with uniform sampling, 89.59% with Sobol coverage, 92.69% with performance-guided collection without counterfactual demonstrations, and 93.52% with both performance guidance and counterfactual demonstrations. The increments are not additive in a strict causal sense, but they indicate that low-discrepancy initialization, failure promotion, and counterfactual supervision each contribute.

Third, counterfactual demonstrations are not uniformly necessary. Their largest effect appears in early rounds and on tasks where corrections commonly begin too late to demonstrate the desired behavior. As the policy improves, ordinary corrections become more informative and the no-counterfactual arm catches up.

Fourth, critic reranking and state selection operate at different task stages. Oursink changes which initial configurations receive supervision, improving upstream grasp and transport competence. The critic uses failure-inclusive data to select among locally plausible action chunks, improving downstream precision. This division explains why combining the two mechanisms performs better than either alone.

Limitations and open questions

The method assumes that initial states can be parameterized and physically instantiated by the operator. This assumption is reasonable for fixture-based supervised deployment but does not hold automatically in environments where objects arrive through uncontrolled dynamics. In such settings, the relevant problem would be selecting among naturally occurring initial states or actively driving the robot to informative configurations.

The empirical scope is also narrower than the generality of the framing. Oursink is evaluated with one diffusion-policy actor, one IQL-derived critic formulation, and one batch-online human-in-the-loop MDP. Its performance with generalist policies, VLA policies, alternative offline RL objectives, or other interactive imitation learners remains unresolved. The paper explicitly presents the sampler as learner-agnostic, but this is a design claim rather than a demonstrated result across a broad set of learner classes.

The critic results require particularly careful interpretation. Although value-based reranking improves later-round performance in the main real-world experiments, offline diagnostics are unreliable. A critic can discriminate successful from failed trajectories without ranking alternative actions correctly. In one Insert Marker example, training-view AUROC increased to 1.000 while excluded-block AUROC declined, indicating overfitting. FQE correlated with measured success across a retrospective set of policies at Spearman $\mathcal{D}_h$2, but it failed to resolve close actor-versus-reranked comparisons. The deployed critic gains therefore remain empirically established but difficult to select or validate offline.

Finally, the actor does not absorb the behavior induced by critic reranking. Adding autonomous successes to actor training degraded performance in the reported ablation, so the gains remain external to the actor and depend on repeated best-of-$\mathcal{D}_h$3 selection at inference. Whether reranked behavior can be distilled into the actor without losing the benefits of human-data curation is left open. The paper also notes that some real-world critic modifications, including state-conditioned action scaling, were not isolated by fully controlled ablations, limiting attribution of the final critic improvements.

Conclusion

The paper argues that supervised robot learning should optimize not only the learner and intervention policy, but also the distribution of initial states presented to the operator. Oursink operationalizes this principle by replaying observed failures, preserving low-discrepancy coverage of unvisited states, and optionally replacing inadequate corrections with counterfactual demonstrations. Across three real-world and two simulated manipulation tasks, it improves sample efficiency at matched collection budgets, with pooled real-world gains of 6.0–11.7 percentage points for collection strategy alone and final full-task gains of 10–34 points when combined with value-based action selection. The evidence supports performance-guided collection as a practical mechanism for wide initial-state distributions, while the dependence on controllable resets, limited learner diversity, and unresolved critic validation remain substantive boundaries on the paper’s conclusions.