Human-Robot Copilot for Data-Efficient Imitation Learning

Abstract: Collecting human demonstrations via teleoperation is a common approach for teaching robots task-specific skills. However, when only a limited number of demonstrations are available, policies are prone to entering out-of-distribution (OOD) states due to compounding errors or environmental stochasticity. Existing interactive imitation learning or human-in-the-loop methods try to address this issue by following the Human-Gated DAgger (HG-DAgger) paradigm, an approach that augments demonstrations through selective human intervention during policy execution. Nevertheless, these approaches struggle to balance dexterity and generality: they either provide fine-grained corrections but are limited to specific kinematic structures, or achieve generality at the cost of precise control. To overcome this limitation, we propose the Human-Robot Copilot framework that can leverage a scaling factor for dexterous teleoperation while maintaining compatibility with a wide range of industrial and research manipulators. Experimental results demonstrate that our framework achieves higher performance with the same number of demonstration trajectories. Moreover, since corrective interventions are required only intermittently, the overall data collection process is more efficient and less time-consuming.

• The paper introduces a human-robot copilot framework that improves data efficiency using bidirectional control and corrective demonstrations.
• It employs a flexible workspace scaling mechanism to switch between rapid coarse movements and precise actions in robotic manipulation.
• Experimental results demonstrate significant gains in task success rates and reduced corrective data collection time in both simulated and real-world settings.

Human-Robot Copilot for Data-Efficient Imitation Learning: A Technical Assessment

Introduction

Teleoperation-based demonstration remains foundational for data-driven robot learning, but the scalability and efficiency of demonstration-centric pipelines are limited by compounding errors, environmental stochasticity, and low data efficiency inherent to out-of-distribution (OOD) behavior correction. Existing human-in-the-loop and HG-DAgger-style frameworks attempt to address these flaws but introduce a trade-off between generality and dexterity, often failing to deliver both precise corrective action and compatibility with heterogeneous manipulators. "Human-Robot Copilot for Data-Efficient Imitation Learning" (2604.03613) presents an integrated, open-source framework that enables cross-embodiment teleoperation, efficient bidirectional control, and targeted policy augmentation, demonstrating strong improvements in data efficiency and policy performance for both simulated and real robotic manipulation tasks.

Copilot System Architecture

The core architecture of the Human-Robot Copilot framework is a bidirectional leader–follower setup employing kinematic workspace overlap rather than strict joint homomorphism. This design enables both rapid large-scale actions and precise fine-grained manipulation through a scaling factor applied to the leader–follower mapping.

Figure 1: Bidirectional control and observation communication. The leader and follower exchange continuous FK/IK and control signals, allowing seamless switching between policy and teleoperator execution.

Importantly, the leader device is constructed from off-the-shelf and 3D-printed hardware (total cost $\approx$ \$1,000), and supports 3-DoF force feedback for translation and lightweight IMU-based rotational mapping. The system’s control logic allows high-frequency alignment and switching between autonomous and human control, with end-effector pose commands translated via IK solvers and all joint/Cartesian synchronization handled through low-latency local communication. During teleoperation, corrective demonstration segments ("clips") are selectively logged for targeted data augmentation, reducing the time and effort necessary for effective policy improvement.

Control Scale Adjustment

A unique aspect of the system is real-time workspace scaling. By modulating the scaling parameter $\alpha$ in the mapping from leader to follower:

Figure 2: Flexible workspace scaling enables rapid movement (large workspace—higher $\alpha$) or precision (small workspace—lower $\alpha$), enhancing dexterity without sacrificing generality.

The scaling factor is operator-adjustable and decoupled from policy training, enabling on-the-fly adaptation to task phase (e.g., gross transport versus insertion). Empirical results confirm that smaller scaling factors yield smoother, lower-RMS alignment trajectories, which are critical for high-precision manipulation tasks.

Human-in-the-Loop Imitation Learning Pipeline

For policy refinement, the framework employs a looped process where a ResNet-18-based Action Chunking Transformer (ACT) policy is initialized from conventional demonstrations and deployed. During deployment, a teleoperator can intervene upon failure, recording only the relevant corrective segments (“clips”) for incremental fine-tuning.

Figure 3: Training and data augmentation: base IL policy is deployed, corrective human interventions during failures yield demonstration clips, which are consolidated for rapid policy finetuning.

This pipeline’s key advantage is its ability to focus human demonstration effort on challenging OOD states, rather than full-trajectory augmentation. Fast retraining on the clipped data (under 10 minutes) enables rapid iteration between deployment and remedial demonstration, drastically improving sample efficiency.

Experimental Validation

The Copilot framework is validated across both controlled simulation (Robomimic: PickPlace(Can), NutAssembly(Square)) and real-world manipulation settings (Cube Sorting and Tower of Hanoi Insertion).

Figure 4: Top—Tower of Hanoi insertion (precision, narrow tolerance); Bottom—Cube sorting (high randomization, OOD discovery). Challenges include tight tolerances, object slip, and highly randomized object placement.

Tasks are designed to expose both OOD state generalization limits and the need for precise action. The system’s utility is evaluated against Keyboard and VR controller interventions.

Data Efficiency and Policy Performance

Across all experiments, the Copilot achieves higher task stage and overall success rates at equivalent demonstration counts compared to baselines, and requires substantially less corrective demonstration time for recovery:

• On Cube Sorting, Copilot achieves 78%/86% stage 1/2 success and 64% total, outperforming Keyboard and VR baselines (see real-world results in paper).
• On Tower of Hanoi Insertion, Copilot reaches 90%/92% for stages and 86% total.
• Data collection time for corrective clips is reduced by factors of 2–4× compared to collecting full new demonstrations.

The performance gap is most pronounced in precision and contact-rich tasks, where bidirectional leader–follower teleoperation enables millimeter-level corrective actions not possible with Keyboard or VR paradigms.

Scaling for Precision

Empirical trajectory analysis on the Tower of Hanoi demonstrates that reduced scaling factors provide lower RMS lateral and forward deviations during critical alignment, yielding more easily imitated and higher-quality correction data.

Figure 5: End-effector trajectories with varying scaling: large scaling ($\alpha = 2.0$) is fast but oscillatory; small scaling ($\alpha = 0.5$) yields smooth, monotonic approach and smaller RMS errors during alignment.

Implications, Limitations, and Directions for Future Work

The Copilot framework’s general and dexterous teleoperation, efficient failure-targeted data augmentation, and rapid fine-tuning pipeline collectively establish a practical paradigm for data-efficient human-in-the-loop imitation learning. Practically, the system enables lower-cost and broader deployment for industrial and research manipulators, as hardware is heterogeneous and affordable.

However, limitations remain: (1) Correction actions should not introduce execution modes inconsistent with the policy’s underlying structure (limiting exploration in multimodal settings); (2) current use of fixed scaling factors is critical for mapping stability, but dynamic or phase-adaptive scaling could further enhance dexterity if workspace drift is addressed. Augmenting ACT with multimodal policy architectures or auxiliary correction-specific policies may further improve correction efficacy for highly diverse tasks.

Conclusion

Human-Robot Copilot represents a significant stride towards scalable, human-in-the-loop robot learning by unifying heterogeneous teleoperation, workspace scaling, and highly efficient, failure-focused data augmentation. Its validation across simulated and real-world settings underscores its practicality and generality for iterative imitation learning. Future extensions to adaptive scaling and multimodal corrections may further improve sample efficiency and robustness in increasingly complex and diverse scenarios.