One Demonstration Is Enough for Real-World Robotic Reinforcement Learning

Abstract: Learning effective robot control policies on physical hardware is challenging due to costly data collection and the difficulty of reward specification. Prior work has incorporated demonstrations into reinforcement learning (RL), yet existing approaches either require large numbers of demonstrations or depend on continuous human intervention during training. To address these limitations, we present AutoSERL, a framework that leverages a single demonstration to fully automate the intervention process in real-world robot RL. The framework includes three complementary mechanisms to accomplish certain tasks: a sliding window intervention mechanism that continuously guides exploration to prevent local optima and unsafe deviations, a safety recovery mechanism that detects and corrects failure states via predefined trajectory recovery points, and an intervention termination criterion that automatically disables guidance once the policy can independently complete the task, preserving its exploration advantage. We evaluate AutoSERL on six contact-intensive manipulation tasks across two robot platforms, spanning insertion, hanging, and hinge-based tasks. AutoSERL consistently outperforms SERL initialized with 20 demonstrations, behavior cloning, and MILES -- a dedicated one-shot imitation learning baseline -- across all tasks while matching HIL-SERL, achieves 100% success rate on insertion tasks, and demonstrates improved robustness to positional variations, all from a single demonstration. Code and videos are available on our project website: https://autoserl.github.io/.

• The paper introduces AutoSERL, a framework that uses a single demonstration to achieve sample-efficient robotic reinforcement learning without continuous human supervision.
• It employs sliding window intervention, safety recovery, and adaptive termination to overcome failures and optimize policy performance.
• Extensive experiments on contact-intensive tasks show that AutoSERL outperforms multi-demonstration and human-guided methods, achieving a 100% success rate in insertion tasks.

One Demonstration Is Enough for Real-World Robotic Reinforcement Learning

Introduction

"One Demonstration Is Enough for Real-World Robotic Reinforcement Learning" (2607.01651) addresses the limitations of current real-world robotic reinforcement learning (RL) approaches which have hindered sample efficiency and scalability due to dependencies on multi-demonstration datasets or continuous human supervision during training. The paper introduces AutoSERL, a closed-loop automated intervention system driven by a single demonstration trajectory. This framework aims to realize fully autonomous, sample-efficient RL for contact-intensive manipulation on physical robotic platforms.

AutoSERL Framework

AutoSERL extends the SERL framework by eliminating the need for ongoing human-in-the-loop guidance during RL and achieves full automation via three synergistic components: sliding window intervention, safety recovery, and intervention termination. Each component targets specific failure and inefficiency modes observed in prior human-guidance-based RL paradigms.

• Sliding Window Intervention leverages geometric proximity and direction checks relative to a demonstration trajectory. The robot is only redirected towards demonstration points within a feasible angular threshold, preventing regression to previously visited states and promoting forward progress in complex manipulation spaces.
• Safety Recovery Mechanism systematically detects stagnation events by monitoring the robot’s motion trajectory against the demonstration and autonomously reverts the agent to safe, contact-validated recovery points. A re-execution protocol replays relevant demonstration segments, enabling automatic extraction from diverse local failure modes without explicit engineering for each scenario.
• Intervention Termination adaptively ceases further intervention based on task completion and intervention frequency, thus maximizing exploration benefits when policies become sufficiently proficient.

Figure 1: Schematic of AutoSERL's sequential interventions and policy training cycle integrating automated guidance via geometric heuristics and demonstration-derived policies.

This system design directly addresses observed failure points, including entrapment in local optima, erroneous Q-value estimations, object collision or entanglements, and general stagnation in the state space, without requiring knowledge of failure reward assignments or low-level environment modeling.

Experimental Setup

Experiments were executed on six real-world contact-intensive tasks encompassing plug/USB insertion, hanging (correction tape, hanger, spoon), and hinge-based (drawer) opening scenarios, utilizing two robotic platforms: Franka (insertion tasks) and UR5 with an Inspire dexterous hand (hanging/hinge-based tasks). Observation inputs combine dual RGB video feeds and robot proprioceptive states; the action space is a 6D delta end-effector pose. Sparse reward signals and fixed episode lengths create a sparse reward, high-precision manipulation setting simulating practical deployment constraints.

Figure 2: Robotic hardware: Franka/pincer platform for insertion and UR5/dexterous hand for hanging and hinge-based experiments.

Figure 3: Task suite overview, illustrating representative object–robot interactions across all evaluated manipulation domains.

Results

Sample Efficiency and Task Mastery

AutoSERL achieves immediate and sustained sample-efficient training, consistently outpacing SERL with 20 demonstrations, behavior cloning (BC), and the one-shot imitation baseline MILES across all tasks. Notably, AutoSERL matches or surpasses the performance of HIL-SERL (which uses continuous human teleoperation) with only a single demonstration trajectory and no human presence. In insertion tasks, AutoSERL achieves a 100% success rate (50/50 successful episodes), while alternatives, including BC with up to 20 demonstrations, fail to generalize.

Figure 4: Training curves for intervention occurrence and episodic return for SERL vs. AutoSERL across all tasks, showing rapid reduction in needed interventions and accelerated reward convergence for AutoSERL.

Robustness and Generalization

Robustness is demonstrated both through inter-seed variance experiments (across five random seeds, AutoSERL maintains near or full task completion rates) and positional shift robustness, where randomized initial plug positions do not degrade final policy performance after interpolation and demonstration-augmented guidance.

Failure case visualizations identify typical stagnation and entrapment states rescued by AutoSERL's recovery mechanism.

Figure 5: Characterization of typical stuck/failure states handled via automated intervention and recovery during real-world RL.

Further, heuristic sensitivity analysis shows that the choice of geometric thresholds ($th_1$, $th_2$) is critical; settings too-small or too-large degrade exploration or intervention quality, reflecting a trade-off between maximized exploration and safety constraints.

Ablation Analysis

Ablation studies demonstrate additive contributions of each AutoSERL mechanism. Removing sliding window guidance or recovery rapidly degrades performance, increasing both training steps required for solution discovery and the likelihood of irrecoverable failures. Disabling intervention termination leads to overdependence on demonstration constraints and reduced policy optimization beyond demonstration-level suboptimality.

Policy trajectory comparisons further demonstrate that AutoSERL-trained policies are not merely imitating; shorter and more efficient trajectories than the original demonstration are produced, indicating the realization of RL-driven trajectory-level optimization constrained only during initial exploration.

Implications and Future Directions

AutoSERL affirms that geometric, demonstration-centric automated intervention strategies can deliver both systematic safety assurance and guidance in sparse-reward manipulation, without scalability and labor bottlenecks of human-in-the-loop schemes. This approach provides a practical roadmap for deploying RL methods in operational and industrial robotic settings, especially where multi-task skill generalization and cost control are paramount.

However, reliance on a single demonstration constrains recovery coverage to encountered failure distributions. The authors suggest that integration with trajectory-augmentation (cf. FARL (Li et al., 12 Jan 2026), UniIntervene (Deng et al., 10 Jun 2026)), broader data coverage, and extension to higher DoF action spaces or more diverse object interaction scenarios is a promising direction. Further, refining the intervention automata for open-set multi-modal failure points and long-horizon generalization remains an open challenge.

Conclusion

AutoSERL demonstrates that only a single demonstration trajectory, when rigorously integrated as an automated intervention schema, is sufficient for high-reliability, sample-efficient RL on a suite of challenging real-world robotic manipulation benchmarks. The system rigorously outperforms or matches competitive multi-demonstration and human-supervised methods, establishing a robust, scalable paradigm for safe and efficient real-world robot learning (2607.01651).