Learning from Successful and Failed Demonstrations via Optimization (2107.11918v2)

Abstract: Learning from Demonstration (LfD) is a popular approach that allows humans to teach robots new skills by showing the correct way(s) of performing the desired skill. Human-provided demonstrations, however, are not always optimal and the teacher usually addresses this issue by discarding or replacing sub-optimal (noisy or faulty) demonstrations. We propose a novel LfD representation that learns from both successful and failed demonstrations of a skill. Our approach encodes the two subsets of captured demonstrations (labeled by the teacher) into a statistical skill model, constructs a set of quadratic costs, and finds an optimal reproduction of the skill under novel problem conditions (i.e. constraints). The optimal reproduction balances convergence towards successful examples and divergence from failed examples. We evaluate our approach through several 2D and 3D experiments in real-world using a UR5e manipulator arm and also show that it can reproduce a skill from only failed demonstrations. The benefits of exploiting both failed and successful demonstrations are shown through comparison with two existing LfD approaches. We also compare our approach against an existing skill refinement method and show its capabilities in a multi-coordinate setting.

• The paper’s main contribution is the TLFSD method, which encodes both successful and failed demonstrations to guide optimal robot trajectories.
• It employs quadratic cost functions and dual Gaussian Mixture Models to derive convergent paths from success and divergent paths from failure.
• Experimental results with a UR5e manipulator validate TLFSD’s robustness in obstacle-rich environments and its potential for iterative learning.

Overview of "Learning from Successful and Failed Demonstrations via Optimization"

The paper "Learning from Successful and Failed Demonstrations via Optimization" presents an innovative approach in the field of Learning from Demonstration (LfD), focusing on the potential of utilizing both successful and failed task demonstrations. Traditionally, LfD systems rely heavily on optimal demonstrations to train robots autonomously. However, sub-optimal human input, labeled as failed demonstrations, often provides invaluable insight into what actions or paths could lead to task failure. These demonstrations, if encoded effectively, can complement successful demonstrations, enhancing a robot's learning process by preventing known errors.

Methodological Contributions

The authors introduce an LfD approach named Trajectory Learning from Failed and Successful Demonstrations (TLFSD), which is distinct in its capacity to encode and process both classes of demonstrations. The core of the approach involves encoding demonstrations in quadratic cost functions and optimizing these under various task conditions. The method constructs separate Gaussian Mixture Models (GMMs) from the successful and failed demonstration sets, allowing the robot to derive both convergent paths towards success and divergent paths from failure. The optimization problem is then defined to minimize the quadratic cost associated with deviating from successful demonstrations while maximizing the divergence from failed ones.

The significance of TLFSD lies in its flexibility and efficiency, functioning with various configurations of available demonstrations. Notably, it can reproduce skills with either subset—successful or failed—entirely absent, a capability not typically found in other approaches.

Experimental Results

The authors validate their approach through a series of experiments in 2D and 3D spaces, using a UR5e robotic manipulator. The experiments range from basic trajectory learning to complex tasks involving obstacles unknown to the learning system. Results demonstrate TLFSD's ability to generate trajectories that avoid paths leading to failure—such as collisions with obstacles—without explicit obstacle detection. In comparison experiments with GMM/GMR-wEM and conventional LfD approaches, TLFSD shows enhanced immediate learning outcomes, especially when failed demonstrations significantly bound the task space.

Further, the iterative learning potential of TLFSD is highlighted, where the system iteratively improves its task reproductions by only learning from failed demonstrations, correcting each unsuccessful attempt until reaching success.

Implications and Future Directions

Practically, TLFSD applies to real-world scenarios where task conditions can vary unpredictably due to changes or uncertainties not captured at training time. By leveraging failed demonstrations, systems can generalize better across diverse conditions without exhaustive iterations or trial-and-error learning typical in combined LfD and Reinforcement Learning frameworks. This efficiency is crucial in human-robot interactions, where operational environments and tasks exhibit high variability.

Theoretically, the approach offers new perspectives in model-based reinforcement learning by suggesting that optimization-based learning from demonstration can bypass the need for defining explicit reward functions—a significant bottleneck of common RL techniques.

For future research, expanding TLFSD to incorporate real-time adjustments and refinements would be beneficial. Moreover, integrating advanced trajectory representation theories, such as those encoding multiple coordinate systems, could enhance the detail with which tasks are learned and executed, unlocking nuanced skill acquisition. Lastly, exploring how TLFSD can integrate seamlessly with perception-based learning systems could provide comprehensive frameworks for autonomous task-solving robots in unstructured environments.

Overall, this method marks a significant evolution in LfD paradigms, showcasing how failed attempts might unlock even more efficient and robust robot learning solutions.