Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching (2309.03835v3)

Abstract: Learning for Demonstration (LfD) enables robots to acquire new skills by imitating expert demonstrations, allowing users to communicate their instructions in an intuitive manner. Recent progress in LfD often relies on kinesthetic teaching or teleoperation as the medium for users to specify the demonstrations. Kinesthetic teaching requires physical handling of the robot, while teleoperation demands proficiency with additional hardware. This paper introduces an alternative paradigm for LfD called Diagrammatic Teaching. Diagrammatic Teaching aims to teach robots novel skills by prompting the user to sketch out demonstration trajectories on 2D images of the scene, these are then synthesised as a generative model of motion trajectories in 3D task space. Additionally, we present the Ray-tracing Probabilistic Trajectory Learning (RPTL) framework for Diagrammatic Teaching. RPTL extracts time-varying probability densities from the 2D sketches, applies ray-tracing to find corresponding regions in 3D Cartesian space, and fits a probabilistic model of motion trajectories to these regions. New motion trajectories, which mimic those sketched by the user, can then be generated from the probabilistic model. We empirically validate our framework both in simulation and on real robots, which include a fixed-base manipulator and a quadruped-mounted manipulator.

• The paper introduces Diagrammatic Teaching, a novel LfD paradigm that enables intuitive robot instruction using 2D sketches.
• It leverages Ray-tracing Probabilistic Trajectory Learning to convert sketched trajectories into precise 3D motion paths.
• Experimental evaluations in simulations and real robots confirm its effectiveness in replicating complex motions for practical tasks.

Instructing Robots by Sketching: Learning from Demonstration via Probabilistic Diagrammatic Teaching

This paper presents a novel approach to Learning from Demonstration (LfD) named Diagrammatic Teaching. This technique allows users to teach robots new skills by sketching desired demonstration trajectories on 2D images of the scene. The paper introduces a computational framework, Ray-tracing Probabilistic Trajectory Learning (RPTL), that transforms these sketches into a generative model of motion trajectories in 3D space.

Key Contributions:

1. Diagrammatic Teaching Paradigm:
   • The authors propose Diagrammatic Teaching as an alternative paradigm for LfD. This eliminates the need for direct physical interaction with robots, as required in kinesthetic teaching, or proficiency in hardware use for teleoperation.
   • Users provide demonstrations by sketching on static 2D images captured or generated by models like NeRF. The sketched trajectories approximate desired robot motions, allowing intuitive skill transfer from humans to robots.
2. Ray-tracing Probabilistic Trajectory Learning (RPTL):
   • RPTL leverages time-varying probability densities extracted from the 2D sketches and applies ray-tracing methods to map these onto corresponding regions in the 3D task space.
   • The framework fits a probabilistic model based on these 3D regions, allowing for the generation of new motion trajectories mimicking the user-provided sketches.

Technical Implementation:

The RPTL framework consists of several steps:

• Density Estimation: This involves using normalizing flow models to compute probability densities over 2D trajectory sketches.
• Ray-tracing for 3D Fitting: Utilizing ray-tracing techniques, the method identifies regions in 3D space that correspond to high-density sketches from different viewpoints.
• Conditional Trajectory Generation: The trajectories can be generated to start from novel positions, increasing the flexibility of the taught skills.

Experimental Evaluation:

The paper validates RPTL through both simulation and physical robot experiments. Simulation results demonstrate that the model can accurately replicate complex trajectories, such as writing alphabetic characters, from minimal sketch input. Real-world tests involve tasks like drawer closing and object dropping, utilizing different robot platforms, including a fixed-base manipulator and a robotic dog equipped with an arm.

Implications and Future Directions:

The proposed approach shows promising implications for non-expert users to instruct robots intuitively through sketches. It lowers the barrier for robot programming and offers a more accessible interaction paradigm for robot skill acquisition. Future work could extend Diagrammatic Teaching by enhancing scene understanding via actively generated virtual perspectives and integrating additional user constraints to refine robot behavior further.

This research contributes to the LfD domain by introducing an innovative method that simplifies the interface between human instructors and robots. It lays a foundation for more fluid and interactive robot teaching systems, potentially broadening the adoption and adaptation of robots in various industries.