Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes (1704.01189v2)

Abstract: We present the Semantic Robot Programming (SRP) paradigm as a convergence of robot programming by demonstration and semantic mapping. In SRP, a user can directly program a robot manipulator by demonstrating a snapshot of their intended goal scene in workspace. The robot then parses this goal as a scene graph comprised of object poses and inter-object relations, assuming known object geometries. Task and motion planning is then used to realize the user's goal from an arbitrary initial scene configuration. Even when faced with different initial scene configurations, SRP enables the robot to seamlessly adapt to reach the user's demonstrated goal. For scene perception, we propose the Discriminatively-Informed Generative Estimation of Scenes and Transforms (DIGEST) method to infer the initial and goal states of the world from RGBD images. The efficacy of SRP with DIGEST perception is demonstrated for the task of tray-setting with a Michigan Progress Fetch robot. Scene perception and task execution are evaluated with a public household occlusion dataset and our cluttered scene dataset.

• The paper presents Semantic Robot Programming (SRP) which fuses demonstration-based programming with semantic mapping to enable goal-directed manipulation in cluttered environments.
• It introduces DIGEST, a hybrid perception method that combines discriminative detection and generative pose estimation to robustly interpret complex scenes.
• The approach generalizes across diverse initial conditions, offering enhanced adaptability and paving the way for more autonomous real-world robotic applications.

Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes

This paper introduces a novel programming paradigm called Semantic Robot Programming (SRP), which synergizes robot programming by demonstration with semantic mapping. The primary objective of SRP is to enable intuitive and effective communication between users and robots, particularly focusing on goal-directed manipulation tasks in cluttered environments. The paradigm is underpinned by the Discriminatively-Informed Generative Estimation of Scenes and Transforms (DIGEST) perception method, which provides a robust approach for scene perception using RGBD images.

Overview of SRP

Semantic Robot Programming allows a user to set a desired goal for a robot by providing a demonstration of the goal scene. The robot interprets this goal as a scene graph, incorporating detailed information on object poses and inter-object spatial relationships. This scene graph essentially acts as a semantic map of the environment, providing the robot with a clear and detailed representation of its task. Task and motion planning algorithms are employed to achieve the user-defined goal, starting from varying initial configurations.

A notable feature of SRP is its ability to generalize across different starting conditions. This adaptability is facilitated by the DIGEST method, which aids in discerning both the initial and target states of the environment using visual data. This dual capability ensures that SRP can be applied in diverse, cluttered settings, thereby significantly broadening its applicability in real-world scenarios.

DIGEST: Advanced Perception Method

DIGEST integrates discriminative object detection with generative pose estimation to effectively interpret scenes with substantial clutter. It employs a combination of discriminative object detectors, which propose likely object detections in an image, and a generative approach that samples possible object poses. This hybrid methodology is crucial in accurately estimating 6 DOF poses of objects from complex scenes, even when the number of objects is predetermined.

In benchmarking studies, DIGEST demonstrated superior accuracy compared to alternative methods such as D2P and OUR-CVFH, particularly at stringent pose estimation thresholds. This performance underscores DIGEST’s efficacy in real-world scenarios involving occlusions and clutter.

Implications and Future Directions

The proposed SRP paradigm has significant implications for the field of robotics, especially in service and domestic robots, where the ability to comprehend and manipulate cluttered scenes is paramount. By enabling robots to reason about tasks in terms of high-level goals rather than specific low-level commands, SRP paves the way for more autonomous and intelligent robotic systems.

From a theoretical standpoint, the integration of semantic mapping with semantic robot programming represents a step toward more versatile frameworks for human-robot interaction. Practically, this can lead to enhanced usability and efficiency in various applications, including household robotics and industrial automation.

The paper points towards several avenues for future work, such as optimizing the computational efficiency of scene perception techniques and enhancing grasp planning methods. Moreover, further research could investigate the interplay between semantic programming and other modalities like natural language, thereby enriching the scope and impact of SRP.

In summary, by bridging semantic mapping with robot programming by demonstration, this work introduces a robust and adaptable framework for robotic manipulation in cluttered environments. Through the effective use of DIGEST for scene perception, SRP stands poised to significantly advance the capabilities and scope of autonomous robotic systems.