LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Abstract: Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.

• The paper presents a novel RGBD labeling pipeline that reduces annotation time dramatically through an ICP-based 3-click alignment method.
• It achieves high precision by leveraging dense 3D reconstructions and ElasticFusion to render pixelwise labels and 6DOF object poses.
• Empirical results show a 190% IoU improvement for multi-object scenes, highlighting the significance of diverse training data in robotic applications.

LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Introduction

The paper "LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes" presents an advanced methodology for the automatic generation of labeled RGBD datasets crucial for training deep neural networks (DNNs) in robotic manipulation tasks. The approach addresses the challenge posed by the necessity for extensive labeled datasets to optimize the performance of DNNs, a task traditionally hindered by prohibitive time and cost demands associated with human annotation. LabelFusion streamlines this process through a novel pipeline capable of rapidly producing high-quality, labeled RGBD data.

The developed pipeline capitalizes on an RGBD camera to capture multi-viewpoint video of scenes, applying dense 3D reconstructions to facilitate manual labeling of object poses via ICP-fitting onto object meshes. This procedure significantly minimizes human effort, allowing for efficient dataset construction. It enables the generation of pixelwise labels and 6DOF object poses, creating over 1,000,000 labeled instances within days and democratizing dataset creation for a variety of robotic manipulation research applications.

Data Generation Pipeline

The essence of the LabelFusion pipeline involves several key stages: RGBD data collection, dense 3D reconstruction, object mesh generation, human-assisted annotation, and labeled image rendering. The pipeline's flexibility supports both automated and manual data collection methods, accommodating environments with clutter, occlusions, and varying lighting conditions.

Figure 1: Examples of labeled data generated by our pipeline: (a) heavily cluttered multi-object, (b) low light conditions, (c) motion blur, (d) distance from object, (e) 25 different environments. All of these scenes were collected by hand-carrying the RGBD sensor.

The reconstruction technique leverages ElasticFusion to process RGBD data into detailed 3D point clouds, providing the structural basis for accurate labeling. Critical to this pipeline is the user-friendly annotation interface which facilitates ICP-aligned object placement using a 3-click technique, significantly reducing annotation time to approximately 30 seconds per object. This methodology enhances alignment precision even in complex, cluttered scenarios.

Empirical Evaluations

The paper explores the practical implications of dataset complexities on DNN training efficacy, particularly how data quantity and scenario variability impact generalization. Specifically, the experiments reveal that incorporating cluttered, multi-object scenes within training sets markedly improves segmentation accuracy compared to single-object scenes, highlighting the value of exposure to occlusions in training data. Results indicate a 190% improvement in IoU for networks trained on multi-object scenes over single-object counterparts when evaluating multi-object test scenes.

Figure 2: Comparisons of training on single-object vs. multi-object scenes and testing on single-object (left) and multi-object (right) scenes.

The paper also examines training set environment diversity, illustrating improvements in generalization when trained across various backgrounds. These findings underscore the need for diverse training data, enhancing model adaptability to novel environments.

Figure 3: Comparison of segmentation performance on novel multi-object test scenes. Networks are either trained on (a) single object scenes only, (b,d), multi-object test scenes only, or a mixture (c,e).

Conclusion

LabelFusion provides a transformative open-source solution for the rapid generation of vast labeled datasets necessary for developing sophisticated DNNs in robotic applications. The pipeline significantly reduces human annotation time while enhancing the accuracy and applicability of training datasets. The empirical evaluations suggest prioritizing multi-object scene data collection across diverse backgrounds to optimize DNN generalization capabilities. This work not only facilitates advancements in robotic perception but also guides future dataset compilation strategies, ensuring comprehensive and high-fidelity training resources.

The contributions of LabelFusion in reducing data labeling efforts and its impact on segmentation quality establish it as an essential tool for researchers aiming to leverage DNNs for complex robotic tasks, fostering advancements in autonomous manipulation through robust, adaptable data-driven models.