FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images (1909.04349v3)

Abstract: Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this real-world dataset, we propose an iterative, semi-automated `human-in-the-loop' approach, which includes hand fitting optimization to infer both the 3D pose and shape for each sample. We show that methods trained on our dataset consistently perform well when tested on other datasets. Moreover, the dataset allows us to train a network that predicts the full articulated hand shape from a single RGB image. The evaluation set can serve as a benchmark for articulated hand shape estimation.

• The paper introduces a novel dataset with 11,592 annotations across 32 subjects to enhance the robustness of hand pose and shape estimation.
• It employs a semi-automated, human-in-loop process with synchronized multi-view captures from eight cameras to ensure high-fidelity 3D keypoint annotations.
• The dataset significantly improves cross-dataset generalization, benefiting applications in robotics, augmented reality, and human-computer interaction.

Overview of the FreiHAND Dataset for Markerless Hand Pose and Shape Capture

The paper conducted by Zimmermann et al. presents a comprehensive dataset, FreiHAND, aimed at advancing the capabilities of hand pose and shape estimation from single RGB images. The dataset addresses significant limitations observed in previous datasets, such as poor cross-dataset generalization and a lack of large-scale, real-world annotated data. By introducing a large-scale, multi-view dataset with 3D hand pose and shape annotations collected in real-world scenarios, this work offers a valuable resource for improving the robustness and accuracy of hand pose estimation models.

Key Contributions and Dataset Characteristics

The authors identified a pervasive issue across existing datasets: models trained on one dataset often fail to generalize effectively to other datasets or "in-the-wild" conditions. To mitigate this problem, FreiHAND is developed using a novel semi-automated iterative annotation process. This process uses a 'human-in-the-loop' strategy, optimizing a deformable hand model through sparse manual annotations and multi-view images. The result is a dataset comprising 11,592 annotations spanning 32 different subjects with diverse hand poses and shapes, captured both with and without various object interactions. This diversity is critical, as it enhances the dataset's ability to serve as a robust training ground for machine learning models, advocating a marked improvement in cross-dataset generalization.

Methodology and Technical Rigor

The methodology leveraged by the authors is rigorous and involves multi-view captures from eight cameras arranged within a portable setup. The authors adhere to a principal strategy of utilizing synchronous captures across different views, removing occlusions, and facilitating efficient manual and automated annotation. The semi-automated annotation is executed by predicting 3D keypoints using a neural network (MVNet) specifically designed to process multi-view images. The method ensures high fidelity in the annotations and reduces manual effort significantly.

The dataset augments classical approaches by incorporating a verification step where high-confidence hand-shape fittings calculated by MVNet and segmentation masks automatically verify plausible annotations, minimizing human intervention where feasible. The careful consideration of annotation fidelity, through both manual and automatic verification, highlights the dataset's accuracy, rendering it a credible benchmark within the research community.

Numerical Results and Performance

The numerical results presented demonstrate the superiority of models trained on FreiHAND. When tested against seven state-of-the-art datasets, algorithms trained using this dataset consistently outperform in cross-dataset evaluations, achieving a leading position in average ranks. Notably, the dataset facilitates substantial improvements in training neural networks for full 3D hand shape estimation—a task traditionally challenged by a lack of publicly available, annotated data. The performance evaluation, measured by area under the curve (AUC) for percentage of correct keypoints, reveals the efficacy of the dataset in enhancing generalization across diverse hand poses and scenarios.

Implications and Future Directions

This research has significant implications for fields that require precise hand pose estimation, including robotics, augmented reality, and human-computer interaction. By providing a dataset that is adaptable to varied conditions and applications, it supports a wide array of potential integrations in both academic and industrial applications. The availability of the dataset facilitates further research in markerless hand pose and shape estimation, laying groundwork for standardized benchmarks and evaluation protocols within the domain.

Looking forward, the paper underscores potential expansions to include further variation in hand-object interactions and in-the-wild captures that could enhance the dataset's comprehensiveness. Furthermore, improvements in automated annotation strategies may progressively reduce the dependency on manual interventions, paving the way for fully automated data acquisition systems in computer vision.

In summary, the FreiHAND dataset not only sets a high bar for markerless capture of hand pose and shape from RGB images but also presents a scalable framework for future datasets, reinforcing robust model training and evaluation standards in hand pose estimation research.