Harvesting Multiple Views for Marker-less 3D Human Pose Annotations (1704.04793v1)

Abstract: Recent advances with Convolutional Networks (ConvNets) have shifted the bottleneck for many computer vision tasks to annotated data collection. In this paper, we present a geometry-driven approach to automatically collect annotations for human pose prediction tasks. Starting from a generic ConvNet for 2D human pose, and assuming a multi-view setup, we describe an automatic way to collect accurate 3D human pose annotations. We capitalize on constraints offered by the 3D geometry of the camera setup and the 3D structure of the human body to probabilistically combine per view 2D ConvNet predictions into a globally optimal 3D pose. This 3D pose is used as the basis for harvesting annotations. The benefit of the annotations produced automatically with our approach is demonstrated in two challenging settings: (i) fine-tuning a generic ConvNet-based 2D pose predictor to capture the discriminative aspects of a subject's appearance (i.e.,"personalization"), and (ii) training a ConvNet from scratch for single view 3D human pose prediction without leveraging 3D pose groundtruth. The proposed multi-view pose estimator achieves state-of-the-art results on standard benchmarks, demonstrating the effectiveness of our method in exploiting the available multi-view information.

• The paper introduces a novel geometry-driven method that integrates multi-view 2D ConvNet predictions with 3D pictorial structures to produce accurate marker-less 3D pose annotations.
• It demonstrates state-of-the-art performance on benchmarks like Human3.6M by fine-tuning models to adapt to personalized and data-scarce conditions.
• The approach eliminates reliance on traditional MoCap datasets, enabling the training of ConvNets from scratch for single-view 3D human pose estimation.

Analysis of Geometry-Driven Annotation Collection for 3D Human Pose Prediction

The paper "Harvesting Multiple Views for Marker-less 3D Human Pose Annotations" introduces a novel methodology that facilitates the automatic collection of 3D human pose annotations using a geometry-driven approach, which addresses challenges related to the dependence on annotated data for training convolutional networks (ConvNets) in computer vision tasks. This methodology is premised on utilizing a multi-view camera setup combined with a generic ConvNet for 2D human pose estimation to derive accurate 3D poses without requiring markers.

This approach leverages the constraints of 3D camera geometry and human anatomical structure to probabilistically integrate 2D ConvNet predictions from multiple views into a cohesive and optimized 3D pose estimation. This optimization is achieved using a 3D pictorial structures model that consolidates per-view evidence into a common 3D space with pairwise constraints representing the human skeletal structure. By computing the marginalized posterior distribution of the 3D model, the approach enables the identification of reliable annotations with uncertainty metrics drawn from this distribution.

The significance of this methodology is particularly evident in two contexts. Firstly, it facilitates the fine-tuning of a generic ConvNet-based 2D pose predictor to adapt to specific subjects—an approach termed as "personalization." Secondly, it permits the training of a ConvNet from scratch for single-view 3D human pose estimation without relying on conventional 3D ground truth data. This latter capability is particularly noteworthy as it addresses the challenge of data scarcity for 3D human pose annotations, typically constrained by the availability of motion capture (MoCap) data collected in controlled settings.

Empirically, this paper demonstrates the effectiveness of their approach through state-of-the-art results on known benchmarks such as KTH Multiview Football II and Human3.6M datasets, which validate the competitive performance of the proposed multi-view 3D pose estimation approach. Furthermore, the adaptation for "personalization" on test-specific conditions showed significant improvements in performance, highlighting the utility of such refinements in varying conditions.

The implications of this work are substantial for both theoretical and practical applications. Theoretically, it affirms the capacity to use geometry-constrained neural network predictions to overcome limitations in acquiring large-scale annotated datasets, which commonly restrict machine learning models. Practically, these techniques could be applied to develop robust human pose estimation systems adaptable to diverse environments and individuals, such as automated video surveillance or motion analysis in sports.

Looking ahead, a promising direction suggested by this work is the collection of 3D annotations in unrestricted environments. This extension could empower the training of comprehensive 3D human pose ConvNets that are no longer limited to in-lab datasets but that also generalize effectively to real-world scenarios. Such advancements hold the potential to democratize the access and applicability of sophisticated human pose estimation technologies across various fields.