Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo (2104.02273v1)

Abstract: Existing approaches for multi-view multi-person 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views and solve for the 3D pose estimation for each person. Establishing cross-view correspondences is challenging in multi-person scenes, and incorrect correspondences will lead to sub-optimal performance for the multi-stage pipeline. In this work, we present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot. Specifically, we propose to perform depth regression for each joint of each 2D pose in a target camera view. Cross-view consistency constraints are implicitly enforced by multiple reference camera views via the plane sweep algorithm to facilitate accurate depth regression. We adopt a coarse-to-fine scheme to first regress the person-level depth followed by a per-person joint-level relative depth estimation. 3D poses are obtained from a simple back-projection given the estimated depths. We evaluate our approach on benchmark datasets where it outperforms previous state-of-the-arts while being remarkably efficient. Our code is available at https://github.com/jiahaoLjh/PlaneSweepPose.

Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo: A Detailed Analysis

The paper in question introduces a novel approach for the challenging task of multi-view multi-person 3D pose estimation, leveraging the concept of plane sweep stereo to conduct depth regression without explicit cross-view correspondence. Traditional methods often rely on establishing explicit correspondences across camera views, which is inherently complex in multi-person scenes due to occlusions and identity ambiguities. This new proposal aims to address these deficiencies by integrating cross-view fusion and 3D pose reconstruction in a unified manner.

The authors propose a framework where the depth is regressed for each joint in a 2D pose from a target camera view. Multiple reference camera views are employed using the plane sweep algorithm to enforce implicit cross-view consistency, enhancing the accuracy of depth regression. The methodology comprises a two-stage depth estimation process involving a person-level depth regression, followed by a joint-level relative depth regression. Depth information is back-projected to generate 3D poses after estimation is completed.

Core Contributions and Methodology

1. Depth Regression through Plane Sweep Stereo: The paper revolutionizes the approach to depth estimation in multi-person scenarios by avoiding the traditional triangulation and explicit cross-view matching. Instead, it utilizes depth regression where cross-view consistency is measured implicitly across multiple camera views using plane sweep stereo principles. This approach has been shown to be computationally efficient in both depth estimation and subsequent 3D pose reconstruction.
2. Coarse-to-Fine Regression Scheme: The authors implement a two-tiered regression strategy. The first tier involves estimating the depth at a person level, regressing the depth of the central hip joint as an anchor. The subsequent tier refines this estimation by calculating relative depths for individual joints—allowing for fine-grained adjustments of depth relative to each person's 3D anchor point. Such a scheme accommodates depth variations at a joint level, enhancing accuracy significantly.
3. Score Aggregation via Geometric Consistency: A geometric consistency score is developed to guide depth regression. This is achieved by measuring pose alignment errors across different depth planes. Essentially, 2D poses are matched against potential positions at various depths across all camera views, assisting in forming a robust depth score utilized for regression.

Evaluation and Discussion

The proposed method demonstrated superior performance against existing state-of-the-art methods on both the campus and shelf datasets, as well as on the CMU Panoptic dataset. Notably, the approach improved Precision Calculation Percentage (PCP) and Mean Per Joint Position Error (MPJPE) metrics while operating with significantly increased computational efficiency. The framework offers an inference speed that is several times faster than traditional techniques while requiring no pre-defined knowledge of the common 3D space dimensions that are typically necessary for voxel-based methods.

The paper's rigorous evaluations highlight the effectiveness of plane sweep stereo in high-density multi-person environments, an area where established approaches often falter due to incorrect correspondence or high computational demands. Moreover, the flexibility of the plane sweep mechanism makes the methodology applicable across different camera configurations, which denotes a significant advantage over voxel-based methods where the configuration needs explicit tweaking.

Implications and Future Directions

This work paves the way for further research into depth estimation methodologies in multi-person scenarios without reliance on explicit matching algorithms. It may lead to developing more robust, adaptive detection systems applicable in real-time settings, such as surveillance or interaction-heavy environments like AR/VR.

Future research could investigate incorporating additional modalities such as temporal information for dynamic scenes or enhancing depth regression models with photometric consistency checks. There is also potential in exploring how such a framework could be integrated with deep learning models to refine adaptability to unseen scenes or align better with semantic information.

Conclusion

The paper provides significant insights into multi-view multi-person 3D pose estimation, demonstrating that plane sweep stereo principles can effectively off-load the complexity of cross-view correspondence, delivering efficient and accurate 3D pose reconstructions. This contribution not only marks an advancement in the methodology of 3D human pose estimation but also foregrounds the importance of developing adaptable, fast, and robust systems for multi-camera settings.