Reconstructing People, Places, and Cameras (2412.17806v2)

Abstract: We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people. Our approach combines data-driven scene reconstruction with the traditional Structure-from-Motion (SfM) framework to achieve more accurate scene reconstruction and camera estimation, while simultaneously recovering human meshes. In contrast to existing scene reconstruction and SfM methods that lack metric scale information, our method estimates approximate metric scale by leveraging a human statistical model. Furthermore, it reconstructs multiple human meshes within the same world coordinate system alongside the scene point cloud, effectively capturing spatial relationships among individuals and their positions in the environment. We initialize the reconstruction of humans, scenes, and cameras using robust foundational models and jointly optimize these elements. This joint optimization synergistically improves the accuracy of each component. We compare our method to existing approaches on two challenging benchmarks, EgoHumans and EgoExo4D, demonstrating significant improvements in human localization accuracy within the world coordinate frame (reducing error from 3.51m to 1.04m in EgoHumans and from 2.9m to 0.56m in EgoExo4D). Notably, our results show that incorporating human data into the SfM pipeline improves camera pose estimation (e.g., increasing RRA@15 by 20.3% on EgoHumans). Additionally, qualitative results show that our approach improves overall scene reconstruction quality. Our code is available at: https://github.com/hongsukchoi/HSfM_RELEASE

• The paper introduces the HSfM framework to jointly reconstruct human meshes, scene point clouds, and camera parameters in a unified metric space.
• It integrates 2D human keypoints with 3D reconstructions using SMPL-X and scene models, significantly reducing human localization errors.
• Evaluation on benchmarks like EgoHumans demonstrates enhanced camera pose accuracy, advancing dynamic scene understanding for applications such as AR/VR.

Reconstructing People, Places, and Cameras: A Synthesis of Human-Structure-Scene Interactions

The paper "Reconstructing People, Places, and Cameras" introduces the Humans and Structure from Motion (HSfM) framework, aimed at jointly reconstructing human meshes, scene point clouds, and camera parameters in a unified metric world coordinate system. This framework builds upon the foundational principles of Structure-from-Motion (SfM) by integrating human statistical models and leveraging robust initializations from state-of-the-art scene and human reconstruction methods. The research highlights significant advancements in producing accurate multi-view reconstructions of dynamic scenes involving human interactions.

Methodology

The HSfM framework explicitly incorporates humans into the SfM pipeline using a two-pronged approach: first, by employing 2D human keypoint correspondences, and second, by optimizing 3D human mesh and scene reconstructions concurrently. Initial reconstructions of scenes, humans, and cameras are derived from separate models, with humans initialized using SMPL-X parameters converted from predictions by existing reconstruction models, and scenes and cameras initialized as per predictions from contemporary scene reconstruction techniques.

The proposed methodology aligns these elements within a unified world framework by estimating the metric scale aligning scene pointmaps and cameras with humans—a crucial innovation given the challenge of accurately situating multiple humans within a consistent spatial and metric framework. This alignment resolves the scale ambiguity inherent in prior multi-view and SfM reconstructions and also enhances camera pose estimation using derived human-specific metric information.

Numerical and Empirical Insights

The HSfM framework is rigorously evaluated against challenging benchmarks like EgoHumans and EgoExo4D, demonstrating substantial improvements in human localization accuracy. For instance, in the EgoHumans dataset, the approach achieves a reduction in human world location error from 3.51 meters to 1.04 meters and shows a marked improvement in camera pose estimation, with a 20.3% boost in RRA@15 scores when compared to previous methodologies. Such results underscore the effectiveness of the joint optimization approach, evidencing how human data within the SfM pipeline can refine camera and scene estimations.

Theoretical and Practical Implications

The paper's contributions have far-reaching implications. Theoretically, it provides a comprehensive framework that bridges the gap between static scene interpretation and dynamic human interaction modeling, introducing a novel synthesis of geometry, learning, and statistical modeling. Practically, the HSfM approach significantly improves the accuracy and utility of 3D reconstructions for applications in areas such as surveillance, AR/VR environments, and interactive systems where understanding human-space interactions are crucial.

Future Directions

Looking forward, the exploration of more automated re-identification processes for people across multiple camera views presents a promising avenue, as manual identification remains a limiting factor. Additionally, extending the framework to a feed-forward mechanism could offer real-time applications of this technology, utilizing the synergy between humans and scene data more efficiently. There is also the potential to apply these methodologies within the context of video sequences to further improve temporal stability and consistency in dynamic scenes.

In summary, the HSfM framework addresses existing limitations in multi-view scene and human reconstruction by proposing an integrated system that enhances spatial accuracy and consistency. These advancements offer a foundational step towards more comprehensive and reliable multi-view reconstruction techniques.