Reconstructing Humans with a Biomechanically Accurate Skeleton (2503.21751v1)

Abstract: In this paper, we introduce a method for reconstructing 3D humans from a single image using a biomechanically accurate skeleton model. To achieve this, we train a transformer that takes an image as input and estimates the parameters of the model. Due to the lack of training data for this task, we build a pipeline to produce pseudo ground truth model parameters for single images and implement a training procedure that iteratively refines these pseudo labels. Compared to state-of-the-art methods for 3D human mesh recovery, our model achieves competitive performance on standard benchmarks, while it significantly outperforms them in settings with extreme 3D poses and viewpoints. Additionally, we show that previous reconstruction methods frequently violate joint angle limits, leading to unnatural rotations. In contrast, our approach leverages the biomechanically plausible degrees of freedom making more realistic joint rotation estimates. We validate our approach across multiple human pose estimation benchmarks. We make the code, models and data available at: https://isshikihugh.github.io/HSMR/

Reconstructing Humans with a Biomechanically Accurate Skeleton

The paper "Reconstructing Humans with a Biomechanically Accurate Skeleton" presents a novel approach for reconstructing 3D human poses from single images utilizing a biomechanically accurate skeletal model, SKEL. This work addresses important limitations in current state-of-the-art methods that often produce unrealistic joint angles, thereby compromising biomechanical validity despite advancements in surface accuracy.

Technical Contributions

The proposed method, Human Skeleton and Mesh Recovery (HSMR), utilizes a transformer network to estimate the SKEL model parameters directly from a single image. Key technical contributions include the adaptation of SKEL to integrate an anatomically faithful skeleton with the SMPL surface mesh, and the development of a pseudo ground truth generation pipeline with iterative refinement. Such a setup efficiently addresses the lack of datasets directly annotated with SKEL parameters, thereby enabling the network to be trained on pseudo ground truth that progressively improves through the training iterations.

The paper emphasizes several significant advancements over traditional 3D human mesh recovery models. Notably, SKEL employs joint constraints calibrated to actual human biomechanical limits, as opposed to treating human joints as simple ball-and-socket entities with three degrees of freedom, which is a common simplification in models like SMPL and its derivatives. The SKEL model incorporates Euler angles for joint rotations, constrained to their biomechanical limits, thereby producing more accurate and anatomically plausible reconstructions.

Experimental Evaluation

Extensive experiments were conducted on various datasets, including both those with 3D annotations (Human3.6M, 3DPW, MOYO) and those without (COCO, PoseTrack, LSP-Extended), enabling a comprehensive evaluation of the proposed method across different benchmarks. HSMR demonstrated competitive performance with state-of-the-art models on standard 2D and 3D pose metrics, with a notable advantage in scenarios involving extreme poses and viewpoints (e.g., MOYO dataset). Specifically, while the quantitative differences on traditional metrics may appear modest, HSMR significantly reduces the violation of realistic joint limits, thus ensuring biomechanical accuracy.

Implications and Future Work

The implications of this research are profound, particularly for fields requiring precise human motion analysis, such as biomechanics, robotics, and AR/VR systems. The ability to recover 3D human poses with biomechanical accuracy opens up direct applications in areas that demand high fidelity in joint movement estimation.

This paper also sets the stage for further research into integrating biomechanical constraints more deeply into human pose and mesh recovery models. Future work may explore expanding the scope of the SKEL model with additional data-driven constraints or incorporating temporal coherence to improve predictions across video sequences. Additionally, optimizing the computational efficiency of the SKEL model fitting process could enhance applicability in real-time systems.

In summary, the paper presents a substantial contribution to human 3D pose estimation by innovating on the reconstruction quality with biomechanical plausibility, thereby bridging the gap between the requirements of computer vision applications and biomechanics. The release of the model's code and data encourages reproducibility and further exploration in the domain.