3D Human Mesh Regression with Dense Correspondence (2006.05734v2)

Abstract: Estimating 3D mesh of the human body from a single 2D image is an important task with many applications such as augmented reality and Human-Robot interaction. However, prior works reconstructed 3D mesh from global image feature extracted by using convolutional neural network (CNN), where the dense correspondences between the mesh surface and the image pixels are missing, leading to suboptimal solution. This paper proposes a model-free 3D human mesh estimation framework, named DecoMR, which explicitly establishes the dense correspondence between the mesh and the local image features in the UV space (i.e. a 2D space used for texture mapping of 3D mesh). DecoMR first predicts pixel-to-surface dense correspondence map (i.e., IUV image), with which we transfer local features from the image space to the UV space. Then the transferred local image features are processed in the UV space to regress a location map, which is well aligned with transferred features. Finally we reconstruct 3D human mesh from the regressed location map with a predefined mapping function. We also observe that the existing discontinuous UV map are unfriendly to the learning of network. Therefore, we propose a novel UV map that maintains most of the neighboring relations on the original mesh surface. Experiments demonstrate that our proposed local feature alignment and continuous UV map outperforms existing 3D mesh based methods on multiple public benchmarks. Code will be made available at https://github.com/zengwang430521/DecoMR

• The paper introduces a novel DecoMR framework that employs dense pixel-to-surface correspondence in a continuous UV space for accurate 3D human mesh regression.
• It utilizes local image features transferred via an IUV image to preserve spatial relationships, enhancing both global accuracy and detail in mesh reconstruction.
• Experimental results on benchmarks like Human3.6M and SURREAL validate DecoMR’s superior performance through improved mesh fidelity and precise alignment.

Overview of "3D Human Mesh Regression with Dense Correspondence"

The paper "3D Human Mesh Regression with Dense Correspondence" introduces a novel framework for 3D human mesh estimation from a single 2D image, named DecoMR. The primary advancement saliently distinguishing DecoMR from prior approaches is its explicit establishment of dense correspondence between image pixels and mesh surface areas in a UV space. This framework diverges from traditional methods that rely heavily on global image features and often yield suboptimal solutions due to the missing dense correspondence.

Key Contributions

1. Dense Correspondence in UV Space: The DecoMR framework employs a model-free methodology to achieve 3D human mesh regression, emphasizing the alignment of local image features with UV space representations. This approach enhances the pixel-to-surface correspondence, significantly influencing the accuracy of mesh reconstruction.
2. Continuous UV Mapping: The paper proposes a novel continuous UV map that preserves the majority of neighboring relations on the original mesh surface, thus optimizing learning efficiency. This contrasts with traditional discontinuous UV maps which can degrade network learning due to lost spatial relations between adjacent mesh parts.
3. Utilization of Local Image Features: By transferring local image features to the UV space via an IUV image, DecoMR explicitly preserves information pertinent to the original mesh structure. This alignment between the feature map and the location map in the UV space addresses correspondence issues pervasive in prior models.

Experimental Insights

The proposed DecoMR framework outperforms existing models across multiple public benchmarks. Notably, on datasets such as Human3.6M and SURREAL, DecoMR achieves superior metric scores (e.g., MPJPE-PA), substantiating the efficacy of integrating dense correspondence and continuous UV mapping into the 3D mesh estimation pipeline. Experimental results underscore the improvement in both global and local feature utilization, enhancing the fidelity and detail of the reconstructed meshes.

Practical and Theoretical Implications

Practically, DecoMR has considerable utility in applications requiring realistic 3D reconstructions from 2D inputs, such as in augmented reality and human-robot interaction. The theoretical grounding presented in the framework offers a substantial leap towards more robust model-free 3D reconstructions by optimizing the integration of local and UV-mapped features.

Speculations on Future Research

Future research could expand upon DecoMR by exploring the representation of finer surface details, such as clothing textures and non-rigid elements (e.g., hair). Furthermore, the applicability of DecoMR in dynamic and multi-view scenarios could be a substantive direction for subsequent exploration, contributing further towards comprehensive 3D scene reconstruction methodologies.

Overall, while DecoMR is not purported to be groundbreaking in every aspect, its contributions to enhancing dense pixel-to-surface correspondence stand as significant advancements in the field of 3D human mesh regression.