Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (1903.08527v2)

Abstract: Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency.However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on three datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance.

• The paper introduces a hybrid loss for single-image 3D face reconstruction, significantly enhancing shape fidelity on benchmark datasets.
• The paper proposes a weakly-supervised multi-image aggregation strategy that leverages confidence scores to integrate complementary facial information.
• The method outperforms existing approaches on MICC, FaceWarehouse, and BU-3DFE, reducing RMSE and advancing 3D modeling accuracy.

Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set

The paper presents a novel approach to 3D face reconstruction from 2D images, leveraging weakly-supervised learning methodologies to overcome limitations posed by the scarcity of ground-truth 3D data. The authors introduce a hybrid loss function designed to enhance the accuracy and robustness of 3D face reconstruction, incorporating multi-image reconstruction techniques to optimize information integration from diverse image conditions.

Key Contributions and Methodology

The paper proposes two primary contributions: a single-image reconstruction method and a multi-image aggregation strategy. Both make use of deep learning techniques without relying heavily on labeled 3D datasets.

1. Hybrid Loss for Single-Image Reconstruction:
   • The authors identify challenges associated with using either pixel-level or perception-level information exclusively. They propose a hybrid loss function that integrates both, achieving improved reconstruction accuracy by balancing low-level pixel information with high-level perceptual features derived from a pretrained face recognition network.
   • This robust approach mitigates local minima and results in better shape fidelity, as demonstrated on various datasets including MICC and FaceWarehouse.
2. Weakly-Supervised Multi-Image Reconstruction:
   • For image sets capturing a subject from multiple angles, the authors design a mechanism to aggregate 3D reconstruction results. This aggregation is performed through a confidence-based mechanism trained in a weakly-supervised fashion, wherein coefficients representing the identity are mined diligently from each image.
   • The approach utilizes element-wise confidence scores to leverage complementary information from different viewing angles, yielding more accurate 3D shapes.

Numerical Results

The paper reports state-of-the-art performance on several benchmark datasets. Notable improvements in reconstruction errors are documented:

• On the MICC Florence dataset, the proposed method outperforms existing techniques such as those of Tran et al. and Genova et al., achieving lower RMSE values across various scenarios.
• Comparisons on the FaceWarehouse and BU-3DFE datasets further attest to the efficacy of the hybrid loss function and the multi-image aggregation strategy, with significant error reductions compared to competitors.

Theoretical and Practical Implications

The presented methodology exemplifies a significant stride in leveraging weak supervision for 3D shape modeling. The hybrid loss function marks a departure from reliance solely on dense labelling, expanding the utility of neural networks in contexts where accurate 3D data is limited. By demonstrating proficiency in reconstructing accurate shapes from unconstrained images, the work underscores the growing potential of deep learning in domains requiring high computational efficiency and robustness against varying image quality.

Future Research Directions

The implications of this research suggest several avenues for future investigation:

• Extending the confidence-based aggregation strategy could further enhance techniques in dynamic and cluttered environments where facial data acquisition occurs in real time.
• Exploring hybrid supervision methods in other applications, such as non-rigid object reconstruction, could yield insights into more complex scenarios beyond facial reconstruction.
• Integrating this approach with more advanced texture modeling techniques could improve the visual realism of the reconstructed faces, opening avenues for its application in augmented reality and virtual environments.

In summary, the paper’s contributions to weakly-supervised learning in 3D face reconstruction present clear advantages over traditional methods. The innovative use of a hybrid loss and confidence-based aggregation aligns with advancing the field of computer vision towards greater efficiency and accuracy.