FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction (2111.01082v2)

Abstract: In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and the corresponding benchmark to evaluate single-view facial 3D reconstruction. By training on FaceScape data, a novel algorithm is proposed to predict elaborate riggable 3D face models from a single image input. FaceScape dataset releases $16,940$ textured 3D faces, captured from $847$ subjects and each with $20$ specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniform. These fine 3D facial models can be represented as a 3D morphable model for coarse shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different from most previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. We also use FaceScape data to generate the in-the-wild and in-the-lab benchmark to evaluate recent methods of single-view face reconstruction. The accuracy is reported and analyzed on the dimensions of camera pose and focal length, which provides a faithful and comprehensive evaluation and reveals new challenges. The unprecedented dataset, benchmark, and code have been released at https://github.com/zhuhao-nju/facescape.

• The paper introduces FaceScape, a comprehensive 3D facial dataset with 16,940 models across 847 subjects and 20 expressions each.
• It presents a novel two-stage pipeline that combines coarse mesh fitting with neural network-driven displacement map refinement for improved detail.
• Quantitative evaluations show the model reduces reconstruction errors compared to previous approaches, supporting advanced facial recognition and animation applications.

Overview of "FaceScape: 3D Facial Dataset and Benchmark for Single-View 3D Face Reconstruction"

This paper presents the FaceScape dataset, a significant contribution to the domains of computer vision and graphics focusing on 3D face reconstruction from a single image. It introduces a detailed large-scale dataset alongside a benchmark for evaluating single-view facial 3D reconstruction methods, addressing several challenges and limitations of prior datasets.

Dataset and Methodology

FaceScape stands out by providing 16,940 meticulously captured 3D facial models from 847 diverse subjects, each demonstrating 20 distinct expressions. The capturing process employed a dense 68-camera array, enabling the acquisition of facial geometry at the pore level. The models are processed into topologically uniform meshes and further enhanced with displacement maps that capture fine geometric details.

The paper introduces a two-stage pipeline for 3D face model prediction using this dataset. Initially, a coarse mesh is fitted through alignment with 2D detected landmarks. Subsequently, displacement maps are predicted using a neural network, allowing the synthesis of dynamic facial details responsive to various expressions. This approach contrasts with previous methodologies by providing riggable, highly detailed 3D models that can adapt to different expressions accurately.

Numerical Results and Evaluations

Quantitative evaluations reveal that the bilinear model generated from FaceScape data exhibits superior representation capabilities compared to existing models like FaceWarehouse and FLAME. The benchmark results indicate a substantial reduction in reconstruction error, underscoring the dataset’s utility in improving model accuracy, particularly for underside views and extreme facial expressions.

In terms of benchmarks, FaceScape provides both in-the-wild and in-the-lab datasets, facilitating comprehensive single-view face reconstruction evaluations. The benchmarks categorize results based on pose angle and focal length, revealing challenges and performance variations across different conditions. Notably, methods evaluated include Extreme3DFace, PRNet, and others, with the proposed model achieving high success rates and competitive accuracy.

Implications and Future Work

FaceScape’s release catalyzes advancements not only in 3D facial reconstruction but also in related fields such as biometric verification, facial animation, and AR applications. The ability to accurately model dynamic facial details from a single image input offers practical benefits, enhancing applications ranging from digital character creation to real-time facial recognition systems.

The paper acknowledges the dataset's limitation in racial diversity, being predominantly composed of Asian subjects. Future work could entail the inclusion of a more varied demographic to mitigate potential biases. Additionally, addressing challenges like improved reconstruction for side views and under extreme lighting conditions remains an open area for progress.

Overall, this work sets a new standard for 3D facial datasets while proposing effective methodologies for realistic and detailed 3D face reconstruction. Continued exploration and expansion of this dataset can foster significant developments in artificial intelligence and computational aesthetics.