MoFaNeRF: Morphable Facial Neural Radiance Field (2112.02308v2)

Abstract: We propose a parametric model that maps free-view images into a vector space of coded facial shape, expression and appearance with a neural radiance field, namely Morphable Facial NeRF. Specifically, MoFaNeRF takes the coded facial shape, expression and appearance along with space coordinate and view direction as input to an MLP, and outputs the radiance of the space point for photo-realistic image synthesis. Compared with conventional 3D morphable models (3DMM), MoFaNeRF shows superiority in directly synthesizing photo-realistic facial details even for eyes, mouths, and beards. Also, continuous face morphing can be easily achieved by interpolating the input shape, expression and appearance codes. By introducing identity-specific modulation and texture encoder, our model synthesizes accurate photometric details and shows strong representation ability. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view synthesis. Experiments show that our method achieves higher representation ability than previous parametric models, and achieves competitive performance in several applications. To the best of our knowledge, our work is the first facial parametric model built upon a neural radiance field that can be used in fitting, generation and manipulation. The code and data is available at https://github.com/zhuhao-nju/mofanerf.

• The paper introduces MoFaNeRF, which integrates 3D morphable models and neural radiance fields to synthesize photorealistic facial images.
• It employs a multi-layer perceptron to map facial features into a vector space, enabling continuous morphing and dynamic editing.
• Experimental results demonstrate improved PSNR, SSIM, and LPIPS scores, outperforming traditional 3D models in realistic rendering.

Overview of MoFaNeRF: Morphable Facial Neural Radiance Field

The paper introduces MoFaNeRF (Morphable Facial Neural Radiance Field), a novel approach in the domain of facial modeling and image synthesis, leveraging the capabilities of neural radiance fields (NeRFs) to construct a parametric facial model. Unlike traditional 3D Morphable Models (3DMMs), which struggle with producing realistic images without complex rendering pipelines, MoFaNeRF directly synthesizes photo-realistic facial details from images under various viewpoints. This model combines the strengths of both 3DMM for scalable shape representation and NeRF for photorealistic rendering to achieve superior synthetic ability in facial depiction, including challenging features like eyes, mouths, and beards.

Structural and Functional Aspects

MoFaNeRF maps free-view facial images into a coded vector space, denoting facial shape, expression, and appearance. This mapping is implemented through a Multi-Layer Perceptron (MLP) using a neural radiance field to output the radiance of a spatial point, facilitating the synthesis of photo-realistic imagery. The model is distinguished by its capability to interpolate between inputs, allowing for continuous facial morphing which enables dynamic applications such as face rigging and editing.

The contribution of identity-specific modulation and texture encoding further enhances MoFaNeRF’s representation abilities. These features contribute to synthesizing accurate photometric details by effectively disentangling the facial shape, appearance, and expression parameters. The architecture's capacity is optimized to memorize the expansive database of face images, addressing the challenges posed by previous methods in achieving the blend of realistic rendering and editable face synthesis.

Experimental Evaluation

Numerical results establish MoFaNeRF's significant improvement over existing parametric models like the 3DMM and the implicit function-based i3DMM. The key metrics – PSNR, SSIM, and LPIPS – demonstrate MoFaNeRF's competitive performance in image reconstruction and generative tasks. The model's superiority is further exemplified by its ability to render high-fidelity images across diverse facial expressions and appearances while maintaining identity consistency across various viewing angles.

Practical Implications and Future Directions

MoFaNeRF lays the groundwork for practical applications in multiple areas: image-based fitting, random face generation, and novel view synthesis. Furthermore, its robust interpolation mechanism supports innovative use cases in facial animation and editing, with the potential for more intricate, real-time consumer and industrial applications. Notably, the model also demonstrates capabilities in novel view synthesis, outperforming traditional rendering techniques used in gaming and virtual reality.

In addressing future developments, the incorporation of more complex dynamic scenes using the model as groundwork holds substantial promise. Expanding beyond facial synthesis, similar techniques could enhance whole-body modeling employing more sophisticated scene understanding. Additionally, integrating MoFaNeRF with advanced lighting and environmental dynamics can enhance realism even further, thus broadening its applicability in diverse and challenging conditions.

Conclusion

MoFaNeRF distinguishes itself as a progressive stride in marrying the complexities of facial morphing models with the adeptness of neural radiance fields to produce highly detailed, editable, and realistic facial images. This work articulates a significant shift from conventional methodologies and opens avenues for advanced research and application, enhancing the toolkit available for computer vision and graphics applications.