DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignment (2305.11522v1)

Abstract: Sensitivity to severe occlusion and large view angles limits the usage scenarios of the existing monocular 3D dense face alignment methods. The state-of-the-art 3DMM-based method, directly regresses the model's coefficients, underutilizing the low-level 2D spatial and semantic information, which can actually offer cues for face shape and orientation. In this work, we demonstrate how modeling 3D facial geometry in image and model space jointly can solve the occlusion and view angle problems. Instead of predicting the whole face directly, we regress image space features in the visible facial region by dense prediction first. Subsequently, we predict our model's coefficients based on the regressed feature of the visible regions, leveraging the prior knowledge of whole face geometry from the morphable models to complete the invisible regions. We further propose a fusion network that combines the advantages of both the image and model space predictions to achieve high robustness and accuracy in unconstrained scenarios. Thanks to the proposed fusion module, our method is robust not only to occlusion and large pitch and roll view angles, which is the benefit of our image space approach, but also to noise and large yaw angles, which is the benefit of our model space method. Comprehensive evaluations demonstrate the superior performance of our method compared with the state-of-the-art methods. On the 3D dense face alignment task, we achieve 3.80% NME on the AFLW2000-3D dataset, which outperforms the state-of-the-art method by 5.5%. Code is available at https://github.com/lhyfst/DSFNet.

• The paper introduces DSFNet, a dual-branch network that fuses pixel-level image cues with 3DMM geometric predictions to handle occlusions effectively.
• It employs a novel architecture where separate branches process image space details and model space coefficients, enhancing facial feature recovery under challenging conditions.
• Empirical evaluations demonstrate a 3.80% NME on AFLW2000-3D, outperforming state-of-the-art methods by 5.5% and showcasing strong resilience to occlusion.

DSFNet: Advancements in Occlusion-Robust 3D Dense Face Alignment

The paper presents DSFNet, a Dual Space Fusion Network, designed to enhance robustness and accuracy in 3D dense face alignment, particularly under conditions of occlusion and extreme view angles. The paper addresses significant challenges in the field of monocular 3D face alignment, striving to overcome the limitations faced by conventional methods when dealing with partially visible or obstructed facial features.

The proposed DSFNet effectively integrates predictions made in both image and model spaces. This dual approach leverages the strengths of pixel-level dense predictions in the visible regions while harnessing the facial geometry knowledge encoded in 3D Morphable Models (3DMM). This fusion is strategically implemented to bolster DSFNet’s efficacy in diverse and unconstrained settings, where existing approaches typically falter.

Methodological Innovations

The DSFNet model adopts a two-branch architecture:

1. Image Space Prediction: This branch first imbibes pixel-level information from the visible regions using a novel 3D facial geometry's 2D image space representation. By employing dense prediction in image space, the method mitigates the issues presented by occlusion and extreme viewing angles, primarily by relying on local rather than global facial cues.
2. Model Space Prediction: The model space branch predicts deeper facial geometry attributes through regression of 3DMM coefficients. This branch excels in providing context in cases with blurred or noisy inputs, where low-level information might degrade the prediction quality.
3. Dual Space Fusion: The output from these branches is synthesized in a dual space fusion module, which maximizes DSFNet’s adaptability by merging predictions from both spaces. The fusion leverages the model space features to appropriately weigh and blend predictions from both image and model spaces.

The paper also highlights a PointNet-based post-processing scheme that translates image space representations into complete 3D facial models. This post-process mechanism efficiently fills in the gaps for occluded or obscured portions of the face, thus improving overall model reliability and precision.

Empirical Evaluation

DSFNet demonstrates its superiority through extensive validation across multiple face alignment and reconstruction benchmarks. The approach achieves a mean normalized mean error (NME) of 3.80% on the AFLW2000-3D dataset, outperforming the current state-of-the-art by 5.5%. The paper also conducts evaluations for head pose estimation, further exhibiting DSFNet's proficiency, particularly in handling challenging cases with large yaw angles.

In assessing robustness against occlusion, DSFNet markedly outperforms competing methods on the newly compiled AFLW2000-3D-occlusion dataset, proving its resilience in scenarios where face visibility is severely compromised.

Implications and Future Directions

DSFNet introduces a significant methodological advancement by implementing a dual processing strategy that synthesizes low-level spatial and semantic information with high-level geometric relations. This innovation is crucial for practical applications involving video conferencing, augmented reality, and other visualization tasks demanding robust face alignment under challenging conditions.

Future directions could involve further refining the model’s generalization capabilities by incorporating semi-supervised learning techniques to tap into unlabeled datasets, thereby enhancing adaptability across more diverse environments. The incorporation of perspective projection techniques could also mitigate limitations associated with current orthographic assumptions, particularly for inputs captured from close distances.

The DSFNet framework sets the stage for upcoming advancements in 3D face computation, emphasizing the collaborative potential of integrating nuanced spatial representations with traditional geometric modeling techniques.