Fitting 3D Morphable Models using Local Features

Abstract: In this paper, we propose a novel fitting method that uses local image features to fit a 3D Morphable Model to 2D images. To overcome the obstacle of optimising a cost function that contains a non-differentiable feature extraction operator, we use a learning-based cascaded regression method that learns the gradient direction from data. The method allows to simultaneously solve for shape and pose parameters. Our method is thoroughly evaluated on Morphable Model generated data and first results on real data are presented. Compared to traditional fitting methods, which use simple raw features like pixel colour or edge maps, local features have been shown to be much more robust against variations in imaging conditions. Our approach is unique in that we are the first to use local features to fit a Morphable Model. Because of the speed of our method, it is applicable for realtime applications. Our cascaded regression framework is available as an open source library (https://github.com/patrikhuber).

• The paper introduces a cascaded regression method that leverages local features to fit 3DMMs with improved robustness.
• It simultaneously optimizes shape and pose parameters, achieving accurate estimation even under challenging imaging conditions.
• The approach offers real-time performance at approximately 200 ms per image, demonstrating its practical applicability in dynamic environments.

Fitting 3D Morphable Models using Local Features

Introduction

The paper introduces a novel method for fitting 3D Morphable Models (3DMMs) to 2D images using local image features. Previous methods often relied on global features or landmarks, which could be less robust under varying imaging conditions. The approach presented here employs cascaded regression, which uses local features derived from the image to simultaneously optimize both shape and pose parameters. This method improves upon traditional techniques by accommodating non-differentiability in feature extraction through learning-based strategies, and it is sufficiently fast for real-time applications.

Background and Methodology

The fitting of 3DMMs involves optimizing a cost function to align the model parameters with an observed 2D image. Traditional techniques typically handle this with either linear or nonlinear approaches, often relying on landmarks or pixel information. In contrast, the proposed method leverages local features, such as SIFT, which provide richer information and are less sensitive to variations in illumination and occlusion.

Cascaded Regression

The main innovation here is the use of cascaded regression, a learning-based method that learns the gradient direction from data, circumventing the non-differentiability issue of local features. The cascaded regression is characterized by successive application of regressors to update the parameter vector $\boldsymbol\theta$, which includes shape and pose parameters.

The process involves extracting local features such as SIFT around projected 2D locations on the image. These features are then used in a regression framework to iteratively refine model parameters.

3D Morphable Model

A 3DMM comprises a shape model, derived typically via PCA, from aligned 3D face data. This model allows the reconstruction of 3D shapes from a set of coefficients. The challenge addressed in the paper is to fit these coefficients and associated pose parameters such that the projected 3D model best matches the 2D image.

Experimental Results

Figure 1: 3D Morphable Model fitting using local features. (a) Input image. (b) The 3D model is projected using current parameters. (c) Regions where features are extracted.

Pose and Shape Fitting

The proposed method achieves superior results in fitting experiments, both for pose estimation and simultaneous optimization of pose and shape:

1. Pose Estimation: The algorithm achieves a mean absolute error in pose estimation, showing robustness even with challenging initializations (Figure 2).

   Figure 2: Cascaded regression-based 3DMM fitting evaluation of pose fitting under different initializations.

2. Joint Shape and Pose Fitting: By incorporating shape as a variable in the regression framework, the method achieves higher fidelity in representing the true 3D structure of the face (Figure 3).

   Figure 3: Simultaneous shape- and pose fitting accuracy on synthetic data.

3. Comparison to Existing Methods: When compared with the POSIT algorithm, the method demonstrates competitive accuracy, especially under realistic conditions where landmark detection may introduce errors.

Evaluation on PIE Database

To demonstrate practical applicability, the method was evaluated on the PIE database. The accuracy on unseen illumination conditions suggests good generalization (Figure 4).

Figure 4: Shape and pose fitting evaluation on PIE database, reflecting generalization across varying conditions.

Runtime and Performance

The approach requires approximately 200 milliseconds per image, highlighting its potential for real-time applications. The use of local features and cascaded regression allows for rapid convergence, which is crucial for performance-critical applications like real-time animation or virtual reality.

Conclusion

The paper presents a significant advancement in 3DMM fitting by harnessing local features through a cascaded regression framework. This combination yields not only robustness to varying imaging conditions but also speed suitable for real-time applications. Future research can expand on the integration of additional model parameters like albedo and explore further unification of landmark detection within this framework, enhancing robustness and applicability in diverse environments.