Style Aggregated Network for Facial Landmark Detection (1803.04108v4)

Abstract: Recent advances in facial landmark detection achieve success by learning discriminative features from rich deformation of face shapes and poses. Besides the variance of faces themselves, the intrinsic variance of image styles, e.g., grayscale vs. color images, light vs. dark, intense vs. dull, and so on, has constantly been overlooked. This issue becomes inevitable as increasing web images are collected from various sources for training neural networks. In this work, we propose a style-aggregated approach to deal with the large intrinsic variance of image styles for facial landmark detection. Our method transforms original face images to style-aggregated images by a generative adversarial module. The proposed scheme uses the style-aggregated image to maintain face images that are more robust to environmental changes. Then the original face images accompanying with style-aggregated ones play a duet to train a landmark detector which is complementary to each other. In this way, for each face, our method takes two images as input, i.e., one in its original style and the other in the aggregated style. In experiments, we observe that the large variance of image styles would degenerate the performance of facial landmark detectors. Moreover, we show the robustness of our method to the large variance of image styles by comparing to a variant of our approach, in which the generative adversarial module is removed, and no style-aggregated images are used. Our approach is demonstrated to perform well when compared with state-of-the-art algorithms on benchmark datasets AFLW and 300-W. Code is publicly available on GitHub: https://github.com/D-X-Y/SAN

• The paper introduces a Style Aggregated Network (SAN) that uses GANs to transform images and mitigate style variance in facial landmark detection.
• It combines a style-aggregated face generation module with a complementary landmark prediction module to improve detection performance.
• Experimental results on 300-W and AFLW datasets demonstrate significantly lower NME and robust performance under diverse image styles.

Style Aggregated Network for Facial Landmark Detection

The paper "Style Aggregated Network for Facial Landmark Detection" introduces a novel approach addressing the overlooked issue of image style variance in facial landmark detection. This approach, named the Style Aggregated Network (SAN), tackles the intrinsic variance found in image styles, such as grayscale versus color images or differences in lighting, which are commonplace due to diverse image sources on the internet.

Methodology

SAN aims to enhance the performance and robustness of facial landmark detectors by utilizing a generative adversarial network (GAN) to create style-aggregated images. The central idea is to transform original face images into a consistently styled format while retaining a complementary original image. The two images work in tandem to train a robust landmark detector.

The proposed framework comprises two key components:

1. Style-Aggregated Face Generation Module: This module employs GANs to transform face images into a common style, thereby addressing the disparities in image styles. By clustering images using style-discriminative features, derived from a fine-tuned ResNet-152, the process categorizes images into hidden style categories for more effective aggregation.
2. Facial Landmark Prediction Module: This component leverages the complementary nature of the original and style-aggregated images. The module's architecture, inspired by Convolutional Pose Machines, integrates information from both image streams to produce robust landmark predictions.

Experimental Analysis

The SAN's effectiveness is validated through experiments on benchmark datasets 300-W and AFLW. The network showcases superior performance in the presence of varying image styles when compared to state-of-the-art methods. Numerical results indicate a noteworthy improvement in Normalized Mean Error (NME), with SAN achieving 3.34 in the common subset of 300-W using ground truth bounding boxes.

Additional experiments employing the newly introduced 300W-Style and AFLW-Style datasets demonstrate SAN's robustness. These datasets, augmented with processed images through Adobe Photoshop, enable a controlled evaluation of style variance impact. SAN consistently outperforms variations of itself when either the style-aggregated or original image is omitted, underscoring the utility of using both streams for landmark detection.

Implications and Future Work

The ability of SAN to mitigate the effects of style variance contributes both practically and theoretically to the landscape of facial landmark detection. Practically, it offers a more robust detection framework that can be adapted to other vision tasks affected by style variance, such as object detection and person re-identification. Theoretically, SAN highlights the importance of addressing overlooked variances in model training, providing insights into how generalizable solutions can be created through careful consideration of input variability.

Future work may explore expanding the application of the style-aggregation methodology across different domains, potentially improving model robustness in a variety of computer vision tasks. Additionally, the decoupled style-aggregation technique can be further refined to generalize across unseen style domains, enhancing the adaptability and usability of detection algorithms.

Overall, the research presents a well-substantiated step forward in refining the robustness and efficacy of facial landmark detection amidst diverse data sources.