EleGANt: Exquisite and Locally Editable GAN for Makeup Transfer (2207.09840v1)

Abstract: Most existing methods view makeup transfer as transferring color distributions of different facial regions and ignore details such as eye shadows and blushes. Besides, they only achieve controllable transfer within predefined fixed regions. This paper emphasizes the transfer of makeup details and steps towards more flexible controls. To this end, we propose Exquisite and locally editable GAN for makeup transfer (EleGANt). It encodes facial attributes into pyramidal feature maps to preserves high-frequency information. It uses attention to extract makeup features from the reference and adapt them to the source face, and we introduce a novel Sow-Attention Module that applies attention within shifted overlapped windows to reduce the computational cost. Moreover, EleGANt is the first to achieve customized local editing within arbitrary areas by corresponding editing on the feature maps. Extensive experiments demonstrate that EleGANt generates realistic makeup faces with exquisite details and achieves state-of-the-art performance. The code is available at https://github.com/Chenyu-Yang-2000/EleGANt.

• The paper introduces EleGANt, a GAN framework that enhances makeup transfer by preserving intricate details and enabling precise local editing.
• It employs a novel Sow-Attention Module that efficiently captures pixel-level makeup characteristics through shifted overlapped window techniques.
• Extensive comparisons demonstrate EleGANt’s superior performance in maintaining spatial alignment and fine-grained makeup nuances over traditional methods.

EleGANt: Exquisite and Locally Editable GAN for Makeup Transfer

This paper introduces EleGANt, a Generative Adversarial Network (GAN) tailored for makeup transfer that emphasizes detail preservation and local editing capabilities. Traditional approaches to makeup transfer often rely heavily on color distribution transfer, resulting in a loss of high-frequency details like eye shadows and blush features. EleGANt aims to address these shortcomings by incorporating a multi-faceted approach, which includes high-resolution feature extraction and novel attention mechanisms, specifically the Sow-Attention Module.

The paper's primary focus is the preservation of fine-grained makeup details while providing flexible and interactive control over the transfer process. EleGANt does so by encoding facial attributes into pyramidal feature maps that maintain both low and high-frequency information. This approach ensures that the makeup transfer retains intricate details and aligns spatially with the source face. The framework integrates an advanced attention mechanism, the Sow-Attention Module, which, through efficient computation within shifted overlapped windows, optimizes the capture of detailed makeup information.

Methodology Overview

EleGANt's architecture consists of three main components:

1. Facial Attribute Encoder (FAEnc): This module encodes facial features into multi-resolution maps, preserving the essential details necessary for realistic transform quality.
2. Makeup Transfer Module (MTM): It implements both Attention and Sow-Attention Modules for different resolution maps, efficiently mapping makeup characteristics from the reference to the target face.
3. Makeup Apply Decoder (MADec): Utilizes the makeup feature maps to integrate these into the source image, ensuring synthesized realism while maintaining original facial identity features.

The attention mechanisms within EleGANt allow pixel-level correspondence adjustments, accommodating facial pose and expression misalignments between reference and source images. This is a notable enhancement compared to prior models that often struggled with spatial discrepancies or relied overly on histogram matching which neglects spatial makeup traits.

Results and Contributions

The paper presents extensive comparisons between EleGANt and prior makeup transfer methodologies, demonstrating superior visual quality and detail continuity in transferred makeup attributes. Key contributions of the EleGANt architecture include:

• Customizability: For the first time in makeup transfer models, this paper showcases customizable local editing capabilities within any arbitrary regions of the face. This is realized through fine-grain control over makeup feature maps.
• Novel Attention Module: The Sow-Attention Module reduces computational overhead through efficient attention within local, shifted, and overlapped windows, ensuring spatial coherence in output while being computationally feasible for high-resolution data.
• High-Resolution Mapping: By leveraging high-resolution feature maps, EleGANt adeptly manages to transfer makeup details that other models overlook, particularly in scenarios involving high complexity and spatial makeup nuances.

Implications and Future Work

The development of EleGANt has significant implications for AI applications in cosmetics, offering potential enhancements in virtual try-on systems and marketing tools that require precise customization and realistic visual outputs. This breakthrough signifies a step ahead in handling spatial details and achieving nuanced, user-specific makeup styles.

Future research on EleGANt could expand its robustness to accommodate extreme makeup styles, which are currently beyond the dataset's diversity. Additionally, improving the efficiency of local editing and exploring other generative tasks where Sow-Attention could be beneficial, like detailed texture synthesis in other non-makeup contexts, remains an open area for exploration.

In summary, the EleGANt model marks a notable advancement in the field of facial attribute transfer, fostering enhanced realism and control which, until now, has been unfeasible in existing frameworks.