Lipstick ain't enough: Beyond Color Matching for In-the-Wild Makeup Transfer (2104.01867v1)

Abstract: Makeup transfer is the task of applying on a source face the makeup style from a reference image. Real-life makeups are diverse and wild, which cover not only color-changing but also patterns, such as stickers, blushes, and jewelries. However, existing works overlooked the latter components and confined makeup transfer to color manipulation, focusing only on light makeup styles. In this work, we propose a holistic makeup transfer framework that can handle all the mentioned makeup components. It consists of an improved color transfer branch and a novel pattern transfer branch to learn all makeup properties, including color, shape, texture, and location. To train and evaluate such a system, we also introduce new makeup datasets for real and synthetic extreme makeup. Experimental results show that our framework achieves the state of the art performance on both light and extreme makeup styles. Code is available at https://github.com/VinAIResearch/CPM.

• The paper introduces a dual-branch framework combining a UV-based Color Transfer Branch with a segmentation-driven Pattern Transfer Branch.
• The paper leverages novel real and synthetic makeup datasets to achieve superior performance, evidenced by improved mIoU and MS-SSIM scores.
• The paper’s method enhances virtual makeup applications by enabling seamless and realistic transfer of complex makeup styles in varied scenarios.

Lipstick Ain't Enough: Beyond Color Matching for In-the-Wild Makeup Transfer

The paper "Lipstick Ain’t Enough: Beyond Color Matching for In-the-Wild Makeup Transfer" provides a comprehensive exploration into the domain of makeup transfer. This research extends beyond traditional color manipulation techniques by integrating both color and pattern aspects of makeup, thus enhancing the realism and applicability of makeup transfer methods in varied real-world scenarios.

Methodological Advancements

The key contribution of this work lies in its holistic framework, which introduces two parallel branches: the Color Transfer Branch and the Pattern Transfer Branch. The Color Transfer Branch enhances traditional approaches by employing a UV space conversion, which aligns facial features across different head poses and expressions, significantly improving histogram matching and the overall color-synthesis process. This branch adopts a CycleGAN-like architecture and uses a histogram-based loss to facilitate more precise color matching across regions such as eyes, lips, and skin.

In contrast, the Pattern Transfer Branch focuses on identifying and transferring intricate makeup patterns such as stickers and facial drawings. This task is handled via a supervised segmentation network, leveraging novel datasets for effective training. The UV mapping ensures the pattern is aligned with the facial surface seamlessly, allowing for an accurate transplantation without deformation artifacts.

Dataset Contributions and Results

The authors introduced several datasets to facilitate their research, particularly the CPM-Real, CPM-Synt-1, and CPM-Synt-2 datasets. These datasets encompass a broad spectrum of makeup styles, ranging from conventional styles to extreme and pattern-based makeups. By utilizing both real and synthetic images, the framework demonstrates its robustness and versatility in handling diverse makeup styles.

Experimental results indicate that this framework achieves superior performance on both light and extreme makeup styles. The framework outperforms existing methods by a wide margin, which is evident from the substantial improvements in mIoU for pattern segmentation and MS-SSIM for makeup transfer quality.

Implications and Future Directions

The implications of this research are manifold. Practically, the ability to transfer complex makeup styles extends the applicability of virtual makeup applications in retail and entertainment industries. Theoretically, this work broadens the scope of image-to-image translation tasks, emphasizing the integration of spatial transformations with texture synthesis in visual computing.

Future developments could explore the integration of this framework with real-time applications, broadening its adaptability across various user interfaces. Additionally, there is potential for this approach to inform related fields, such as augmented reality and digital fashion, where visual realism is paramount.

In conclusion, by addressing both color transformation and pattern addition, this research constructs a more complete makeup transfer solution, setting a new benchmark in the field and paving the way for further innovations in the field of computer vision and aesthetic computing.