ReliableSwap: Boosting General Face Swapping Via Reliable Supervision (2306.05356v1)

Abstract: Almost all advanced face swapping approaches use reconstruction as the proxy task, i.e., supervision only exists when the target and source belong to the same person. Otherwise, lacking pixel-level supervision, these methods struggle for source identity preservation. This paper proposes to construct reliable supervision, dubbed cycle triplets, which serves as the image-level guidance when the source identity differs from the target one during training. Specifically, we use face reenactment and blending techniques to synthesize the swapped face from real images in advance, where the synthetic face preserves source identity and target attributes. However, there may be some artifacts in such a synthetic face. To avoid the potential artifacts and drive the distribution of the network output close to the natural one, we reversely take synthetic images as input while the real face as reliable supervision during the training stage of face swapping. Besides, we empirically find that the existing methods tend to lose lower-face details like face shape and mouth from the source. This paper additionally designs a FixerNet, providing discriminative embeddings of lower faces as an enhancement. Our face swapping framework, named ReliableSwap, can boost the performance of any existing face swapping network with negligible overhead. Extensive experiments demonstrate the efficacy of our ReliableSwap, especially in identity preservation. The project page is https://reliable-swap.github.io/.

• The paper introduces cycle triplets as a novel supervision strategy that significantly improves identity preservation in face swapping.
• It integrates FixerNet to capture detailed lower-face features, leading to enhanced identity alignment and measurable improvements in key metrics.
• Extensive experiments on FaceForensics++ and CelebA-HQ validate ReliableSwap's superior performance compared to existing methods.

An Evaluation of ReliableSwap: Enhancing Face Swapping Performance

The paper "ReliableSwap: Boosting General Face Swapping Via Reliable Supervision" introduces a novel framework designed to improve face swapping methodologies, which is a crucial technological function with applications in privacy protection, the film industry, and face forgery detection. The core innovation in this paper lies in the introduction of "cycle triplets," which serve as a new form of reliable supervision during face swapping training.

Key Contributions

1. Cycle Triplets for Reliable Supervision: The authors address a notable challenge within face swapping, where existing methods struggle with source identity preservation due to limited pixel-level supervision when dealing with different source and target identities. Cycle triplets, as proposed in the paper, involve using face reenactment and blending techniques to construct training pairs that provide reliable image-level guidance even when the source and target are different. By using synthetic faces as inputs and real images as supervision, this technique enables the network to be trained in a way that aligns more faithfully with natural image distributions, thus enhancing identity preservation.
2. Integration of FixerNet: The paper critiques existing identity recognition metrics for being insufficiently sensitive to lower-face details (such as mouth and jawline). To counteract this, the authors introduce FixerNet, an auxiliary network that extracts detailed embeddings specifically from the lower face. These embeddings are then incorporated into the face swapping network to enrich the identity representation and ensure the preservation of these nuanced details. This approach results in quantitatively improved performance, as demonstrated by the introduction of new metrics such as lower-face identity retrieval (L Ret.) and lower-face identity similarity (L Sim.).

Experimental Validation

Extensive experiments highlight the reliability and effectiveness of the ReliableSwap framework. Tested on datasets like FaceForensics++ and CelebA-HQ, with baselines such as FaceShifter and SimSwap, ReliableSwap consistently demonstrates superior identity preservation capabilities. This is quantitatively reflected in higher ID Ret. and L Ret. metrics, which indicate better alignment with the source identity, and is supported by qualitative assessments showcasing clear visual enhancements in identity permanence and attribute retention.

Theoretical and Practical Implications

The introduction of cycle triplets proposes a novel supervision strategy that could have broader implications beyond just face swapping. This method could potentially be adapted for other tasks requiring synthetic-natural image pairings. Moreover, by demonstrating that the lower facial region is underrepresented in typical identity assessments, the paper prompts reevaluation of standard metrics in facial recognition domains, encouraging the development of more comprehensive evaluation tools.

Future Directions

The paper opens doors for future research in more reliably synthesizing face swaps with nuanced identity features intact, potentially leading to more advanced applications in media production and digital privacy tools. Future work could involve extending the cycle triplet approach to higher resolution images or complex multi-domain face swapping scenarios.

In summary, ReliableSwap makes a substantial contribution to the field of face swapping, offering a framework that integrates reliable supervision with enhanced identity feature extraction. This work not only improves current methodologies but also lays the groundwork for future innovations that ensure the accurate and nuanced transfer of identities in digital imagery.