ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving (2404.16771v1)

Abstract: Diffusion-based technologies have made significant strides, particularly in personalized and customized facialgeneration. However, existing methods face challenges in achieving high-fidelity and detailed identity (ID)consistency, primarily due to insufficient fine-grained control over facial areas and the lack of a comprehensive strategy for ID preservation by fully considering intricate facial details and the overall face. To address these limitations, we introduce ConsistentID, an innovative method crafted for diverseidentity-preserving portrait generation under fine-grained multimodal facial prompts, utilizing only a single reference image. ConsistentID comprises two key components: a multimodal facial prompt generator that combines facial features, corresponding facial descriptions and the overall facial context to enhance precision in facial details, and an ID-preservation network optimized through the facial attention localization strategy, aimed at preserving ID consistency in facial regions. Together, these components significantly enhance the accuracy of ID preservation by introducing fine-grained multimodal ID information from facial regions. To facilitate training of ConsistentID, we present a fine-grained portrait dataset, FGID, with over 500,000 facial images, offering greater diversity and comprehensiveness than existing public facial datasets. % such as LAION-Face, CelebA, FFHQ, and SFHQ. Experimental results substantiate that our ConsistentID achieves exceptional precision and diversity in personalized facial generation, surpassing existing methods in the MyStyle dataset. Furthermore, while ConsistentID introduces more multimodal ID information, it maintains a fast inference speed during generation.

• The paper introduces a novel multimodal framework that enhances identity preservation in portrait generation using a single reference image.
• It employs a Multimodal Facial Prompt Generator and an ID-Preservation Network to integrate text and image features for precise facial detail retention.
• Experiments on the FGID dataset show significant improvements in fidelity and diversity over conventional portrait generation methods.

Comprehensive Analysis of ConsistentID: Multimodal Fine-Grained Identity Preservation in Portrait Generation

Introduction to ConsistentID

ConsistentID introduces a sophisticated approach targeted at addressing critical challenges in high-fidelity, personalized portrait generation using a single reference image. The main obstacle in this domain has been the accurate preservation of identity features across variations introduced via textual prompts. Existing solutions often compromise either identity fidelity or details in facial features, unable to ensure both concurrently.

The core contribution of ConsistentID is its capability to generate portraits that strongly preserve the subject's identity even when altered according to detailed multimodal prompts. This preservation is achieved through two primary mechanisms within the system:

1. Multimodal Facial Prompt Generator: Integrates detailed textual descriptions of facial features with corresponding images to refine identity preservation at a granular level.
2. ID-Preservation Network: Optimizes identity consistency across facial regions using a specialized facial attention localization strategy, which carefully manages the preservation of unique identity traits within specific facial zones.

Dataset Contribution: FGID

A significant hindrance in this research area has been the lack of appropriate datasets capable of training models to respect fine-grained identity nuances. ConsistentID's proposition includes the creation of the Fine-Grained ID Preservation (FGID) dataset, which comprises over half a million images annotated with detailed identity markers. This dataset is instrumental in training models to focus on minute identity details that standard datasets tend to overlook.

Methodology and Technical Enhancements

Technical Composition

ConsistentID's approach involves two significant facets:

• The Multimodal Facial Prompt Generator handles the input consisting of a single facial image, combining this with region-specific facial descriptions to generate comprehensive multimodal identity details. This step uses advanced techniques to fuse text and image features, ensuring a detailed representation of facial identity factors.
• The ID-Preservation Network builds upon the generated detailed identity prompts to produce facial images. It employs a facial attention localization strategy, which anchors the generation process to respect identity features specific to defined facial areas like the eyes, nose, or mouth.

Performance and Evaluation

Experiments conducted using the FGID dataset demonstrate that ConsistentID substantially outperforms contemporary methods in generating personalized portraits with high fidelity. Metrics such as precision in identity preservation and generation diversity mark significant improvements over existing methods.

Implications and Future Work

The introduction of ConsistentID opens several pathways for both practical applications and future academic endeavors:

• Practical Applications: In areas like digital entertainment, ecommerce, and personalized avatars, ensuring high fidelity in identity features while accommodating user-specific modifications can significantly enhance user experience.
• Future Research Directions: The challenge of integrating increasingly granular details without compromising generation speed or scalability remains. Future models could explore deeper integrations with emerging AI technologies or more sophisticated multimodal training techniques.

Conclusion

ConsistentID represents a significant stride in the confluence of identity preservation and personalized portrait generation. By judiciously integrating detailed multimodal inputs and optimizing facial region-specific identity features, this method sets a new benchmark for the fidelity and accuracy of personalized facial generation technologies.