DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (2402.09812v2)

Abstract: The objective of text-to-image (T2I) personalization is to customize a diffusion model to a user-provided reference concept, generating diverse images of the concept aligned with the target prompts. Conventional methods representing the reference concepts using unique text embeddings often fail to accurately mimic the appearance of the reference. To address this, one solution may be explicitly conditioning the reference images into the target denoising process, known as key-value replacement. However, prior works are constrained to local editing since they disrupt the structure path of the pre-trained T2I model. To overcome this, we propose a novel plug-in method, called DreamMatcher, which reformulates T2I personalization as semantic matching. Specifically, DreamMatcher replaces the target values with reference values aligned by semantic matching, while leaving the structure path unchanged to preserve the versatile capability of pre-trained T2I models for generating diverse structures. We also introduce a semantic-consistent masking strategy to isolate the personalized concept from irrelevant regions introduced by the target prompts. Compatible with existing T2I models, DreamMatcher shows significant improvements in complex scenarios. Intensive analyses demonstrate the effectiveness of our approach.

Leveraging Semantic Matching for Improved Text-to-Image Personalization in DreamMatcher

Introduction

Text-to-image (T2I) personalization has emerged as a cutting-edge domain within AI research, aiming to adapt pre-trained T2I models to generate images that match user-provided text prompts while incorporating visual cues from reference concepts. DreamMatcher introduces an innovative approach to this challenge, fundamentally rethinking T2I personalization through semantic matching. Unlike conventional methods that optimize textual embeddings or model parameters, DreamMatcher operates by aligning reference values within a diffusion model's self-attention mechanism, preserving the pre-trained model's structural integrity. This strategy enables the generation of images that not only respect the target prompt's context but also closely mirror the appearance characteristics of the reference images.

Methodology

DreamMatcher executes its objective via a dual-pronged approach: appearance matching self-attention and semantic matching guidance.

• Appearance Matching Self-Attention (AMA): At its core, DreamMatcher modifies the self-attention mechanism of a denoising U-Net to incorporate appearance features from reference images without disrupting the target's structural layout. This is achieved by retaining the target's structure path—determined by query-key similarities—and selectively integrating the reference's appearance path via semantic matching.
• Semantic Matching Guidance: Recognizing limitations in early step appearances during the diffusion process, the method further introduces a guidance technique that enriches reference attributes, ensuring fine-grained detail preservation throughout the image synthesis.

Comparative Analysis

Comparative analyses show that DreamMatcher significantly outperforms existing baselines and several state-of-the-art methods in encompassing challenging personalization scenarios. When assessed through metrics like CLIP and DINO image similarity, DreamMatcher demonstrates superior ability to capture subject appearance while adhering closely to the intentions of text prompts. Notably, the method excels in complex scenarios involving large displacements, occlusions, and novel-view synthesis, underscoring its robustness and adaptability.

Practical Implications and Future Prospects

The implications of DreamMatcher extend far beyond basic image personalization. Its adeptness at handling complex, semantic-rich personalization tasks without requiring additional fine-tuning positions it as a significant step forward in the development of more intuitive and human-like generative AI models. Looking forward, DreamMatcher sets the stage for future explorations into more nuanced and context-aware text-to-image generation tasks. Its underlying principles could also inspire advancements in related fields such as video synthesis and interactive media creation.

Conclusion

In summary, DreamMatcher marks a pivotal advancement in the field of T2I personalization, showcasing the profound impact of semantic matching on enhancing the fidelity and versatility of generated images. By elegantly balancing the preservation of target prompts with the nuanced integration of reference appearances, DreamMatcher not only enriches the toolkit available for generative AI research but also broadens the horizon for creative and practical applications of T2I technologies.