Imagine yourself: Tuning-Free Personalized Image Generation (2409.13346v1)

Abstract: Diffusion models have demonstrated remarkable efficacy across various image-to-image tasks. In this research, we introduce Imagine yourself, a state-of-the-art model designed for personalized image generation. Unlike conventional tuning-based personalization techniques, Imagine yourself operates as a tuning-free model, enabling all users to leverage a shared framework without individualized adjustments. Moreover, previous work met challenges balancing identity preservation, following complex prompts and preserving good visual quality, resulting in models having strong copy-paste effect of the reference images. Thus, they can hardly generate images following prompts that require significant changes to the reference image, \eg, changing facial expression, head and body poses, and the diversity of the generated images is low. To address these limitations, our proposed method introduces 1) a new synthetic paired data generation mechanism to encourage image diversity, 2) a fully parallel attention architecture with three text encoders and a fully trainable vision encoder to improve the text faithfulness, and 3) a novel coarse-to-fine multi-stage finetuning methodology that gradually pushes the boundary of visual quality. Our study demonstrates that Imagine yourself surpasses the state-of-the-art personalization model, exhibiting superior capabilities in identity preservation, visual quality, and text alignment. This model establishes a robust foundation for various personalization applications. Human evaluation results validate the model's SOTA superiority across all aspects (identity preservation, text faithfulness, and visual appeal) compared to the previous personalization models.

Imagine yourself: Tuning-Free Personalized Image Generation

The paper "Imagine yourself: Tuning-Free Personalized Image Generation" presents a significant advancement in personalized image generation utilizing diffusion models. The key contribution of this research is the proposal of a tuning-free model called "Imagine yourself," which allows for the customization of image generation without the need for individualized tuning processes specific to each user. The model notably addresses persistent challenges in previous personalization methods, such as overfitting and the inability to generate diverse images from complex prompts, and maintains three principal objectives: identity preservation, visual fidelity, and prompt alignment.

Key Contributions:

1. Synthetic Paired Data Generation: The paper introduces a synthetic paired data generation mechanism aimed at diversifying the generated images. Traditional models often suffer from a "copy-paste" effect, leading to poor performance in adhering to complex prompts. By employing a synthetic data generation technique that produces paired datasets with varied expressions, poses, and lighting conditions, "Imagine yourself" mitigates this issue. The data generation pipeline involves multi-modal LLM-based captioning, LLM rewriting, and high-quality synthesis using text-to-image models, refined to match identity features of reference images.
2. Fully Parallel Attention Architecture: The proposed model features an innovative architecture with three text encoders (CLIP, UL2, and ByT5) and a fully trainable vision encoder. This setup improves text faithfulness and balances vision and text control more effectively than traditional concatenation methods. Additionally, the vision encoder, initialized with zero_conv to prevent noisy control signals, extracts identity information which is processed via parallel cross-attention with the text signals.
3. Multi-Stage Finetuning Methodology: A coarse-to-fine multi-stage finetune approach progressively enhances visual quality. This methodology involves pretraining on large-scale datasets followed by fine-tuning with real and synthetic high-quality datasets. The paper highlights how training with real images enhances identity preservation, while synthetic images improve prompt alignment. An interleaved training process optimizes the balance between identity fidelity and the ability to follow complex prompts.

Quantitative and Qualitative Results:

Extensive evaluation showcases the superiority of "Imagine yourself" over state-of-the-art (SOTA) personalization models. Human annotations across thousands of test examples indicate significant improvements. Specifically, the model achieves a +27.8% enhancement in text alignment on complex prompts. A comparison table (Table \ref{tab:hev_h2h_x2}) highlights the following metrics:

• Prompt Alignment: 46.3% (win rate) compared to 1.2% for the SOTA control-based model and 32.4% for the SOTA adapter-based model.
• Identity Preservation: 81.7% tie, 3.2% (win rate) for SOTA control-based model, 5.5% for SOTA adapter-based model.
• Visual Appeal: 31.6% (win rate) over SOTA control-based model, 4.2% (win rate) over SOTA adapter-based model, with dominant tie rates indicating overall higher visual quality.

Ablation Study:

The ablation studies confirm the effectiveness of each component. For instance:

• Removing multi-stage finetuning drops prompt alignment by 25.5% and visual appeal by 42.0%.
• Eliminating the fully parallel attention architecture reduces all metrics, notably visual appeal by 22.0%.
• Omitting synthetic paired data impacts prompt alignment negatively, reinforcing its importance for complex prompt adherence.

Implications and Future Directions:

The proposed model facilitates significant practical applications in personalized content creation without the latency and cost associated with individualized tuning processes. The use of a shared, tuning-free model makes practical deployment more feasible in various personalization contexts, from entertainment to digital marketing.

For future developments, the research suggests two primary directions:

• Extending the personalized generation from images to videos, ensuring temporal coherence in identity and visual quality.
• Enhancing the model's ability to adhere to even more complex and dynamic prompts, pushing the boundaries of generative models' creative capabilities.

In conclusion, "Imagine yourself" presents robust advancements in tuning-free personalized image generation, surpassing SOTA models in critical metrics and offering a compelling framework for future research and application in AI-driven personalization.