RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance (2405.14677v4)

Abstract: Customizing diffusion models to generate identity-preserving images from user-provided reference images is an intriguing new problem. The prevalent approaches typically require training on extensive domain-specific images to achieve identity preservation, which lacks flexibility across different use cases. To address this issue, we exploit classifier guidance, a training-free technique that steers diffusion models using an existing classifier, for personalized image generation. Our study shows that based on a recent rectified flow framework, the major limitation of vanilla classifier guidance in requiring a special classifier can be resolved with a simple fixed-point solution, allowing flexible personalization with off-the-shelf image discriminators. Moreover, its solving procedure proves to be stable when anchored to a reference flow trajectory, with a convergence guarantee. The derived method is implemented on rectified flow with different off-the-shelf image discriminators, delivering advantageous personalization results for human faces, live subjects, and certain objects. Code is available at https://github.com/feifeiobama/RectifID.

• The paper introduces a training-free method using rectified flow to preserve user-specific identity in image generation.
• It redefines classifier guidance as a fixed-point problem by anchoring the diffusion trajectory for enhanced stability and convergence.
• The approach achieves state-of-the-art results in face-centric tasks with competitive identity similarity scores and rapid generation times.

Personalizing Diffusion Models for Identity-Preserving Image Generation

Introduction

This article explores a paper that seeks to make diffusion models more adaptable for creating images that preserve the identity of a reference image provided by the user. Traditional approaches require a lot of training on domain-specific images, which isn't very flexible. To tackle this challenge, the paper utilizes classifier guidance, a training-free technique, in combination with a nifty framework called rectified flow. The solution offers a flexible and stable approach to personalizing image generation while maintaining identity consistency.

Key Concepts

Diffusion Models and Personalization

Diffusion models have steadily gained traction in generating personalized images by integrating user-specific subjects into the generated content. However, traditional methods, including Textual Inversion and DreamBooth, often grapple with high computational costs and limitations in preserving identity accurately. Training-free methods are now emerging but still have limitations, primarily due to their reliance on pre-trained, domain-specific data.

Classifier Guidance

Classifier guidance works by modifying the output of a diffusion model using gradients from an existing pre-trained classifier. This pre-trained classifier can offer domain knowledge without additional, expensive training. The catch? Traditional classifier guidance requires a "noise-aware" classifier trained on noised inputs, limiting its practical utility.

Methodology

Rectified Flow for Classifier Guidance

Building on the rectified flow framework, this paper redefines classifier guidance as a fixed-point problem, handling only the trajectory endpoints of the diffusion process. This simplifies the application of classifier guidance and obviates the need for a special noise-aware classifier.

Anchored Classifier Guidance

To stabilize the trajectory in the diffusion process, the classifier-guided flow trajectory is anchored to a reference trajectory. This not only stabilizes the solving procedure but also guarantees convergence, thanks to the mathematical properties derived in the research.

Piecewise Linear Implementation

The paper also extends its methodologies to piecewise rectified flow, acknowledging that real-world rectified flows are not perfectly straight. The proposed algorithm can handle these practical scenarios effectively, maintaining flexibility across various tasks.

Practical Implications

This paper is particularly insightful for tasks like:

1. Face-Centric Personalization:
   • Achieves state-of-the-art results in identity preservation and prompt consistency.
   • Outperforms several baseline methods, including IP-Adapter and InstantID, even when operating in a training-free manner.
2. Subject-Driven Generation:
   • Extends capabilities to animals and objects.
   • Competes closely with models like Emu2, despite being training-free.
3. Multi-Subject Personalization:
   • Adapts seamlessly to scenarios involving multiple subjects by using a bipartite matching step.

Numerical Results

• Face-Centric Tasks: The method scored highly on identity similarity (0.5930) and showed competitive prompt consistency, outperforming both earlier and recent state-of-the-art methods.
• Computation Efficiency: Achieving results within 9 to 46 seconds, it offers a practical solution without the need for large datasets and extensive training.

Future Directions

Research could further focus on:

• Model Improvements: Optimizing rectified flow models for faster and more accurate generation.
• Expanding Domain Flexibility: Exploring how this method can be applied to broader domains, such as complex scenes or more intricate identities.

Conclusion

This paper introduces a flexible and effective training-free approach for personalized image generation using rectified flow and classifier guidance. It bridges the performance gap with training-based personalization methods, showcasing a promising direction for future AI advancements in identity-preserving image generation.