Customizing Text-to-Image Models with a Single Image Pair (2405.01536v2)

Abstract: Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We ask if such an image pair can be used to customize a generative model to capture the demonstrated stylistic difference. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process. Unlike existing methods that learn to mimic a single concept from a collection of images, our method captures the stylistic difference between paired images. This allows us to apply a stylistic change without overfitting to the specific image content in the examples. To address this new task, we employ a joint optimization method that explicitly separates the style and content into distinct LoRA weight spaces. We optimize these style and content weights to reproduce the style and content images while encouraging their orthogonality. During inference, we modify the diffusion process via a new style guidance based on our learned weights. Both qualitative and quantitative experiments show that our method can effectively learn style while avoiding overfitting to image content, highlighting the potential of modeling such stylistic differences from a single image pair.

• The paper introduces Pair, a method that customizes text-to-image models by learning artistic styles from just one image pair.
• It employs dual parameter optimization with LoRA and orthogonality constraints to effectively disentangle style from content.
• The approach minimizes overfitting and data requirements, paving the way for efficient personalized generative art creation.

Customizing Text-to-Image Models with a Single Image Pair

Introduction

Meet Pair, a novel method designed to customize generative models by learning artistic styles from just a single pair of style-content images. Unlike traditional approaches that might need a library of examples to grasp a style, Pair hones in on the style differences exhibited between two related images. This is particularly useful in scenarios where the objective is to apply a distinct stylistic flair to various inputs without losing the essence of the original content.

Concept Breakdown

Generative models like the ones used for creating text-to-image translations are fantastic tools for artists and designers, letting them transform ideas into visual art. However, these models often require extensive training data to learn specific styles, which isn't always feasible. The innovation in Pair lies in its ability to extract and apply stylistic nuances effectively from a minimal dataset — in this case, a single image pair consisting of a style image and a content image.

Key Challenges and Solutions

• Overfitting: A common pitfall when training on limited data is that the model might overlearn from those few examples, failing to generalize the style beyond the exact images seen during training. To mitigate this, Pair uses a dual approach that separates the learning of style and content, preventing the model from conflating the two.
• Style-Content Disentanglement: To further refine the model's ability to distinguish between the content's structure and the image's style, Pair employs a technique that optimizes separate parameters for style and content through a process called Low-Rank Adapters (LoRA), as well as enforcing an orthogonality constraint between these parameters.

How Pair Works

The setup involves a pre-trained generative model which undergoes customization with the help of two sets of parameters: one for the style and one for the content. During training:

1. Content Learning: The model first learns to recreate the content image using a content-specific text prompt complemented by a unique identifier.
2. Style Application: The style parameters are then adjusted to apply the learned style onto the content image, guided by a combined prompt indicating both content and desired style.

Inference involves modifying the model's typical output pathway by integrating a new component called style guidance. This component is crucial for adjusting the intensity of the applied style during generation, allowing for more precise control over the final image's appearance.

Practical Applications and Theoretical Implications

This method opens up new possibilities for personalizing generative models in practical applications like digital art creation, where artists can quickly establish and apply new styles across various works. Theoretically, the research advances our understanding of style-content disentanglement in image generation, contributing insights into more efficient ways to adapt generative models with sparse data.

Future Outlook

One exciting direction for future research could be exploring how Pair might perform with different types of content beyond images, such as video frames. Additionally, enhancing the robustness of the style guidance mechanism could allow even finer control over the stylization process, potentially leading to more personalized and varied artistic expressions.

Conclusion

Pair demonstrates a significant step forward in the customization of generative models using minimal data. By effectively learning from just a single image pair, it significantly reduces the data requirements typically associated with training these models, all while preserving the original content's structure and applying the learned style accurately across varied inputs. This capability not only makes it a powerful tool for artists and designers but also marks an important advancement in the field of generative modeling.