PALP: Prompt Aligned Personalization of Text-to-Image Models (2401.06105v1)

Abstract: Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfiLLMent of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a \emph{single} prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.

Understanding PALP: Personalizing AI-Generated Images

Introduction to Personalized Images

Artificial intelligence has made significant strides in generating creative and diverse images from textual descriptions. Text-to-image models, such as "a sketch of Paris on a rainy day," can produce a wide range of image settings and styles. However, incorporating specific personal features, like a particular subject, style, or ambiance into these images while maintaining prompt alignment, is a challenge for these models. This paper introduces a novel technique aimed at enhancing personalization without sacrificing the adherence to intricate textual prompts, known as prompt-aligned personalization.

The Challenge of Personalization and Prompt Alignment

Pre-trained text-to-image models offer shape-shifting capabilities, transforming text prompts into vivid images. But striking a balance between retaining the unique attributes of personalized subjects and remaining true to the intricacies of the prompt has been problematic. The introduction of an additional score distillation sampling term establishes a method that improves image generation aligned with complex prompts. This is particularly beneficial when content creators seek detailed personalization within a specific context, such as a "sketch of a beloved pet in the style of Van Gogh."

Methodology Behind Prompt-Aligned Personalization

The innovative approach, termed Prompt Aligned Personalization of Text-to-Image Models or PALP, keeps the personalized model closely tied to the target prompt through training. It leverages the existing knowledge within pre-trained models and uses it as a scaffold to introduce personal subjects without losing the essence of the prompt. This is achieved by optimizing two components concurrently: personalization, which introduces the subject, and prompt alignment, which ensures the image resonates with the target prompt. Results displayed in the paper illustrate that PALP outperforms other methods, offering creatives the freedom to generate personalized images with high fidelity to both the subject and prompt.

Potential and Applications

PALP extends the capabilities of text-to-image models, proving effective in both multi-shot and single-shot settings. This means it can personalize images with one or several reference images. PALP's versatility shows through its adeptness at composing images with multiple personal subjects, drawing from single artworks for inspiration, or aligning with complex, layered prompts. The research findings point toward a future where AI-driven image creation can cater more precisely to detailed and unique user prompts, making personalized digital art more accessible and aligned with the creator's vision.

Conclusively, this methodology offers a nuanced path to personalized content creation, blending the specificity of individual elements with the broad knowledge of pre-trained models. Content creators can now look forward to utilizing AI that better understands intricate prompts, marrying personalized features with styles, places, and the ambiance of their choosing, opening up a new avenue of digital creativity.