ViPer: Visual Personalization of Generative Models via Individual Preference Learning (2407.17365v1)

Abstract: Different users find different images generated for the same prompt desirable. This gives rise to personalized image generation which involves creating images aligned with an individual's visual preference. Current generative models are, however, unpersonalized, as they are tuned to produce outputs that appeal to a broad audience. Using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable. We propose to personalize the image generation process by first capturing the generic preferences of the user in a one-time process by inviting them to comment on a small selection of images, explaining why they like or dislike each. Based on these comments, we infer a user's structured liked and disliked visual attributes, i.e., their visual preference, using a LLM. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference. Through a series of user studies and LLM guided evaluations, we demonstrate that the proposed method results in generations that are well aligned with individual users' visual preferences.

ViPer: Visual Personalization of Generative Models via Individual Preference Learning

The paper "ViPer: Visual Personalization of Generative Models via Individual Preference Learning" introduces a novel framework, ViPer, for personalizing the output of generative models based on individual users' visual preferences. This approach addresses the limitation of current generative models, which tend to produce outputs appealing to a broad audience and require inefficient, iterative, manual prompt engineering to cater to individual preferences.

Methodology

The proposed method entails capturing a user's generic preferences through a one-time process where users comment on a small, diverse set of images. Free-form comments allow users to articulate why they like or dislike certain images. These comments are then processed by a Visual Preference Extractor (VPE), a fine-tuned IDEFICS2-8b model, to infer a structured representation of the user’s visual preferences, which includes liked and disliked visual attributes.

To generate personalized images, the system combines the user's preference embeddings with a text-to-image model, specifically Stable Diffusion. This integration is accomplished by modifying the embedding of the input prompt and incorporating the visual preferences into the denoising process. The core mechanism involves adjusting the predicted noise during the denoising steps to steer the output toward the individual's preferences without requiring additional fine-tuning of the generative model.

Key Contributions

1. Free-Form Comment Analysis: The use of comments for capturing user preferences allows for a richer and more nuanced understanding compared to methods relying on binary likes/dislikes or ranking a set of images.
2. Structured Preference Representation: The paper constructs a comprehensive set of visual attributes categorized into various art features such as color palette, texture, lighting, and more. This structured representation facilitates detailed personalization.
3. Integration with Stable Diffusion: By modifying the prompt embeddings and the denoising process, ViPer seamlessly integrates user preferences with a state-of-the-art generative model without additional training overhead.
4. Proxy Evaluation Metric: The paper introduces a proxy measure fine-tuned on a dataset containing pairs of liked and disliked images. This metric can evaluate the alignment of generated images with individual preferences, offering a scalable alternative to human evaluations.
5. Flexibility and Scalability: ViPer’s approach is generalizable and can accommodate varying degrees of personalization through simple parameters, enhancing its applicability across different scenarios and user bases.

Results and Evaluation

ViPer was subjected to extensive user studies, showing a strong preference for its outputs compared to non-personalized or less targeted personalization methods like ZO-RankSGD, FABRIC, Textual Inversion, fine-tuning Stable Diffusion, and Prompt Personalization. Key findings include:

• User Studies: ViPer achieved a top-one accuracy of 86.1% when contrasting personalized versus non-personalized images, and 65.4% when comparing images personalized for the user versus other users.
• Proxy Metric Correlation: The proxy metric's results aligned closely with human evaluations, validating its efficacy as an automated evaluation tool.

Implications and Future Directions

ViPer presents significant implications both practically and theoretically. Practically, it offers a more user-friendly and efficient approach to achieving high-quality personalized image generation. Its ability to fine-tune the level of personalization dynamically makes it versatile for different applications, from art and media to advertising and personalized content creation.

Theoretically, this work contributes to the understanding of integrating LLMs with generative models for personalization tasks. It opens avenues for exploring richer, context-aware personalization mechanisms, potentially extending beyond visual preferences to other domains like music or text generation.

Future research could delve into optimizing the efficiency and scalability of the VPE, exploring alternative reward tuning strategies for stable diffusion, and expanding the attribute set to capture even more nuanced user preferences. Additionally, leveraging advanced LLMs to refine the extraction of visual preferences without predefined attributes could further enhance the flexibility and robustness of the personalization process.

In conclusion, ViPer introduces a sophisticated and user-centric method for personalized image generation, addressing significant gaps in current generative modeling techniques and laying the groundwork for future advancements in personalized AI systems.