Nested Attention: Semantic-aware Attention Values for Concept Personalization (2501.01407v1)

Abstract: Personalizing text-to-image models to generate images of specific subjects across diverse scenes and styles is a rapidly advancing field. Current approaches often face challenges in maintaining a balance between identity preservation and alignment with the input text prompt. Some methods rely on a single textual token to represent a subject, which limits expressiveness, while others employ richer representations but disrupt the model's prior, diminishing prompt alignment. In this work, we introduce Nested Attention, a novel mechanism that injects a rich and expressive image representation into the model's existing cross-attention layers. Our key idea is to generate query-dependent subject values, derived from nested attention layers that learn to select relevant subject features for each region in the generated image. We integrate these nested layers into an encoder-based personalization method, and show that they enable high identity preservation while adhering to input text prompts. Our approach is general and can be trained on various domains. Additionally, its prior preservation allows us to combine multiple personalized subjects from different domains in a single image.

• The paper introduces a Nested Attention mechanism that generates query-dependent subject values to enhance identity fidelity and text alignment.
• It integrates expressive image representations into cross-attention layers to effectively personalize multiple subjects within single images.
• Experimental results demonstrate superior performance over traditional methods, achieving higher fidelity and editability across diverse domains.

Nested Attention: Semantic-aware Attention Values for Concept Personalization

The paper introduces a novel mechanism called Nested Attention, designed to enhance text-to-image personalization tasks by improving the expressiveness and identity preservation of specific subjects within generated images. This research addresses key challenges in the personalization of text-to-image models, specifically the balance between maintaining identity fidelity and aligning with text prompts.

Overview

Traditional text-to-image models often struggle with the dichotomy of identity preservation and prompt adherence. Previous methods tend to use either single textual tokens, which are limited in expressiveness, or richer representations that disrupt the model's learned prior, thus diminishing prompt alignment. Nested Attention, the novel mechanism proposed by this work, circumvents these issues by integrating an expressive image representation into the model's existing cross-attention layers without overwhelming the learned prior.

Key Contributions

1. Nested Attention Mechanism: This mechanism generates query-dependent subject values through nested attention layers. These layers learn to select relevant subject features for each region in a generated image, thus enabling high identity preservation while adhering to input text prompts.
2. Combined Personalization: The approach allows the synthesis of images featuring multiple personalized subjects from different domains within a single frame, maintaining coherence with the text prompt.
3. General Applicability: The Nested Attention is generalizable and does not depend on specialized datasets with repeated identities, demonstrating efficacy on both human faces and non-human domains such as pets.
4. Prior Preservation: It leverages the model's prior by tying the personalized concept to a single text token, allowing better trade-offs and improved disentanglement of representations.

Results and Implications

The paper presents experimental results exhibiting the superior performance of Nested Attention over existing methods in both identity preservation and text adherence. It quantitatively outperforms common subject-injection methods like decoupled cross-attention in scenarios demanding high fidelity and editability. The approach is evaluated across various domains, showing robustness and flexibility.

Future Directions

The implications of this research extend into several areas of AI development. The ability to integrate personalized subjects while preserving the model's learned prior opens avenues for more complex multi-subject generation tasks and domain-agnostic applications. Further investigation may explore adaptations of Nested Attention to tasks beyond image generation, such as adaptive inpainting or real-time video synthesis. Additionally, integrating these concepts with other state-of-the-art personalization techniques could further augment the capabilities of automated media generation systems.

In conclusion, the Nested Attention mechanism offers a compelling solution to the longstanding problem of balancing identity fidelity and prompt alignment in text-to-image models, substantially enhancing both personalization and applicability of these systems.