Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions (2403.17064v1)

Abstract: In recent years, advances in text-to-image (T2I) diffusion models have substantially elevated the quality of their generated images. However, achieving fine-grained control over attributes remains a challenge due to the limitations of natural language prompts (such as no continuous set of intermediate descriptions existing between person'' andold person''). Even though many methods were introduced that augment the model or generation process to enable such control, methods that do not require a fixed reference image are limited to either enabling global fine-grained attribute expression control or coarse attribute expression control localized to specific subjects, not both simultaneously. We show that there exist directions in the commonly used token-level CLIP text embeddings that enable fine-grained subject-specific control of high-level attributes in text-to-image models. Based on this observation, we introduce one efficient optimization-free and one robust optimization-based method to identify these directions for specific attributes from contrastive text prompts. We demonstrate that these directions can be used to augment the prompt text input with fine-grained control over attributes of specific subjects in a compositional manner (control over multiple attributes of a single subject) without having to adapt the diffusion model. Project page: https://compvis.github.io/attribute-control. Code is available at https://github.com/CompVis/attribute-control.

Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions

The paper "Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions" addresses an ongoing challenge in the field of text-to-image (T2I) models: achieving fine-grained control over attribute expressions within generated images. Text-to-image models, built upon diffusion processes, have significantly improved in generating high-quality images, yet the capacity to control attributes of specific subjects continuously and accurately remains a difficult task. This paper presents novel approaches to tackle this challenge by exploiting directions in token-level CLIP embeddings to introduce subject-specific control in T2I models without additional costs during image generation.

Key Contributions

• Identification of Semantic Directions: The authors identify directions in the token-level CLIP text embeddings that enable fine-grained, subject-specific control over high-level attributes in T2I models. This finding is crucial as it implies that diffusion models can interpret these token-level directions meaningfully to adjust the expression of attributes in the generated images.
• Optimization-Free and Optimization-Based Methods: Two methodologies are proposed: a simple optimization-free method and a more robust optimization-based method. These methods are designed to identify the semantic directions associated with specific attributes from contrasting text prompts.
• Composable Control of Attributes: The research demonstrates that the derived semantic directions can be used to augment text prompts, allowing for compositional control over multiple attributes of a single subject. This control is achieved without modifying the diffusion model itself, reducing computational overhead.
• Transferability of Edit Directions: The paper explores the transferability of learned edit directions across various prompts and subjects, showing that learned semantic edit deltas can generalize across different contexts, which indicates the wide applicability of these techniques.

Methodology and Results

The paper details the methodology for learning semantic edit directions through token-wise modifications of CLIP embeddings. The research includes training processes involving single image-caption pairs to backpropagate semantic information and demonstrates the capability of learned directions to modulate semantic attributes like age or vehicle price.

Quantitative analysis in the experiments section shows the comparative advantage of learned edit directions over traditional CLIP embedding differences, underscoring their robustness in maintaining image integrity while modifying attribute expressions. Moreover, the delayed sampling technique helps reduce global correlations, thereby enhancing subject-specific control.

Implications and Further Applications

This work significantly enhances the control and precision of T2I models, which is critical for various applications, from creative industries to assistive technologies where specific visual attributes must be consistently manipulated. The ability to exert fine-grained control over image attributes without extensive computational costs or model retraining represents a substantial advancement in the field.

Future directions could include integrating these methodologies with complementary strategies to further reduce cross-subject attribute entanglement. Additionally, expanding the framework's adaptability across various model architectures could enhance its versatility and effectiveness across broader applications.

Conclusion

Overall, this research provides substantial insights into the manipulation of semantic attributes in text-to-image models, contributing a sophisticated method to identify and utilize semantic directions within CLIP embeddings. The advancements in subject-specific attribute control presented in this paper could serve as a foundation for future work aiming to refine and expand the capabilities of text-to-image generation frameworks in the AI community.