Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis (2411.07132v1)

Abstract: Although text-to-image (T2I) models exhibit remarkable generation capabilities, they frequently fail to accurately bind semantically related objects or attributes in the input prompts; a challenge termed semantic binding. Previous approaches either involve intensive fine-tuning of the entire T2I model or require users or LLMs to specify generation layouts, adding complexity. In this paper, we define semantic binding as the task of associating a given object with its attribute, termed attribute binding, or linking it to other related sub-objects, referred to as object binding. We introduce a novel method called Token Merging (ToMe), which enhances semantic binding by aggregating relevant tokens into a single composite token. This ensures that the object, its attributes and sub-objects all share the same cross-attention map. Additionally, to address potential confusion among main objects with complex textual prompts, we propose end token substitution as a complementary strategy. To further refine our approach in the initial stages of T2I generation, where layouts are determined, we incorporate two auxiliary losses, an entropy loss and a semantic binding loss, to iteratively update the composite token to improve the generation integrity. We conducted extensive experiments to validate the effectiveness of ToMe, comparing it against various existing methods on the T2I-CompBench and our proposed GPT-4o object binding benchmark. Our method is particularly effective in complex scenarios that involve multiple objects and attributes, which previous methods often fail to address. The code will be publicly available at \url{https://github.com/hutaihang/ToMe}.

• The paper introduces Token Merging, a novel training-free method that aggregates relevant tokens to ensure correct semantic binding between objects and their attributes or sub-objects in text-to-image synthesis.
• Empirical evaluation on T2I-CompBench and a GPT-4o benchmark demonstrated that the Token Merging method outperforms state-of-the-art techniques, particularly excelling in complex scenarios with multiple objects and attributes.
• Practically, this training-free approach reduces computational overhead for developing and deploying T2I models, simplifying the process by removing the need for extensive re-training or complex user specifications.

Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

The research paper titled "Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis" addresses a critical challenge within the domain of text-to-image (T2I) models, specifically the difficulty these models face in establishing correct semantic bindings as per the input prompts. Despite the impressive image generation capabilities exhibited by T2I models, aligning generated images accurately with the semantic details in text prompts, termed semantic binding, remains a challenge. The proposed methodology introduces an innovative approach that eschews the need for complex fine-tuning processes or the intervention of LLMs.

Overview of Methodology

Semantic binding is defined in this research as ensuring a given object in a prompt is correctly associated with its attributes (attribute binding) or its related sub-objects (object binding). The novel approach put forward by the authors—termed Token Merging—aggregates relevant tokens into a single, composite token. This crucial step ensures that the object along with its attributes and sub-objects adhere to the same cross-attention maps during the generation process. To address potential ambiguities that arise with complex textual prompts featuring multiple main objects, an end token substitution strategy is suggested as a complementary measure.

In the initial phases of T2I synthesis, where image layouts are established, the introduction of auxiliary entropy and semantic binding losses plays a pivotal role. These losses guide the iterative update of the composite token towards achieving enhanced generation integrity. Consequently, semantic alignment is fortified without necessitating extensive re-training or reliance on supplementary layout information.

Empirical Results and Evaluation

The efficacy of the proposed Token Merging method was demonstrated through rigorous evaluation on benchmarks such as T2I-CompBench and a novel GPT-4o object binding benchmark. These experiments reveal that the method excels particularly in complex scenarios involving numerous objects and attributes, outperforming several state-of-the-art techniques. The numerical results underline the robustness of the approach in practical settings, where semantic coherence is crucial for reliable T2I synthesis.

Implications and Future Directions

Practically, this research has significant implications for the development and deployment of T2I models. The training-free nature of the proposed approach reduces the computational overhead and eliminates the need for exhaustive datasets or intricate user specifications. Moreover, the integration of auxiliary losses to refine semantic precision at initial synthesis stages offers an avenue for future improvements in model robustness and adaptability.

Theoretically, this work provides insights into the token representation and binding mechanisms that can be leveraged more broadly within AI-driven content generation systems. The semantic additivity property of text embeddings as explored in the paper could find applications beyond T2I synthesis, suggesting potential future research directions in multimodal AI and cross-domain semantic understanding.

In conclusion, the Token Merging approach delineated in this paper exemplifies a strategic advancement in T2I synthesis by fostering semantic alignment through efficient and innovative use of token representations. This contribution not only refines current methodologies but also sets the stage for further explorations into enhancing AI model coherence and user alignment in generative applications.