Break-A-Scene: Extracting Multiple Concepts from a Single Image (2305.16311v2)

Abstract: Text-to-image model personalization aims to introduce a user-provided concept to the model, allowing its synthesis in diverse contexts. However, current methods primarily focus on the case of learning a single concept from multiple images with variations in backgrounds and poses, and struggle when adapted to a different scenario. In this work, we introduce the task of textual scene decomposition: given a single image of a scene that may contain several concepts, we aim to extract a distinct text token for each concept, enabling fine-grained control over the generated scenes. To this end, we propose augmenting the input image with masks that indicate the presence of target concepts. These masks can be provided by the user or generated automatically by a pre-trained segmentation model. We then present a novel two-phase customization process that optimizes a set of dedicated textual embeddings (handles), as well as the model weights, striking a delicate balance between accurately capturing the concepts and avoiding overfitting. We employ a masked diffusion loss to enable handles to generate their assigned concepts, complemented by a novel loss on cross-attention maps to prevent entanglement. We also introduce union-sampling, a training strategy aimed to improve the ability of combining multiple concepts in generated images. We use several automatic metrics to quantitatively compare our method against several baselines, and further affirm the results using a user study. Finally, we showcase several applications of our method. Project page is available at: https://omriavrahami.com/break-a-scene/

• The paper presents a two-phase optimization that initially freezes weights to tune concept handles, then fine-tunes model weights to maintain distinct concept identities.
• It employs a masked diffusion loss and a novel cross-attention loss to accurately disentangle and reproduce multiple visual concepts from a single scene.
• Evaluations show marked improvements in prompt and identity similarity over methods like Textual Inversion, DreamBooth, and Custom Diffusion.

Break-A-Scene: Extracting Multiple Concepts from a Single Image

The research paper "Break-A-Scene: Extracting Multiple Concepts from a Single Image" presents a novel approach aimed at improving text-to-image (T2I) model personalization by addressing the challenge of extracting discrete visual concepts from a single image. Traditional methods predominantly focus on deriving a singular concept from numerous images that vary in context; however, they falter when tasked with isolating multiple concepts within a lone image. This task, defined by the authors as textual scene decomposition, endeavors to assign distinct text tokens that encapsulate diverse concepts identified within a solitary scene, thus enabling refined control over the generation of scenes through text prompts.

The proposed method hinges on augmenting input images with masks that distinguish the presence of selected concepts. These masks can either be user-provided or generated automatically using pre-established segmentation models. The paper introduces a two-phase customization process combining optimization of textual embeddings (handles) specifically allocated for each concept and the adaptive fine-tuning of the model weights. The first phase dedicates to freezing model weights and optimizing these handles to facilitate initial reconstructions, laying groundwork for the subsequent stage which involves subtle weight tuning to prevent overfitting. This carefully calibrated approach aims to accurately grasp individual concept identities while nurturing editability and contextual adaptability.

A pivotal aspect of their methodology is employing a masked diffusion loss designed to ensure precise reproduction of identified concepts. Diverging from existing techniques, this method also introduces a novel loss function based on cross-attention maps, ensuring disentanglement of concept-specific handles. Complementing their methodology, the authors propose a union-sampling technique to aggregate multiple concepts simultaneously, enhancing the reliability of generating complex, multi-concept imagery.

Extensive evaluations and user studies underpin the effectiveness of this approach, illustrating noticeable improvements in both prompt and identity similarity metrics compared to existing baselines such as Textual Inversion, DreamBooth, and Custom Diffusion. The quantitative and qualitative results validate that the proposed method strikingly balances fidelity to concept identity with compliance to textual prompts, placing it at the forefront in scenarios involving single-image, multi-concept extractions.

The implications of this paper are multifaceted. Practically, it opens avenues for more versatile applications within creative industries by enabling elaborate image transformations. Theoretically, it encourages further research into scene decomposition and concept disentanglement, potentially influencing methodologies in automated training without necessitating large databases.

Given these advancements, the future may reveal further enhancements in model efficiency and speed, alongside addressing current limitations, such as light sensitivity overfitting and pose fixation. This research underscores a significant stride in refining T2I personalization, providing a foundational toolset for generating complex visuals from minimalist inputs, ultimately broadening the horizons for AI-mediated image synthesis.