Composer: Creative and Controllable Image Synthesis with Composable Conditions (2302.09778v2)

Abstract: Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as composable elements, leading to a huge design space (i.e., exponentially proportional to the number of decomposed factors) for customizable content creation. It is noteworthy that our approach, which we call Composer, supports various levels of conditions, such as text description as the global information, depth map and sketch as the local guidance, color histogram for low-level details, etc. Besides improving controllability, we confirm that Composer serves as a general framework and facilitates a wide range of classical generative tasks without retraining. Code and models will be made available.

• The paper introduces a novel compositional diffusion model that decomposes images into multiple conditions for modular control.
• It utilizes a two-stage process—decomposition of features and adaptive composition with a modified UNet—for versatile image generation.
• Experimental results demonstrate its capabilities across tasks like text-to-image synthesis, style transfer, and pose transfer with competitive metrics.

Composer: Creative and Controllable Image Synthesis with Composable Conditions

The paper "Composer: Creative and Controllable Image Synthesis with Composable Conditions" by Lianghua Huang et al. presents a novel framework for generative image models, with a focus on increasing the controllability and compositionality of the output. This framework, referred to as Composer, addresses the limitations of existing large-scale image generative models that, while capable of producing high-quality imagery, often lack detailed control over the generation process.

Overview of the Approach

Composer adopts a diffusion model approach, an advanced class of generative models recognized for their efficacy in producing high-fidelity images. The key innovation in Composer is the introduction of a multi-conditional diffusion model with an emphasis on compositionality. This approach allows for the decomposition of images into various representative factors, such as spatial arrangement, depth maps, and color palettes, which can be recomposed at the inference stage to generate new imagery. By doing so, Composer enables a vast design space exponentially proportional to the number of decomposed factors.

Methodology

The methodology of Composer involves two main stages:

1. Decomposition Phase: Images are decomposed into several independent representations or conditions. These include global features such as captions and image embeddings, and local features like sketches, depth maps, and segmentation maps. These conditions are extracted through computer vision algorithms and pretrained models.
2. Composition Phase: Using a modified UNet-based diffusion model, Composer reconstructs images from the aforementioned representations. The model incorporates both global and localized conditioning, and is designed to adaptively handle missing or additional conditions, making it highly versatile.

The model's joint training strategy employs a probabilistic dropout of conditions during training to enhance its ability to generalize across variable condition spaces effectively.

Experimental Results

Experiments demonstrate that Composer performs well across a diverse set of tasks, including text-to-image synthesis, multi-modal image generation, and traditional image manipulation tasks like style transfer and virtual try-on. Notably, Composer achieves a competitive FID score of 9.2 on the COCO dataset, reaffirming its capacity to produce high-quality images.

Composer also excels in task reformulation, offering new solutions for:

• Colorization: Transforming grayscale images into colorized versions using a targeted color palette.
• Style Transfer: Applying stylistic features from one image onto another while maintaining content integrity.
• Image Translation and Pose Transfer: Changing the style or pose of an object in an image seamlessly.

The framework's capability to isolate and manipulate specific image components independently is a highlighted strength, allowing for unprecedented levels of creativity and control in image synthesis.

Implications and Future Work

Composer's contribution lies not only in enhancing the controllability of image generation models but also in its potential application across fields that require customized content creation. Its methodology encourages further research into decomposition algorithms that could increase the precision of controllable image synthesis.

While the framework presents significant advances, challenges remain in scenarios involving conflicting conditions. Future work may focus on improving the model's handling of such cases and exploring more sophisticated condition handling strategies. Additionally, understanding and mitigating potential risks of misuse, such as the creation of deceptive imagery, will be critical as such technologies become more publicly accessible.

In summary, Composer represents a substantial advance in generative modeling, bringing the concept of compositionality to the forefront of controllable image synthesis and opening pathways for highly customized and creative image generation in practical applications.