MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis (2402.05408v2)

Abstract: We present a Multi-Instance Generation (MIG) task, simultaneously generating multiple instances with diverse controls in one image. Given a set of predefined coordinates and their corresponding descriptions, the task is to ensure that generated instances are accurately at the designated locations and that all instances' attributes adhere to their corresponding description. This broadens the scope of current research on Single-instance generation, elevating it to a more versatile and practical dimension. Inspired by the idea of divide and conquer, we introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. Initially, we break down the MIG task into several subtasks, each involving the shading of a single instance. To ensure precise shading for each instance, we introduce an instance enhancement attention mechanism. Lastly, we aggregate all the shaded instances to provide the necessary information for accurately generating multiple instances in stable diffusion (SD). To evaluate how well generation models perform on the MIG task, we provide a COCO-MIG benchmark along with an evaluation pipeline. Extensive experiments were conducted on the proposed COCO-MIG benchmark, as well as on various commonly used benchmarks. The evaluation results illustrate the exceptional control capabilities of our model in terms of quantity, position, attribute, and interaction. Code and demos will be released at https://migcproject.github.io/.

• The paper introduces the MIGC framework that divides multi-instance text-to-image synthesis into manageable subtasks using attention mechanisms for enhanced precision.
• It implements an instance enhancement attention mechanism to accurately shade and control individual instances, preventing attribute leakage and spatial errors.
• Benchmarking on COCO-MIG and DrawBench shows MIGC boosting the instance success rate from 32.39% to 58.43% and significantly improving spatial accuracy.

Multi-Instance Generation Controller for Text-to-Image Synthesis

The paper introduces the Multi-Instance Generation Controller (MIGC), a novel approach for addressing the Multi-Instance Generation (MIG) task in text-to-image synthesis. This task requires generating multiple instances within a single image, ensuring each instance adheres to predefined attributes, positions, and quantities. Unlike traditional single-instance generation, MIG broadens the applicability of text-to-image models to more complex and realistic scenarios.

Key Contributions

1. MIGC Framework: Inspired by the divide and conquer strategy, the paper proposes breaking down the MIG task into simpler subtasks, each focusing on generating individual instances. This decomposition leverages stable diffusion's strength in Single-Instance Generation and extends its capability to handle multiple instances.
2. Instance Enhancement: Introduction of an instance enhancement attention mechanism helps in accurately shading each instance while maintaining distinct characteristics. This process is crucial to avoid attribute leakage and spatial inaccuracies commonly found in existing methods.
3. Benchmark Development: The authors propose the COCO-MIG benchmark to evaluate the effectiveness of generation models on MIG tasks. This benchmark underscores the need for precise position, attribute, and quantity control.

Methodology

The proposed MIGC approach operates through a structured pipeline:

• Instance Division: The method divides the MIG task into individual instance shading subtasks, effectively managing the complexity by targeting each instance separately.
• Attention Mechanisms: Utilizes an Enhancement Attention layer to sharpen instance-specific attributes and a Layout Attention layer to maintain coherence among instances in the generated image.
• Shading Aggregation: A Shading Aggregation Controller combines individual shading outputs into a cohesive final image, ensuring consistency and quality.

Experimental Results

The paper conducts extensive experiments on COCO-MIG, COCO-Position, and DrawBench benchmarks, demonstrating the efficacy of MIGC. Notable improvements include:

• COCO-MIG: The Instance Success Rate increased from 32.39% to 58.43%, highlighting enhanced control over instance attributes and locations.
• COCO-Position: Achieved significant gains in spatial accuracy metrics, with a higher success rate and mean IoU, indicating better positional control over instances.
• DrawBench: MIGC achieved superior performance in both automated and manual evaluations across various aspects like position, color, and count.

Implications and Future Directions

By advancing the capabilities of text-to-image models to accurately control multiple instances, this paper pushes the boundary of what is feasible in creative and industrial applications where detailed scene generation is required. The insights gained from the divide and conquer strategy may be applicable in other complex machine-learning tasks, suggesting a broader potential impact.

Future research could explore enhancing the model's ability to manage interactive relationships between instances, a crucial factor for applications needing contextual awareness and interaction.

In conclusion, the MIGC framework represents a pivotal step in advancing text-to-image synthesis, offering a robust solution for complex multi-instance generation tasks.