InstanceControl: Controllable Complex Image Generation without Instance Labeling

Abstract: Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-LLM (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.

• The paper introduces a two-stage framework that automatically associates instance descriptions with image regions, eliminating the need for manual labeling.
• It employs a vision-language model and an adaptive mask refinement module to enhance MIoU, region accuracy, and overall generation fidelity in complex scenes.
• Numerical benchmarks show significant performance gains over existing methods, enabling scalable, controllable image synthesis in multi-instance scenarios.

InstanceControl: A Framework for Controllable Multi-Instance Image Generation Without Instance Labeling

Motivation and Problem Formulation

Most existing controllable image generation methods, such as ControlNet and its derivatives, are effective only in scenarios with a limited number of instances. In complex multi-instance scenes, these approaches typically encounter attribute confusion due to the inability to accurately associate each textual instance description with the corresponding region in the visual condition (e.g., edge or depth map). Manual annotation of instance correspondences mitigates this issue but introduces a significant overhead for practical applications. The paper proposes InstanceControl, a two-stage framework that eliminates the need for instance labeling during inference, leveraging vision-LLMs (VLMs) for automatic instance-level text-visual association.

Methodology

InstanceControl operates in two stages: (1) instance-level text-visual condition association and (2) instance-aware controllable generation.

Stage 1: Instance-Level Text-Visual Condition Association

A VLM, fine-tuned on a custom dataset, parses detailed instance descriptions from the text prompt and predicts instance masks in the visual condition. The shared SEG token strategy is employed for grouping multiple mentions of the same instance, ensuring semantic consistency across references. SAM is integrated as a mask decoder, refining query embeddings from the VLM into dense instance masks, with associated confidence scores.

Figure 1: Overview of InstanceControl's two-stage pipeline, establishing instance-level text-visual correspondences and refining masks for injection into controlled generation.

Stage 2: Instance-Aware Controllable Generation with Mask Refinement

Predicted instance masks from Stage 1 are potentially noisy; direct enforcement as hard constraints is suboptimal. The framework introduces a Mask Refinement Module (MRM): confidence scores from the VLM, averaged cross-attention maps from the diffusion model, and image latent features are adaptively combined in a lightweight U-Net with intra/inter-instance attention to yield rectified masks. These refined masks are used in a correspondence mask, constraining image-text attention such that image tokens attend exclusively to relevant text tokens. When mask confidence is low, attention-based masks dominate the constraint, reducing susceptibility to inaccurate spatial localization.

Figure 2: The mask refinement mechanism enables interactive correction, allowing user-guided refinement when prediction errors emerge.

Data Construction and Training Protocol

A high-density multi-instance dataset is built, combining segmentation masks from SAM, COCO, and UniWorld-V1, supplemented with detailed prompts (average 183 tokens) generated by Gemini 2.5 Pro, and analogy-based augmentations using Nano Banana for robustness against semantic and visual ambiguities. Training proceeds in two stages with LoRA fine-tuning on VLM and generative backbone, leveraging cross-entropy and dice losses for both text generation and mask prediction, and a rectified flow-matching loss for the generative model.

Numerical Results and Comparative Study

InstanceControl is benchmarked against state-of-the-art controllable image generation baselines (FLUX ControlNet, DreamRenderer, EliGen, CreatiLayout, Seg2Any) with and without instance labeling. Across MIG-Eval and COCO-POS datasets, InstanceControl achieves strong numerical results:

• On MIG-Eval for canny conditions, InstanceControl (label-free) achieves MIoU of 0.8250, region accuracy of 93.54%, and FID of 10.03, representing a $\sim$12.3% gain in accuracy compared to FLUX ControlNet.
• For depth and HED conditions, consistent improvements are observed in MIoU, local CLIP score, and spatial alignment.
• On COCO-POS, InstanceControl maintains superiority across all metrics compared with both instance-labeled and label-free baselines.

InstanceControl consistently outperforms Qwen-Image ControlNet and Nano Banana on unified vision-language benchmarks, with MIoU gains and higher region-wise accuracy.

Figure 3: InstanceControl enables precise attribute control for each instance in complex scenarios, while FLUX ControlNet exhibits frequent attribute confusion.

Qualitative Analysis

Visualizations demonstrate InstanceControl's capability to accurately parse instance-level associations and produce fine-grained controllable generations across variable numbers of instances. Attribute binding errors from FLUX ControlNet are rectified, providing strict semantic adherence even in overlapping or ambiguous layouts. Interactive mask correction further boosts fidelity when user guidance is incorporated.

Figure 4: Qualitative comparison highlights the finer attribute control realized by InstanceControl in challenging multi-instance prompts.

Figure 5: Learned correspondences between text and visual conditions are visualized alongside generated images, demonstrating robust multi-instance grounding.

Ablation Studies

Ablation experiments on the SST strategy and mask refinement confirm their contribution: SST improves MIoU and region accuracy by mitigating double-reference inconsistencies, while the MRM outperforms naive fusion strategies in accuracy and local CLIP score. Stage 1 MIoU positively correlates with Stage 2 accuracy; the mask refinement module reduces the impact of incomplete regions and localization offsets, evidenced by metric improvements.

Implications and Future Directions

InstanceControl advances controllable image generation in multi-instance, complex scenes without manual annotation. The effective utilization of VLMs for robust multi-modal grounding and adaptive constraint refinement positions this approach as highly scalable for practical deployment. Theoretical implications include the demonstration that cross-modal reasoning and adaptive attention-guided spatial grounding are sufficient for fine-grained control in diffusion-based generation models. Practically, the framework is extensible to other domains (video, graphics, interactive design) and compatible with unified vision-LLMs.

Further research could enhance grounding robustness, handle even denser instance distributions, and enable real-time interactive correction in generation workflows. Integration with large-scale unified multimodal models and multimodal reinforcement learning systems is a promising avenue.

Conclusion

InstanceControl introduces a technically rigorous framework for label-free, controllable multi-instance image generation, leveraging VLMs for instance-level text-visual association and adaptive mask refinement. The system achieves strong numerical results across multiple benchmarks and substantially mitigates inter-instance attribute confusion compared to prior methods. The methodology demonstrates high potential for scaling controllable generation in real-world, complex multi-object scenarios.