InstanceAssemble: Layout-Aware Diffusion

• The paper introduces InstanceAssemble, a layout-to-image generation method that uses instance assembling attention to integrate instance-specific latent regions.
• It employs a cascaded architecture combining a global diffusion model with a dedicated Layout Encoder and Assemble-MMDiT module, using LoRA adaptation.
• Comprehensive experiments on the Denselayout benchmark demonstrate significant improvements in mIoU and semantic scores under dense and overlapping layouts.

Searching arXiv for the focal paper and closely related "instance assembly" terminology in adjacent areas. InstanceAssemble is a layout-to-image generation method for diffusion transformers that conditions image synthesis on a global prompt together with instance-wise layout conditions, where each instance is specified by content and a bounding box. Its defining mechanism is instance assembling attention, which crops latent image tokens by instance boxes, performs attention between each instance token and its associated regional image tokens, and then assembles the updated regional features back into a global latent representation. The method is implemented as a light-weighted LoRA adaptation on top of existing DiT-based text-to-image models such as SD3-Medium and Flux.1, and is introduced together with the Denselayout benchmark and the Layout Grounding Score (LGS) evaluation protocol (Xiang et al., 20 Sep 2025).

1. Problem setting and conceptual scope

InstanceAssemble addresses layout-to-image (L2I) generation under a formulation in which generation is controlled by a global prompt $\boldsymbol{p}$ and a set of instance-wise layout conditions $\boldsymbol{L}$:

$$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$

Here, $b_i$ denotes spatial position as a bounding box and $c_i$ denotes instance content. The method supports bounding-box-based positional control, textual content control, and additional visual content control such as a per-instance reference image, depth map, or edge map (Xiang et al., 20 Sep 2025).

The paper motivates this design by separating prior L2I methods into training-free and training-based approaches. Training-free methods are described as sensitive to hyperparameters, slow at inference, and mainly effective for simple or sparse layouts, while training-based methods improve grounding but often add many parameters and increase training complexity. The specific modeling claim behind InstanceAssemble is that allowing all image tokens to attend to all instance tokens is suboptimal in dense scenes because instance signals interfere under overlapping boxes, tiny objects, and crowded layouts (Xiang et al., 20 Sep 2025).

Within the broader literature, the term “assemble” has distinct meanings. In video instance segmentation, “instance flow assembly” refers to assembling dense offset predictions into instance-level center flows for temporal association (Li et al., 2021). In 3D part assembly, assembly refers to predicting rigid poses for object parts or generating assembled geometry from part sets (Zhang et al., 2022, Zhao et al., 20 Jun 2025). InstanceAssemble instead uses “assembly” to denote the composition of instance-conditioned latent image regions inside a 2D diffusion pipeline (Xiang et al., 20 Sep 2025).

2. Layout representation and architectural organization

The global conditioning structure combines an existing pretrained text-to-image backbone with two added components: a Layout Encoder and Assemble-MMDiT. The backbone models reported are SD3-Medium, Flux.1-Dev, and Flux.1-Schnell. The paper states that the global text prompt and image latents are passed through the original MMDiT first, and that the resulting image tokens together with instance tokens are then processed by Assemble-MMDiT. This yields a cascaded architecture in which the base model first handles global scene synthesis and the layout-aware module then enforces instance grounding (Xiang et al., 20 Sep 2025).

For each instance box

$b_i = (x_1, y_1, w, h) \in [0,1]^4,$

the Layout Encoder applies DenseSample, sampling $K^2$ uniformly spaced points:

$$\mathcal{P}_i = \left\{\left(x_1 + k_x \cdot \frac{w}{K}, y_1 + k_y \cdot \frac{h}{K}\right) \middle| k_x,k_y \in \{0,\dots,K-1\}\right\}.$$

These sampled points are Fourier embedded and fused with instance content. For textual instance content, the instance token is

$$h^i_l = \mathrm{MLP}\left([\boldsymbol{\tau}(c_i), \mathrm{Fourier}(\mathcal{P}_i)]\right),$$

where $\boldsymbol{\tau}$ is the text encoder. For visual instance content, the reported encoding path is

$\boldsymbol{L}$0

The experiments report support for text only, text + image, text + depth, and text + edge, although the exact multimodal fusion operator is not fully specified in the paper (Xiang et al., 20 Sep 2025).

Adaptation is performed with LoRA while keeping the base model frozen. The reported parameter increase is 71M on SD3-Medium, described as 3.46%, and 102M on Flux.1, described as 0.84%. The paper does not provide LoRA rank or exact placement details (Xiang et al., 20 Sep 2025).

3. Instance assembling attention

Instance assembling attention is the paper’s central technical mechanism. Let the latent image representation be

$\boldsymbol{L}$1

and the instance-token set be

$\boldsymbol{L}$2

For each instance $\boldsymbol{L}$3, the method crops the image tokens by the corresponding box:

$\boldsymbol{L}$4

Attention is then performed only between the cropped regional image tokens and the corresponding instance token. The paper describes this by projecting $\boldsymbol{L}$5 and $\boldsymbol{L}$6 into queries, keys, and values, then applying attention to obtain updated regional image tokens $\boldsymbol{L}$7 and an updated instance token $\boldsymbol{L}$8 (Xiang et al., 20 Sep 2025).

After all instances are processed independently, the updated regions are written back into the global image-token map using an instance density map $\boldsymbol{L}$9 that counts how many boxes cover each location. The assembled output is:

$$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$0

with the updated instance tokens retained as

$$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$1

In non-overlapping regions, one instance update is written back directly; in overlapping regions, updates are averaged (Xiang et al., 20 Sep 2025).

The bounding boxes therefore affect conditioning in two ways. First, they define the sampled positional encoding $$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$2 in the Layout Encoder. Second, they define the crop region $$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$3, which restricts each attention computation to the spatial region associated with a single instance. This design contrasts with global mixing of all instance tokens and all image tokens, and the paper presents it as the main reason the method remains effective under dense and overlapping layouts (Xiang et al., 20 Sep 2025).

4. Training objective and inference schedule

The paper first recalls the standard latent-diffusion objective and then states the training objective actually used for InstanceAssemble:

$$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$4

where

$$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$5

Here, $$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$6 denotes frozen base-model parameters and $$\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).$$7 denotes the trainable parameters of the Layout Encoder and Assemble-MMDiT (Xiang et al., 20 Sep 2025).

Training updates only the added modules; the base model remains frozen. The textual-only model is trained on LayoutSAM at 1024 × 1024 resolution. The reported optimizer is Prodigy. For SD3-Medium, training runs for 380K iterations with batch size 2 on 8× H800 GPUs for 7 days. For Flux.1-Dev, training runs for 300K iterations with batch size 1 on 8× H800 GPUs for 5 days. A visual-content version on SD3-M reports 85M added parameters and uses the same settings (Xiang et al., 20 Sep 2025).

At inference time, layout-conditioned denoising is applied only during the first 30% of diffusion steps. The paper states that layout primarily forms in early stages, while later steps are more important for high-frequency detail refinement; applying layout control beyond 30% is reported to hurt image quality and increase compute, and applying it for less than 30% gives insufficient layout alignment (Xiang et al., 20 Sep 2025).

The paper does not report auxiliary grounding losses, an explicit unified LGS loss, a learning-rate value, or LoRA rank. It also does not provide an explicit mathematical expression for multimodal fusion when textual and visual instance conditions are used jointly (Xiang et al., 20 Sep 2025).

5. Denselayout and Layout Grounding Score

The paper introduces Denselayout as a dense open-set layout-to-image benchmark intended to stress-test methods under many-instance conditions. It contains 5,000 images and 90,339 instances, with an average of 18.1 instances per image. Only images with at least 15 instances are retained, so density is the defining property of the benchmark (Xiang et al., 20 Sep 2025).

The reported construction pipeline has five stages. Images are first generated with Flux.1-Dev from generic prompts, then tagged with RAM++. Objects are detected with GroundingDINO using box_threshold = 0.35 and text_threshold = 0.25. Each detected region is captioned with Qwen2.5-VL. Finally, images are filtered to keep only dense cases. The benchmark annotation includes global_caption, image_info, and instance_info; each instance stores a bounding box, a coarse description from GroundingDINO, and a fine-grained detail_description from Qwen2.5-VL (Xiang et al., 20 Sep 2025).

To evaluate grounding, the paper introduces Layout Grounding Score (LGS). LGS is described as having two parts. Spatial Accuracy (DetectIoU) uses GroundingDINO to detect instances and computes mean IoU between detected boxes and conditioned boxes. Semantic Accuracy then uses a VLM-based VQA model to assess color, texture, and shape on crops whose IoU exceeds 0.5. In practice, the reported components are mIoU, color, texture, and shape, rather than a single unified scalar formula. The tools used are GroundingDINO for localization and MiniCPM-V for semantic verification (Xiang et al., 20 Sep 2025).

This evaluation design separates localization from semantics. The paper contrasts it with AP/AR-style detection metrics, CropVQA, and SAMIoU, arguing that those alternatives either assume fixed label spaces or fail to decouple spatial correctness from semantic correctness (Xiang et al., 20 Sep 2025).

6. Empirical performance and ablation findings

On LayoutSAM-Eval, the SD3-Medium version of InstanceAssemble reports CropVQA spatial 94.97, color 77.53, texture 80.72, shape 80.11, LGS mIoU 78.88, LGS color 63.89, LGS texture 66.27, LGS shape 65.86, VQA 93.12, Pick 21.79, and CLIP 12.76. The paper notes that some methods remain strong under CropVQA while exhibiting much lower LGS mIoU, and uses this contrast to motivate LGS as a stricter grounding metric (Xiang et al., 20 Sep 2025).

On COCO-Layout, InstanceAssemble with SD3-Medium reports mIoU 27.85, VQA 89.06, Pick 21.58, and CLIP 25.68. The paper explicitly notes that InstanceDiff is slightly higher on mIoU at 30.39, attributing that result to InstanceDiff’s fine-grained COCO training data and entity-wise generation strategy (Xiang et al., 20 Sep 2025).

The clearest reported advantage appears on Denselayout. For Flux.1-based comparisons, the reported mIoU scores are 15.88 for Flux.1-Dev, 14.06 for Regional-Flux, 17.23 for RAG, 43.42 for InstanceAssemble with Flux.1-Dev, and 45.33 for InstanceAssemble with Flux.1-Schnell. The corresponding semantic scores for the Flux variants are color 27.60 / 27.73, texture 29.50 / 30.06, and shape 29.14 / 29.62, with VQA 93.36 / 93.52 (Xiang et al., 20 Sep 2025).

For SD3-Medium on Denselayout, the reported mIoU values are 47.31 for InstanceDiff, 34.39 for MIGC, 22.42 for HICO, 15.54 for CreatiLayout, and 52.07 for InstanceAssemble. Additional visual content conditioning improves performance further: text only reports mIoU 43.72, text + image 55.29, text + depth 49.64, and text + edge 50.73. The text + image setting also yields color 42.15, texture 44.50, and shape 44.24 (Xiang et al., 20 Sep 2025).

The ablations isolate the main components. Without any proposed modules, the base model reports mIoU 11.69, color 9.16, texture 9.68, shape 9.56, and VQA 93.75. Adding Assemble-Attn alone raises mIoU to 43.98 but lowers VQA to 84.57. Adding the cascaded structure raises mIoU to 45.96 and VQA to 92.71. Adding LoRA raises mIoU to 51.28, and the final addition of DenseSample yields mIoU 52.07, color 33.77, texture 36.21, shape 35.81, and VQA 93.54 (Xiang et al., 20 Sep 2025).

These results indicate that the custom attention mechanism is the main source of layout grounding, while the cascaded structure and LoRA preserve or recover image quality. The paper also reports compatibility with style LoRAs such as Cute3D LoRA, Oil Painting LoRA, and Ghibli LoRA, presented qualitatively (Xiang et al., 20 Sep 2025).

7. Position, interpretation, and limitations

InstanceAssemble is best understood as a layout-aware latent conditioning method rather than a general-purpose assembly framework. Its “assembly” operation is the overlap-aware reintegration of independently updated instance regions into a shared latent token map. This meaning differs from “assembly” in adjacent fields such as instance tracking from center-flow aggregation (Li et al., 2021) and 3D pose-based part assembly from point clouds or anchor sets (Zhang et al., 2022, Zhao et al., 20 Jun 2025).

The paper’s main limitations are also clearly delimited. It states that the current system requires sequential Assemble-MMDiT calls, and identifies parallelization as future work (Xiang et al., 20 Sep 2025). The method demonstrates support for textual and visual instance conditions, but the exact multimodal fusion operator is not fully described. Likewise, the paper does not provide a unified closed-form equation for LGS, a LoRA rank, or low-level optimization hyperparameters such as learning rate (Xiang et al., 20 Sep 2025).

A broader implication is that InstanceAssemble treats dense layout grounding as an instance-local composition problem rather than a purely global conditioning problem. This suggests that the paper’s most durable contribution is not only its benchmark numbers, but its architectural claim that strong layout control in DiT/MMDiT backbones can be obtained by combining a global scene pass with a later stage of per-instance regional attention and overlap-aware feature assembly (Xiang et al., 20 Sep 2025).