---
title: 'InstanceAssemble: Layout-Aware Diffusion'
url: https://www.emergentmind.com/topics/instanceassemble
type: topic
---

# InstanceAssemble: Layout-Aware Diffusion

Searching arXiv for the focal paper and closely related "instance assembly" terminology in adjacent areas.
InstanceAssemble is a layout-to-image generation method for diffusion transformers that conditions image synthesis on a global prompt together with instance-wise layout conditions, where each instance is specified by content and a bounding box. Its defining mechanism is **instance assembling attention**, which crops latent image tokens by instance boxes, performs attention between each instance token and its associated regional image tokens, and then assembles the updated regional features back into a global latent representation. The method is implemented as a light-weighted LoRA adaptation on top of existing DiT-based text-to-image models such as **SD3-Medium** and **Flux.1**, and is introduced together with the **Denselayout** benchmark and the **Layout Grounding Score (LGS)** evaluation protocol [2509.16691].

## 1. Problem setting and conceptual scope

InstanceAssemble addresses **layout-to-image (L2I) generation** under a formulation in which generation is controlled by a global prompt $\boldsymbol{p}$ and a set of instance-wise layout conditions $\boldsymbol{L}$:
$$
\boldsymbol{L} = \{l_1, \dots, l_N\}, \quad \text{where} \ l_i = (c_i, b_i).
$$
Here, $b_i$ denotes spatial position as a bounding box and $c_i$ denotes instance content. The method supports **bounding-box-based positional control**, **textual content control**, and **additional visual content control** such as a per-instance reference image, depth map, or edge map [2509.16691].

The paper motivates this design by separating prior L2I methods into **training-free** and **training-based** approaches. Training-free methods are described as sensitive to hyperparameters, slow at inference, and mainly effective for simple or sparse layouts, while training-based methods improve grounding but often add many parameters and increase training complexity. The specific modeling claim behind InstanceAssemble is that allowing all image tokens to attend to all instance tokens is suboptimal in dense scenes because instance signals interfere under overlapping boxes, tiny objects, and crowded layouts [2509.16691].

Within the broader literature, the term “assemble” has distinct meanings. In **video instance segmentation**, “instance flow assembly” refers to assembling dense offset predictions into instance-level center flows for temporal association [2110.10599]. In **3D part assembly**, assembly refers to predicting rigid poses for object parts or generating assembled geometry from part sets [2207.01779; 2506.17074]. InstanceAssemble instead uses “assembly” to denote the composition of **instance-conditioned latent image regions** inside a 2D diffusion pipeline [2509.16691].

## 2. Layout representation and architectural organization

The global conditioning structure combines an existing pretrained text-to-image backbone with two added components: a **Layout Encoder** and **Assemble-MMDiT**. The backbone models reported are **SD3-Medium**, **Flux.1-Dev**, and **Flux.1-Schnell**. The paper states that the global text prompt and image latents are passed through the original MMDiT first, and that the resulting image tokens together with instance tokens are then processed by Assemble-MMDiT. This yields a **cascaded architecture** in which the base model first handles global scene synthesis and the layout-aware module then enforces instance grounding [2509.16691].

For each instance box
$$
b_i = (x_1, y_1, w, h) \in [0,1]^4,
$$
the Layout Encoder applies **DenseSample**, sampling $K^2$ uniformly spaced points:
$$
\mathcal{P}_i = \left\{\left(x_1 + k_x \cdot \frac{w}{K}, y_1 + k_y \cdot \frac{h}{K}\right) \middle| k_x,k_y \in \{0,\dots,K-1\}\right\}.
$$
These sampled points are Fourier embedded and fused with instance content. For textual instance content, the instance token is
$$
h^i_l = \mathrm{MLP}\left([\boldsymbol{\tau}(c_i), \mathrm{Fourier}(\mathcal{P}_i)]\right),
$$
where $\boldsymbol{\tau}$ is the text encoder. For visual instance content, the reported encoding path is
$$
h^i_l = \mathrm{MLP}\left(\mathrm{VAE}(c_i)\right).
$$
The experiments report support for **text only**, **text + image**, **text + depth**, and **text + edge**, although the exact multimodal fusion operator is not fully specified in the paper [2509.16691].

Adaptation is performed with **LoRA** while keeping the base model frozen. The reported parameter increase is **71M** on SD3-Medium, described as **3.46%**, and **102M** on Flux.1, described as **0.84%**. The paper does not provide LoRA rank or exact placement details [2509.16691].

## 3. Instance assembling attention

Instance assembling attention is the paper’s central technical mechanism. Let the latent image representation be
$$
\boldsymbol{h}^z \in \mathbb{R}^{C\times W\times H},
$$
and the instance-token set be
$$
\boldsymbol{h}^l \in \mathbb{R}^{C\times N}.
$$
For each instance $l_i$, the method crops the image tokens by the corresponding box:
$$
\boldsymbol{h}^z_{l_i} = \boldsymbol{h}^z[\boldsymbol{b}_i] \in \mathbb{R}^{C\times w \times h}.
$$
Attention is then performed only between the cropped regional image tokens and the corresponding instance token. The paper describes this by projecting $\boldsymbol{h}^z_{l_i}$ and $l_i$ into queries, keys, and values, then applying attention to obtain updated regional image tokens $\boldsymbol{h}^{z^\prime}_{l_i}$ and an updated instance token $\boldsymbol{h}^{l_i^\prime}$ [2509.16691].

After all instances are processed independently, the updated regions are written back into the global image-token map using an **instance density map** $M \in \mathbb{N}^{W\times H}$ that counts how many boxes cover each location. The assembled output is:
$$
\boldsymbol{h}^{z^\prime}[:,i,j] = \frac{1}{M[i,j]} \sum_{k=1}^N \boldsymbol{h}^{z^\prime}_{l_k}[:, i,j],
$$
with the updated instance tokens retained as
$$
\boldsymbol{h}^{l^\prime}[:,k] = \boldsymbol{h}^{l_k^\prime}.
$$
In non-overlapping regions, one instance update is written back directly; in overlapping regions, updates are averaged [2509.16691].

The bounding boxes therefore affect conditioning in two ways. First, they define the sampled positional encoding $\mathcal{P}_i$ in the Layout Encoder. Second, they define the crop region $\boldsymbol{h}^z[\boldsymbol{b}_i]$, which restricts each attention computation to the spatial region associated with a single instance. This design contrasts with global mixing of all instance tokens and all image tokens, and the paper presents it as the main reason the method remains effective under dense and overlapping layouts [2509.16691].

## 4. Training objective and inference schedule

The paper first recalls the standard latent-diffusion objective and then states the training objective actually used for InstanceAssemble:
$$
\mathcal{L} = \mathbb{E}_{\mathbf{\epsilon} \sim \mathcal{N}(0,\mathbf{I}), \,\mathbf{x}, \, t, \,\mathbf{p}, \boldsymbol{L}}
\left[
\left\| v_{\{\theta,\theta'\}}\!\big(\mathbf{z}_t, t, \mathbf{p}, \boldsymbol{L}\big) -(\mathbf{\epsilon} - \mathbf{x}) \right\|_2^2
\right],
$$
where
$$
\mathbf{z}_t = (1-t)\mathbf{x} + t\mathbf{\epsilon}.
$$
Here, $\theta$ denotes frozen base-model parameters and $\theta'$ denotes the trainable parameters of the Layout Encoder and Assemble-MMDiT [2509.16691].

Training updates only the added modules; the base model remains frozen. The textual-only model is trained on **LayoutSAM** at **1024 × 1024** resolution. The reported optimizer is **Prodigy**. For SD3-Medium, training runs for **380K iterations** with **batch size 2** on **8× H800 GPUs** for **7 days**. For Flux.1-Dev, training runs for **300K iterations** with **batch size 1** on **8× H800 GPUs** for **5 days**. A visual-content version on SD3-M reports **85M** added parameters and uses the same settings [2509.16691].

At inference time, layout-conditioned denoising is applied only during the **first 30% of diffusion steps**. The paper states that layout primarily forms in early stages, while later steps are more important for high-frequency detail refinement; applying layout control beyond 30% is reported to hurt image quality and increase compute, and applying it for less than 30% gives insufficient layout alignment [2509.16691].

The paper does not report auxiliary grounding losses, an explicit unified LGS loss, a learning-rate value, or LoRA rank. It also does not provide an explicit mathematical expression for multimodal fusion when textual and visual instance conditions are used jointly [2509.16691].

## 5. Denselayout and Layout Grounding Score

The paper introduces **Denselayout** as a dense open-set layout-to-image benchmark intended to stress-test methods under many-instance conditions. It contains **5,000 images** and **90,339 instances**, with an average of **18.1 instances per image**. Only images with at least **15 instances** are retained, so density is the defining property of the benchmark [2509.16691].

The reported construction pipeline has five stages. Images are first generated with **Flux.1-Dev** from generic prompts, then tagged with **RAM++**. Objects are detected with **GroundingDINO** using `box_threshold = 0.35` and `text_threshold = 0.25`. Each detected region is captioned with **Qwen2.5-VL**. Finally, images are filtered to keep only dense cases. The benchmark annotation includes `global_caption`, `image_info`, and `instance_info`; each instance stores a bounding box, a coarse description from GroundingDINO, and a fine-grained `detail_description` from Qwen2.5-VL [2509.16691].

To evaluate grounding, the paper introduces **Layout Grounding Score (LGS)**. LGS is described as having two parts. **Spatial Accuracy (DetectIoU)** uses GroundingDINO to detect instances and computes mean IoU between detected boxes and conditioned boxes. **Semantic Accuracy** then uses a VLM-based VQA model to assess **color**, **texture**, and **shape** on crops whose IoU exceeds 0.5. In practice, the reported components are **mIoU**, **color**, **texture**, and **shape**, rather than a single unified scalar formula. The tools used are **GroundingDINO** for localization and **MiniCPM-V** for semantic verification [2509.16691].

This evaluation design separates localization from semantics. The paper contrasts it with AP/AR-style detection metrics, CropVQA, and SAMIoU, arguing that those alternatives either assume fixed label spaces or fail to decouple spatial correctness from semantic correctness [2509.16691].

## 6. Empirical performance and ablation findings

On **LayoutSAM-Eval**, the SD3-Medium version of InstanceAssemble reports **CropVQA spatial 94.97**, **color 77.53**, **texture 80.72**, **shape 80.11**, **LGS mIoU 78.88**, **LGS color 63.89**, **LGS texture 66.27**, **LGS shape 65.86**, **VQA 93.12**, **Pick 21.79**, and **CLIP 12.76**. The paper notes that some methods remain strong under CropVQA while exhibiting much lower LGS mIoU, and uses this contrast to motivate LGS as a stricter grounding metric [2509.16691].

On **COCO-Layout**, InstanceAssemble with SD3-Medium reports **mIoU 27.85**, **VQA 89.06**, **Pick 21.58**, and **CLIP 25.68**. The paper explicitly notes that **InstanceDiff** is slightly higher on mIoU at **30.39**, attributing that result to InstanceDiff’s fine-grained COCO training data and entity-wise generation strategy [2509.16691].

The clearest reported advantage appears on **Denselayout**. For Flux.1-based comparisons, the reported mIoU scores are **15.88** for Flux.1-Dev, **14.06** for Regional-Flux, **17.23** for RAG, **43.42** for InstanceAssemble with Flux.1-Dev, and **45.33** for InstanceAssemble with Flux.1-Schnell. The corresponding semantic scores for the Flux variants are **color 27.60 / 27.73**, **texture 29.50 / 30.06**, and **shape 29.14 / 29.62**, with **VQA 93.36 / 93.52** [2509.16691].

For SD3-Medium on Denselayout, the reported mIoU values are **47.31** for InstanceDiff, **34.39** for MIGC, **22.42** for HICO, **15.54** for CreatiLayout, and **52.07** for InstanceAssemble. Additional visual content conditioning improves performance further: **text only** reports **mIoU 43.72**, **text + image 55.29**, **text + depth 49.64**, and **text + edge 50.73**. The **text + image** setting also yields **color 42.15**, **texture 44.50**, and **shape 44.24** [2509.16691].

The ablations isolate the main components. Without any proposed modules, the base model reports **mIoU 11.69**, **color 9.16**, **texture 9.68**, **shape 9.56**, and **VQA 93.75**. Adding **Assemble-Attn** alone raises mIoU to **43.98** but lowers VQA to **84.57**. Adding the **cascaded** structure raises mIoU to **45.96** and VQA to **92.71**. Adding **LoRA** raises mIoU to **51.28**, and the final addition of **DenseSample** yields **mIoU 52.07**, **color 33.77**, **texture 36.21**, **shape 35.81**, and **VQA 93.54** [2509.16691].

These results indicate that the custom attention mechanism is the main source of layout grounding, while the cascaded structure and LoRA preserve or recover image quality. The paper also reports compatibility with style LoRAs such as **Cute3D LoRA**, **Oil Painting LoRA**, and **Ghibli LoRA**, presented qualitatively [2509.16691].

## 7. Position, interpretation, and limitations

InstanceAssemble is best understood as a **layout-aware latent conditioning method** rather than a general-purpose assembly framework. Its “assembly” operation is the overlap-aware reintegration of independently updated instance regions into a shared latent token map. This meaning differs from “assembly” in adjacent fields such as instance tracking from center-flow aggregation [2110.10599] and 3D pose-based part assembly from point clouds or anchor sets [2207.01779; 2506.17074].

The paper’s main limitations are also clearly delimited. It states that the current system requires **sequential Assemble-MMDiT calls**, and identifies **parallelization** as future work [2509.16691]. The method demonstrates support for textual and visual instance conditions, but the exact multimodal fusion operator is not fully described. Likewise, the paper does not provide a unified closed-form equation for LGS, a LoRA rank, or low-level optimization hyperparameters such as learning rate [2509.16691].

A broader implication is that InstanceAssemble treats dense layout grounding as an **instance-local composition problem** rather than a purely global conditioning problem. This suggests that the paper’s most durable contribution is not only its benchmark numbers, but its architectural claim that strong layout control in DiT/MMDiT backbones can be obtained by combining a global scene pass with a later stage of per-instance regional attention and overlap-aware feature assembly [2509.16691].

Source: https://www.emergentmind.com/topics/instanceassemble