---
title: Layout-Controllable Multi-Subject Synthesis
url: https://www.emergentmind.com/topics/layout-controllable-multi-subject-synthesis-lms
type: topic
---

# Layout-Controllable Multi-Subject Synthesis

Searching arXiv for recent papers on layout-controllable multi-subject synthesis and closely related methods.
arxiv_search(query="layout controllable multi-subject synthesis diffusion personalization layout-guided multi-subject", max_results=10, sort_by="relevance")
arxiv_search(query="Layout-Controllable Multi-Subject Synthesis MUSE MS-Diffusion AnyMS LAMIC MultiCrafter", max_results=10, sort_by="relevance")
arxiv_search(query="2511.06888 2406.07209 2508.14440 2512.23537 2508.00477 2509.21953", max_results=10, sort_by="relevance")
Layout-Controllable Multi-Subject Synthesis (LMS) is the task of generating a single image that simultaneously preserves the identities or appearances of multiple specified subjects and places them in designated regions while remaining consistent with global scene semantics. In recent diffusion-based formulations, LMS is defined by the need to satisfy two hard constraints at once: faithful reconstruction of reference subjects and accurate placement of those subjects in specified regions within a unified image; earlier layout-to-image models framed a related problem as synthesizing complex multi-object scenes from reconfigurable layouts, styles, and later per-object attributes [2508.14440][2003.11571][2103.13722]. Across personalization, compositing, and in-context generation, the recurrent bottleneck is that subject fidelity, text alignment, and composition/layout control must all be satisfied simultaneously; typical failure modes include subject omission, subject blending, subject overcontrol, and layout ambiguity [2406.07209][2512.23537].

## 1. Historical development and problem consolidation

Early precursors of LMS arose in layout-to-image generation rather than in subject personalization. LostGAN introduced the layout-to-mask-to-image paradigm, arguing that a coarse spatial layout is too weak a signal to directly synthesize realistic images and that learned object masks should bridge the gap between layout and pixels [2003.11571]. AttrLostGAN extended this interface from object labels and bounding boxes to object labels, bounding boxes, and attributes, so that a layout could specify not only where an object should appear but also fine-grained appearance aspects such as color, form, material, or state [2103.13722]. In parallel, DocSynth adapted layout-guided generation to document pages, where multiple semantic regions with precise spatial relations must be rendered consistently from bounding boxes and object categories [2107.02638].

Diffusion-based work shifted the emphasis from generic multi-object scenes to user-specified subjects and explicit controllability during denoising. Cones 2 treated layout as a very abstract and easy-to-obtain prior, rectifying cross-attention maps with subject-specific masks so that different customized subjects could be appointed to separate image regions without retraining for each combination [2305.19327]. “Obtaining Favorable Layouts for Multiple Object Generation” moved further toward inference-time layout control by extracting subject masks from cross-attention maps, rearranging those masks to reduce overlap, and migrating latent pixels to the new locations [2405.00791]. MS-Diffusion then formalized layout-guided zero-shot image personalization with multiple subjects, placing spatial control and identity preservation within the same framework [2406.07209].

This lineage suggests that LMS emerged from the convergence of three research streams: layout-conditioned scene generation, subject-driven personalization, and controllable diffusion. Recent papers increasingly treat LMS not as a minor extension of text-to-image prompting but as a composite problem in which layout, identity, and semantics interfere with one another unless they are represented and optimized explicitly [2508.14440].

## 2. Formal task definitions and control interfaces

A common LMS formulation separates the semantic inventory of required instances from the spatial plan. In the two-stage layout-controlled system for table settings, the input is an object-count specification
\[
O = \{(c_i, n_i)\}_{i=1}^{N},
\]
where \(c_i\) is an object class and \(n_i\) is its required count, and the output layout is
\[
L = \{(c_j, b_j)\}_{j=1}^{M},
\]
with each instance paired to a bounding box
\[
b_j = (x_{\min}^j, y_{\min}^j, x_{\max}^j, y_{\max}^j).
\]
The key distinction is that the first stage produces a structured spatial plan rather than pixels [2511.06888]. MUSE adopts a subject-personalization variant of the task: the model receives a global text prompt \(T_{\text{global}}\), and for each of \(N\) subjects it also receives a reference image \(I_i\), a bounding box \(B_i\), and a class text \(T_i\), with the objective of synthesizing one image that matches the prompt, places each subject in \(B_i\), and preserves appearance from \(I_i\) [2508.14440].

Other systems broaden the control interface beyond plain rectangles. MS-CustomNet constructs a layout map \(M_L\) that explicitly encodes subject positions, scale or extent, relative layering or ordering, and spatial hierarchy, so that users can specify not only where each subject appears but also front-behind relations and compositional nesting [2603.21136]. LLMControl replaces rectangular grounding with a polygonal Path Clip representation,
\[
\tau := [ c_x, c_y, w, h, x_i, y_i, \ldots, x_k, y_k ],
\]
which binds Fourier-encoded geometric parameters to appearance descriptions and thus represents irregular object boundaries more directly than bounding boxes [2507.19939]. MUSIC uses a coarser semantics-driven layout plan by dividing the target image into an \(M \times M\) patch grid with \(M=8\), assigning each patch a subject category or “others,” and serializing the resulting patch map into text [2604.07422].

Not all LMS methods rely on explicit user boxes at inference. MOSAIC treats layout as semantic correspondence rather than as a geometric primitive: for each reference subject it provides correspondence pairs
\[
\mathcal{C}^{(i,k)} = \{(u_{i,j}, v_{i,j})\}_{j=1}^{P^{(k)}},
\]
where \(u_{i,j}\) denotes a semantic point in the reference image and \(v_{i,j}\) the corresponding target latent position, with target sets constrained to be disjoint across references [2509.01977]. This indicates that the central LMS requirement is not restricted to rectangles or masks; more fundamentally, it is the requirement that the correct subject control the correct spatial responsibility.

## 3. Recurrent architectural strategies

Recent LMS systems repeatedly decompose generation into planning and rendering. In the two-stage layout-controlled system, an LLM first produces a structured layout from an object list and a layout-conditioned diffusion model then synthesizes the final image. In that setting, directly asking the LLM to generate all objects in a complex 15–25 object scene led to 57.2% recall, whereas simplifying generation to core objects and completing the rest by deterministic rules raised recall to 99.9%; for rendering, ControlNet preserved text-based stylistic control but suffered from object hallucination, while GLIGEN achieved stronger layout fidelity at the cost of reduced prompt-based controllability after finetuning [2511.06888]. LLMControl adopts a related decomposition in which a multimodal LLM acts as a global planner over layout, structure, and appearance [2507.19939], and MUSIC formalizes the same pattern as
\[
f_1(T_{instr}, \{I_{subj_i}\}) \rightarrow (C_{CoT}, L_{spatial}),
\quad
f_2(C_{CoT}, L_{spatial}) \rightarrow \hat I_{tgt},
\]
so that reasoning and image synthesis are explicitly separated [2604.07422].

A second dominant pattern is attention localization. MS-Diffusion modifies image cross-attention with layout masks so that each subject condition influences only its designated region, while dummy background tokens preserve text-driven background generation outside subject boxes [2406.07209]. AnyMS performs a dual decoupling: globally, text and image conditions are separated as
\[
Z_{\mathrm{out}} = Z_{\mathrm{text}} + Z_{\mathrm{image}},
\]
and locally, each subject attends only within its assigned crop region before the result is merged back into the global feature map [2512.23537]. LAMIC achieves a related effect in a training-free MMDiT setting through Group Isolation Attention, which suppresses inter-group interaction, and Region-Modulated Attention, which enforces stronger region-wise separation during early denoising [2508.00477]. MultiCrafter reaches the same objective by supervising subject-specific attention maps against ground-truth masks with a Dice loss, explicitly targeting attention bleeding as the cause of attribute leakage [2509.21953].

A third pattern concerns semantic fusion of layout and identity. MUSE argues that naive dual-attention integration causes control collision, because layout is a local explicit constraint while the text pathway already carries the global semantic prior. Its concatenated cross-attention expands the text-side semantic space with layout grounding tokens, and the final combined attention is
\[
\text{FCA}=\sigma\left(\frac{Q[K, K_T]^T}{\sqrt{d}}\right)[V, V_T] + \lambda \cdot \sigma\left(\frac{Q K_I^T}{\sqrt{d}}\right)V_I,
\]
learned through a progressive layout-first then subject-synthesis schedule [2508.14440]. LCP-Diffusion similarly couples subject identity and layout by combining Dynamic-Static Complementary Visual Refining with Dual Layout Control, building grounding tokens from static subject features, entity embeddings, and Fourier-encoded boxes [2505.20909]. CAG introduces a hierarchical Concept-to-Appearance Guidance design: VAE dropout shifts the model toward concept-level semantic reasoning from VLM features, while correspondence-aware masked attention restricts each referential word to its matched reference region [2602.03448]. MOSAIC applies the same principle in representation space by supervising the reference-to-target attention submatrix with semantic correspondence and multi-reference disentanglement losses [2509.01977].

## 4. Data construction, training regimes, and evaluation practice

Large-scale LMS training data are difficult to obtain because identity, spatial layout, and textual description must all be aligned. MS-Diffusion constructs training samples of the form \([subjects, entities, spatial layouts]\) from a large internal video dataset using NER, Grounding DINO, SAM, and subject matching across frames [2406.07209]. Multitwine creates aligned compositing data from video relation datasets, in-the-wild images, and manually collected images, with captions, object crops, and interactions obtained through visual-language models and segmentation tools [2502.05165]. MUSIC introduces an automatic and scalable pipeline comprising category sampling, caption generation, scene synthesis, subject detection, filtering, crop transformation, instruction generation, CoT generation, and patch-wise spatial layout planning; its reported error analysis retains 68.1% of generated candidates after cascading filters [2604.07422]. MOSAIC’s SemAlign-MS contributes 1.2M image pairs with validated semantic correspondences [2509.01977], while MS-CustomNet’s MSI dataset filters COCO 2017 train into 14,537 multi-subject scenes spanning all 80 COCO categories [2603.21136].

Training schedules are often specialized to stabilize competing objectives. MUSE uses progressive two-stage optimization, first learning layout control alone and then freezing that model before adding subject synthesis [2508.14440]. CAG reports that performance improves as VAE dropout rises from 0 to around \(0.3\!-\!0.5\), with a main setting of 0.5 that substantially improves robustness when reference VAE features are absent at inference [2602.03448]. MultiCrafter combines spatial supervision, MoE-LoRA capacity specialization, and online reinforcement learning with identity-preserving reward design [2509.21953]. AnyMS and LAMIC, by contrast, transfer much of the complexity to inference: both are training-free, relying on pre-trained image adapters or pre-trained single-reference MMDiT backbones rather than subject-specific retraining [2512.23537][2508.00477].

Evaluation remains fragmented because LMS spans layout-conditioned generation, customization, personalization, compositing, and MLLM planning. MUSE separates layout control from full LMS using MIG Bench, MS-Bench, and MS-Bench-Random, with layout success defined through Grounding DINO detection and an IoU \(>0.5\) criterion for all instances [2508.14440]. AnyMS evaluates layout with mIoU and AP@50, identity with CLIP-I, DreamSim, and DINO, text alignment with CLIP-T, and efficiency through memory and inference time [2512.23537]. LAMIC adds Inclusion Ratio, Fill Ratio, and Background Similarity, with
\[
BG\text{-}S = 0.4 \cdot DINO + 0.25 \cdot CLIP + 0.2 \cdot SSIM + 0.15 \cdot CH
\]
to capture semantic, structural, and low-level background consistency [2508.00477]. LCP-Diffusion uses AP, AP\(_{50}\), and AP\(_{75}\) from LW-DETR detections to quantify layout controllability [2505.20909]. This suggests that empirical comparisons are strongest within benchmark families rather than across the entire LMS literature.

## 5. Representative systems and reported empirical behavior

Reported results do not identify a single universally dominant LMS architecture; instead, different systems excel under different assumptions about layout explicitness, identity preservation, training budget, and planning complexity. Representative findings are summarized below.

| System | Core mechanism | Representative reported result |
|---|---|---|
| Two-stage LLM + layout-conditioned diffusion [2511.06888] | LLM layout planning, rule-based completion, ControlNet/GLIGEN synthesis | object recall improves from 57.2% to 99.9% in complex scenes |
| MS-Diffusion [2406.07209] | Grounding Resampler + Multi-subject Cross-Attention on SDXL | multi-subject CLIP-I 0.698, DINO 0.425, M-DINO 0.108, CLIP-T 0.341 |
| MUSE [2508.14440] | CCA + progressive two-stage LMS training | 88.4% average layout success on MIG Bench; fixed-layout MS-Bench SR-0.6 0.890 |
| AnyMS [2512.23537] | training-free global/local attention decoupling | AP50 35.65, mIOU 49.75, CLIP-T 35.82, CLIP-I 74.46 |
| LAMIC [2508.00477] | training-free GIA + RMA on MMDiT | IN-R 92.39, 91.90, and 89.81 for 2, 3, and 4 references |
| CAG [2602.03448] | VAE dropout + correspondence-aware masked attention | PF 7.308, SC 7.906, Overall 7.568 |
| MultiCrafter [2509.21953] | spatially disentangled attention, MoE-LoRA, and RL | Multi-human Face-Sim 0.5284; Overall average 0.5592 |

MLLM-driven planning forms another notable line of evidence. On the MSIC benchmark, MUSIC reports DINO 0.622, CLIP-I 0.812, and CLIP-T 0.322, while the test-time scaled MUSIC* reaches 0.631, 0.822, and 0.330 and also improves learned layout quality and relative spatial relation understanding over UNO [2604.07422]. In compositing-centric LMS, Multitwine reports CLIP-I 0.741 and DINO 0.532 on MultiComp-overlap, alongside user preference up to 97.1% for realistic interactions, indicating that simultaneous multi-object generation can outperform sequential insertion when interactions or reposing are required [2502.05165].

Personalized layout control also yields strong explicit spatial metrics. LCP-Diffusion reports multi-subject CLIP-I 0.717, DINO 0.413, CLIP-T 0.364, and AP/AP\(_{50}\)/AP\(_{75}\) of 34.6/57.1/39.3, with higher layout scores than compared layout-controlled personalization baselines [2505.20909]. MS-CustomNet emphasizes explicit hierarchical composition and reports YOLO-L 0.94 together with DINO-I 0.61 on multi-subject customization, reflecting strong positional control under user-directed subject arrangements [2603.21136].

## 6. Trade-offs, limitations, and unresolved questions

A persistent LMS trade-off is that stronger spatial control often weakens another form of control. In the two-stage table-setting system, ControlNet preserved text-based stylistic control but hallucinated extra objects, whereas GLIGEN adhered more strictly to layout but became less responsive to stylistic prompts after finetuning [2511.06888]. LAMIC reports that increasing the first-stage RMA ratio generally improves Inclusion Ratio and Fill Ratio but tends to reduce AES and sometimes DPG [2508.00477]. This suggests that layout exactness, stylistic responsiveness, and global aesthetics remain partially competing objectives.

Another limitation concerns the form of layout supervision itself. MOSAIC achieves subject-region binding through semantic correspondences yet does not provide direct user-controlled explicit layouts at inference time [2509.01977]. MUSIC remains relatively robust from 1 to 12 subjects but still degrades as subject count increases, and its Pass@N planning improves quality by increasing inference time linearly [2604.07422]. Cones 2 reports that combining more than six subjects is limited by the base model’s capability and also requires the user-provided layout to remain roughly consistent with the prompt [2305.19327]. These results indicate that scaling subject count while preserving deterministic control is still unresolved.

Many systems also depend on auxiliary modules, heuristics, or expensive inference-time procedures. The two-stage LLM planner relies on domain-specific rule-based completion and a manually tuned heuristic layout score [2511.06888]. LLMControl notes that polygon fitting is not always ideal for sharp shapes, that the optimal vertex count remains underexplored, and that DDIM inversion plus feature extraction increase computational cost [2507.19939]. “Obtaining Favorable Layouts for Multiple Object Generation” roughly doubles inference time and can impose awkward arrangements because it is not aware of subject proportions [2405.00791]. A plausible implication is that future LMS systems will need tighter integration of geometric control, subject identity preservation, and efficient generation, rather than treating planning, grounding, and rendering as loosely coupled stages.

Source: https://www.emergentmind.com/topics/layout-controllable-multi-subject-synthesis-lms