Layout-Controllable Multi-Subject Synthesis
- LMS is a multi-subject synthesis approach that generates a single image with personalized subjects placed accurately within specified regions while maintaining overall scene consistency.
- Recent diffusion-based methods split the process into planning and rendering stages, using attention localization to balance subject identity preservation with precise layout control.
- State-of-the-art systems integrate hierarchical control interfaces and progressive training regimes to address trade-offs between spatial precision, stylistic adherence, and computational efficiency.
Searching arXiv for papers on layout-controllable multi-subject synthesis and closely related methods. arxiv_search(query="layout controllable multi-subject synthesis diffusion personalization layout-guided multi-subject", max_results=10, sort_by="relevance") arxiv_search(query="Layout-Controllable Multi-Subject Synthesis MUSE MS-Diffusion AnyMS LAMIC MultiCrafter", max_results=10, sort_by="relevance") arxiv_search(query="(Koch et al., 10 Nov 2025, Wang et al., 2024, Peng et al., 20 Aug 2025, Yu et al., 29 Dec 2025, Chen et al., 1 Aug 2025, Wu et al., 26 Sep 2025)", max_results=10, sort_by="relevance") Layout-Controllable Multi-Subject Synthesis (LMS) is the task of generating a single image that simultaneously preserves the identities or appearances of multiple specified subjects and places them in designated regions while remaining consistent with global scene semantics. In recent diffusion-based formulations, LMS is defined by the need to satisfy two hard constraints at once: faithful reconstruction of reference subjects and accurate placement of those subjects in specified regions within a unified image; earlier layout-to-image models framed a related problem as synthesizing complex multi-object scenes from reconfigurable layouts, styles, and later per-object attributes (Peng et al., 20 Aug 2025, Sun et al., 2020, Frolov et al., 2021). Across personalization, compositing, and in-context generation, the recurrent bottleneck is that subject fidelity, text alignment, and composition/layout control must all be satisfied simultaneously; typical failure modes include subject omission, subject blending, subject overcontrol, and layout ambiguity (Wang et al., 2024, Yu et al., 29 Dec 2025).
1. Historical development and problem consolidation
Early precursors of LMS arose in layout-to-image generation rather than in subject personalization. LostGAN introduced the layout-to-mask-to-image paradigm, arguing that a coarse spatial layout is too weak a signal to directly synthesize realistic images and that learned object masks should bridge the gap between layout and pixels (Sun et al., 2020). AttrLostGAN extended this interface from object labels and bounding boxes to object labels, bounding boxes, and attributes, so that a layout could specify not only where an object should appear but also fine-grained appearance aspects such as color, form, material, or state (Frolov et al., 2021). In parallel, DocSynth adapted layout-guided generation to document pages, where multiple semantic regions with precise spatial relations must be rendered consistently from bounding boxes and object categories (Biswas et al., 2021).
Diffusion-based work shifted the emphasis from generic multi-object scenes to user-specified subjects and explicit controllability during denoising. Cones 2 treated layout as a very abstract and easy-to-obtain prior, rectifying cross-attention maps with subject-specific masks so that different customized subjects could be appointed to separate image regions without retraining for each combination (Liu et al., 2023). “Obtaining Favorable Layouts for Multiple Object Generation” moved further toward inference-time layout control by extracting subject masks from cross-attention maps, rearranging those masks to reduce overlap, and migrating latent pixels to the new locations (Battash et al., 2024). MS-Diffusion then formalized layout-guided zero-shot image personalization with multiple subjects, placing spatial control and identity preservation within the same framework (Wang et al., 2024).
This lineage suggests that LMS emerged from the convergence of three research streams: layout-conditioned scene generation, subject-driven personalization, and controllable diffusion. Recent papers increasingly treat LMS not as a minor extension of text-to-image prompting but as a composite problem in which layout, identity, and semantics interfere with one another unless they are represented and optimized explicitly (Peng et al., 20 Aug 2025).
2. Formal task definitions and control interfaces
A common LMS formulation separates the semantic inventory of required instances from the spatial plan. In the two-stage layout-controlled system for table settings, the input is an object-count specification
$O = \{(c_i, n_i)\}_{i=1}^{N},$
where $c_i$ is an object class and $n_i$ is its required count, and the output layout is
$L = \{(c_j, b_j)\}_{j=1}^{M},$
with each instance paired to a bounding box
$b_j = (x_{\min}^j, y_{\min}^j, x_{\max}^j, y_{\max}^j).$
The key distinction is that the first stage produces a structured spatial plan rather than pixels (Koch et al., 10 Nov 2025). MUSE adopts a subject-personalization variant of the task: the model receives a global text prompt $T_{\text{global}}$, and for each of $N$ subjects it also receives a reference image $I_i$, a bounding box $B_i$, and a class text $T_i$, with the objective of synthesizing one image that matches the prompt, places each subject in $c_i$0, and preserves appearance from $c_i$1 (Peng et al., 20 Aug 2025).
Other systems broaden the control interface beyond plain rectangles. MS-CustomNet constructs a layout map $c_i$2 that explicitly encodes subject positions, scale or extent, relative layering or ordering, and spatial hierarchy, so that users can specify not only where each subject appears but also front-behind relations and compositional nesting (Cai et al., 22 Mar 2026). LLMControl replaces rectangular grounding with a polygonal Path Clip representation,
$c_i$3
which binds Fourier-encoded geometric parameters to appearance descriptions and thus represents irregular object boundaries more directly than bounding boxes (Wang et al., 26 Jul 2025). MUSIC uses a coarser semantics-driven layout plan by dividing the target image into an $c_i$4 patch grid with $c_i$5, assigning each patch a subject category or “others,” and serializing the resulting patch map into text (Zhou et al., 8 Apr 2026).
Not all LMS methods rely on explicit user boxes at inference. MOSAIC treats layout as semantic correspondence rather than as a geometric primitive: for each reference subject it provides correspondence pairs
$c_i$6
where $c_i$7 denotes a semantic point in the reference image and $c_i$8 the corresponding target latent position, with target sets constrained to be disjoint across references (She et al., 2 Sep 2025). This indicates that the central LMS requirement is not restricted to rectangles or masks; more fundamentally, it is the requirement that the correct subject control the correct spatial responsibility.
3. Recurrent architectural strategies
Recent LMS systems repeatedly decompose generation into planning and rendering. In the two-stage layout-controlled system, an LLM first produces a structured layout from an object list and a layout-conditioned diffusion model then synthesizes the final image. In that setting, directly asking the LLM to generate all objects in a complex 15–25 object scene led to 57.2% recall, whereas simplifying generation to core objects and completing the rest by deterministic rules raised recall to 99.9%; for rendering, ControlNet preserved text-based stylistic control but suffered from object hallucination, while GLIGEN achieved stronger layout fidelity at the cost of reduced prompt-based controllability after finetuning (Koch et al., 10 Nov 2025). LLMControl adopts a related decomposition in which a multimodal LLM acts as a global planner over layout, structure, and appearance (Wang et al., 26 Jul 2025), and MUSIC formalizes the same pattern as
$c_i$9
so that reasoning and image synthesis are explicitly separated (Zhou et al., 8 Apr 2026).
A second dominant pattern is attention localization. MS-Diffusion modifies image cross-attention with layout masks so that each subject condition influences only its designated region, while dummy background tokens preserve text-driven background generation outside subject boxes (Wang et al., 2024). AnyMS performs a dual decoupling: globally, text and image conditions are separated as
$n_i$0
and locally, each subject attends only within its assigned crop region before the result is merged back into the global feature map (Yu et al., 29 Dec 2025). LAMIC achieves a related effect in a training-free MMDiT setting through Group Isolation Attention, which suppresses inter-group interaction, and Region-Modulated Attention, which enforces stronger region-wise separation during early denoising (Chen et al., 1 Aug 2025). MultiCrafter reaches the same objective by supervising subject-specific attention maps against ground-truth masks with a Dice loss, explicitly targeting attention bleeding as the cause of attribute leakage (Wu et al., 26 Sep 2025).
A third pattern concerns semantic fusion of layout and identity. MUSE argues that naive dual-attention integration causes control collision, because layout is a local explicit constraint while the text pathway already carries the global semantic prior. Its concatenated cross-attention expands the text-side semantic space with layout grounding tokens, and the final combined attention is
$n_i$1
learned through a progressive layout-first then subject-synthesis schedule (Peng et al., 20 Aug 2025). LCP-Diffusion similarly couples subject identity and layout by combining Dynamic-Static Complementary Visual Refining with Dual Layout Control, building grounding tokens from static subject features, entity embeddings, and Fourier-encoded boxes (Li et al., 27 May 2025). CAG introduces a hierarchical Concept-to-Appearance Guidance design: VAE dropout shifts the model toward concept-level semantic reasoning from VLM features, while correspondence-aware masked attention restricts each referential word to its matched reference region (Xu et al., 3 Feb 2026). MOSAIC applies the same principle in representation space by supervising the reference-to-target attention submatrix with semantic correspondence and multi-reference disentanglement losses (She et al., 2 Sep 2025).
4. Data construction, training regimes, and evaluation practice
Large-scale LMS training data are difficult to obtain because identity, spatial layout, and textual description must all be aligned. MS-Diffusion constructs training samples of the form $n_i$2 from a large internal video dataset using NER, Grounding DINO, SAM, and subject matching across frames (Wang et al., 2024). Multitwine creates aligned compositing data from video relation datasets, in-the-wild images, and manually collected images, with captions, object crops, and interactions obtained through visual-LLMs and segmentation tools (Tarrés et al., 7 Feb 2025). MUSIC introduces an automatic and scalable pipeline comprising category sampling, caption generation, scene synthesis, subject detection, filtering, crop transformation, instruction generation, CoT generation, and patch-wise spatial layout planning; its reported error analysis retains 68.1% of generated candidates after cascading filters (Zhou et al., 8 Apr 2026). MOSAIC’s SemAlign-MS contributes 1.2M image pairs with validated semantic correspondences (She et al., 2 Sep 2025), while MS-CustomNet’s MSI dataset filters COCO 2017 train into 14,537 multi-subject scenes spanning all 80 COCO categories (Cai et al., 22 Mar 2026).
Training schedules are often specialized to stabilize competing objectives. MUSE uses progressive two-stage optimization, first learning layout control alone and then freezing that model before adding subject synthesis (Peng et al., 20 Aug 2025). CAG reports that performance improves as VAE dropout rises from 0 to around $n_i$3, with a main setting of 0.5 that substantially improves robustness when reference VAE features are absent at inference (Xu et al., 3 Feb 2026). MultiCrafter combines spatial supervision, MoE-LoRA capacity specialization, and online reinforcement learning with identity-preserving reward design (Wu et al., 26 Sep 2025). AnyMS and LAMIC, by contrast, transfer much of the complexity to inference: both are training-free, relying on pre-trained image adapters or pre-trained single-reference MMDiT backbones rather than subject-specific retraining (Yu et al., 29 Dec 2025, Chen et al., 1 Aug 2025).
Evaluation remains fragmented because LMS spans layout-conditioned generation, customization, personalization, compositing, and MLLM planning. MUSE separates layout control from full LMS using MIG Bench, MS-Bench, and MS-Bench-Random, with layout success defined through Grounding DINO detection and an IoU $n_i$4 criterion for all instances (Peng et al., 20 Aug 2025). AnyMS evaluates layout with mIoU and AP@50, identity with CLIP-I, DreamSim, and DINO, text alignment with CLIP-T, and efficiency through memory and inference time (Yu et al., 29 Dec 2025). LAMIC adds Inclusion Ratio, Fill Ratio, and Background Similarity, with
$n_i$5
to capture semantic, structural, and low-level background consistency (Chen et al., 1 Aug 2025). LCP-Diffusion uses AP, AP$n_i$6, and AP$n_i$7 from LW-DETR detections to quantify layout controllability (Li et al., 27 May 2025). This suggests that empirical comparisons are strongest within benchmark families rather than across the entire LMS literature.
5. Representative systems and reported empirical behavior
Reported results do not identify a single universally dominant LMS architecture; instead, different systems excel under different assumptions about layout explicitness, identity preservation, training budget, and planning complexity. Representative findings are summarized below.
| System | Core mechanism | Representative reported result |
|---|---|---|
| Two-stage LLM + layout-conditioned diffusion (Koch et al., 10 Nov 2025) | LLM layout planning, rule-based completion, ControlNet/GLIGEN synthesis | object recall improves from 57.2% to 99.9% in complex scenes |
| MS-Diffusion (Wang et al., 2024) | Grounding Resampler + Multi-subject Cross-Attention on SDXL | multi-subject CLIP-I 0.698, DINO 0.425, M-DINO 0.108, CLIP-T 0.341 |
| MUSE (Peng et al., 20 Aug 2025) | CCA + progressive two-stage LMS training | 88.4% average layout success on MIG Bench; fixed-layout MS-Bench SR-0.6 0.890 |
| AnyMS (Yu et al., 29 Dec 2025) | training-free global/local attention decoupling | AP50 35.65, mIOU 49.75, CLIP-T 35.82, CLIP-I 74.46 |
| LAMIC (Chen et al., 1 Aug 2025) | training-free GIA + RMA on MMDiT | IN-R 92.39, 91.90, and 89.81 for 2, 3, and 4 references |
| CAG (Xu et al., 3 Feb 2026) | VAE dropout + correspondence-aware masked attention | PF 7.308, SC 7.906, Overall 7.568 |
| MultiCrafter (Wu et al., 26 Sep 2025) | spatially disentangled attention, MoE-LoRA, and RL | Multi-human Face-Sim 0.5284; Overall average 0.5592 |
MLLM-driven planning forms another notable line of evidence. On the MSIC benchmark, MUSIC reports DINO 0.622, CLIP-I 0.812, and CLIP-T 0.322, while the test-time scaled MUSIC* reaches 0.631, 0.822, and 0.330 and also improves learned layout quality and relative spatial relation understanding over UNO (Zhou et al., 8 Apr 2026). In compositing-centric LMS, Multitwine reports CLIP-I 0.741 and DINO 0.532 on MultiComp-overlap, alongside user preference up to 97.1% for realistic interactions, indicating that simultaneous multi-object generation can outperform sequential insertion when interactions or reposing are required (Tarrés et al., 7 Feb 2025).
Personalized layout control also yields strong explicit spatial metrics. LCP-Diffusion reports multi-subject CLIP-I 0.717, DINO 0.413, CLIP-T 0.364, and AP/AP$n_i$8/AP$n_i$9 of 34.6/57.1/39.3, with higher layout scores than compared layout-controlled personalization baselines (Li et al., 27 May 2025). MS-CustomNet emphasizes explicit hierarchical composition and reports YOLO-L 0.94 together with DINO-I 0.61 on multi-subject customization, reflecting strong positional control under user-directed subject arrangements (Cai et al., 22 Mar 2026).
6. Trade-offs, limitations, and unresolved questions
A persistent LMS trade-off is that stronger spatial control often weakens another form of control. In the two-stage table-setting system, ControlNet preserved text-based stylistic control but hallucinated extra objects, whereas GLIGEN adhered more strictly to layout but became less responsive to stylistic prompts after finetuning (Koch et al., 10 Nov 2025). LAMIC reports that increasing the first-stage RMA ratio generally improves Inclusion Ratio and Fill Ratio but tends to reduce AES and sometimes DPG (Chen et al., 1 Aug 2025). This suggests that layout exactness, stylistic responsiveness, and global aesthetics remain partially competing objectives.
Another limitation concerns the form of layout supervision itself. MOSAIC achieves subject-region binding through semantic correspondences yet does not provide direct user-controlled explicit layouts at inference time (She et al., 2 Sep 2025). MUSIC remains relatively robust from 1 to 12 subjects but still degrades as subject count increases, and its Pass@N planning improves quality by increasing inference time linearly (Zhou et al., 8 Apr 2026). Cones 2 reports that combining more than six subjects is limited by the base model’s capability and also requires the user-provided layout to remain roughly consistent with the prompt (Liu et al., 2023). These results indicate that scaling subject count while preserving deterministic control is still unresolved.
Many systems also depend on auxiliary modules, heuristics, or expensive inference-time procedures. The two-stage LLM planner relies on domain-specific rule-based completion and a manually tuned heuristic layout score (Koch et al., 10 Nov 2025). LLMControl notes that polygon fitting is not always ideal for sharp shapes, that the optimal vertex count remains underexplored, and that DDIM inversion plus feature extraction increase computational cost (Wang et al., 26 Jul 2025). “Obtaining Favorable Layouts for Multiple Object Generation” roughly doubles inference time and can impose awkward arrangements because it is not aware of subject proportions (Battash et al., 2024). A plausible implication is that future LMS systems will need tighter integration of geometric control, subject identity preservation, and efficient generation, rather than treating planning, grounding, and rendering as loosely coupled stages.