Hierarchical Layout Generation

• Hierarchical Layout Generation is a method that decomposes layout prediction into successive structural stages, preserving semantics and improving controllability.
• It employs diverse representations—bounding boxes, graphs, SVG trees, and diffusion branches—to manage spatial hierarchy and facilitate human edits.
• Evaluation protocols focus on structural correctness, practical editability, and physical plausibility across applications such as UI, urban planning, posters, and 3D scenes.

Hierarchical Layout Generation (HLG) denotes a family of generative formulations in which layout prediction is decomposed across multiple structural levels rather than learned as a flat, one-shot mapping. In published work, those levels have been instantiated as semantic layout stages for text-to-image synthesis, unordered graphic elements with inferred layer order, typed graphs for user interfaces and cities, SVG trees for posters, branch-structured conditioning for diffusion, and room–support–object hierarchies for 3D scenes (Hong et al., 2018, Cheng et al., 2024, He et al., 2024, Cheng et al., 2024, Jin et al., 26 May 2025, Wang et al., 25 Aug 2025). The unifying idea is that explicit intermediate structure—such as boxes, masks, relation matrices, support regions, or placeholders—can preserve semantics, improve controllability, and reduce geometric or physical violations while exposing interpretable states for editing and partial specification.

1. Emergence of the hierarchical formulation

One early and influential decomposition appears in text-to-image synthesis, where semantic layout is inferred before pixel rendering. Hong et al. model the latent layout as bounding boxes and object masks, factorizing

$$p(L \mid T) = p(B_{1:T} \mid T)\cdot p(M_{1:T} \mid B_{1:T}, T),$$

so that the generator first predicts object boxes and then refines each box into a binary mask (Hong et al., 2018). This formulation separates “what and where” from later image synthesis, and it explicitly supports user intervention by editing the generated scene layout before rerendering.

The term HLG was later made explicit in graphic design. Graphist defines HLG over an unordered set of RGBA design elements $M=\{M_1,\dots,M_n\}$, predicting a composition $S=\{s_1,\dots,s_n\}$ with $s_i=(x_i,y_i,w_i,h_i,l_i)$, where $l_i$ is the layer index or z-order (Cheng et al., 2024). That formalization marks a decisive shift from earlier Graphic Layout Generation (GLG), which assumed that the input order was already correct. In HLG, the system must infer both placement and hierarchy from an unordered set.

A parallel line of work treats hierarchy as multi-level decision making rather than sequence factorization. In virtual indoor scenes, a two-furniture layout problem was cast as an MDP and solved with hierarchical actor-critic, using a manager policy $\pi_1$ that proposes sub-goals and a worker policy $\pi_0$ that executes primitive actions (Di et al., 2022). This establishes that HLG is not tied to a single model family: it can be framed as latent-variable generation, structured prediction, or hierarchical control.

The literature therefore does not present HLG as one canonical architecture. Instead, it uses hierarchy to reduce the complexity of layout reasoning, to impose structure on multimodal conditioning, or to expose controllable intermediate variables. That breadth is visible in later work spanning UI design, urban generation, posters, slides, webpages, and 3D indoor scenes (He et al., 2024, Hsu et al., 6 May 2025, Tang et al., 9 Jun 2025, Li et al., 16 Apr 2026).

2. Structural representations used in HLG

The primary technical divergence across HLG systems lies in how hierarchy is represented.

Domain	Representation	Hierarchical unit
UI layout (Jin et al., 26 May 2025)	Directed attributed graph $G=(V,E)$	contain, parallel, top, left edges
City-scale urban layout (He et al., 2024)	Canonical graph $G=(B,E)$	blocks, communities, city context
Graphic design (Cheng et al., 2024)	JSON sequence over unordered RGBA patches	spatial transforms plus layer index $l_i$
Poster design (Hsu et al., 6 May 2025)	SVG-style tree $M=\{M_1,\dots,M_n\}$0	parent–child enclosure and intent nodes
Layout-to-image (Cheng et al., 2024)	$M=\{M_1,\dots,M_n\}$1 conditioning branches	background level and per-object levels
3D indoor scenes (Wang et al., 25 Aug 2025, Pun et al., 21 Mar 2025, Li et al., 8 Jun 2026)	scene graphs, support trees, XML/CSS trees	room, zones/surfaces, objects

In UI generation, ASR models each layout as a directed attributed graph $M=\{M_1,\dots,M_n\}$2. Each node carries a category label from $M=\{M_1,\dots,M_n\}$3 and a 4-D coordinate vector $M=\{M_1,\dots,M_n\}$4; edges encode positional relations (“top”, “left”) and semantic relations (“contain”, “parallel”) (Jin et al., 26 May 2025). The hierarchy is induced by contain edges, while sibling structure and local order are represented by parallel, top, and left.

In COHO, the hierarchy is city-scale and contextual rather than containment-based. The city is represented as a graph over city-block nodes, each with a 4-D shape/location descriptor $M=\{M_1,\dots,M_n\}$5 and a 512-D quantized code $M=\{M_1,\dots,M_n\}$6 describing the internal building layout, while edges encode adjacency and distance between blocks (He et al., 2024). The same graph simultaneously captures neighboring blocks, communities, and city-level context.

PosterO uses an SVG tree whose leaves include both element nodes and design-intent nodes. Every node carries a shape tag from $M=\{M_1,\dots,M_n\}$7, normalized geometric parameters, and, for intent nodes, a latent embedding from a U-Net-style intent model (Hsu et al., 6 May 2025). Parent–child grouping is derived by enclosure tests, and absolute coordinates are recovered by accumulating parent offsets down the tree.

In 3D indoor generation, hierarchy is usually support-centric. HLG for room construction defines a scene $M=\{M_1,\dots,M_n\}$8, assigns each object to a parent, and decomposes constraints across levels $M=\{M_1,\dots,M_n\}$9 such that $S=\{s_1,\dots,s_n\}$0 for $S=\{s_1,\dots,s_n\}$1 (Wang et al., 25 Aug 2025). HSM similarly models a rooted tree $S=\{s_1,\dots,s_n\}$2 of support regions and object-placement nodes, with factorization over levels and parents (Pun et al., 21 Mar 2025). HDSL turns that principle into an explicit XML/CSS-style DSL in which nesting simultaneously represents containment and support, and coordinates are local to the parent frame (Li et al., 8 Jun 2026).

These formalisms indicate that “hierarchy” in HLG is not restricted to depth order. It may refer to containment, support, spatial scale, semantic intent, or disentangled object-conditioned branches.

3. Core generation mechanisms

Once a hierarchy is defined, HLG systems differ in how they propagate information through it.

ASR uses a 5-layer GNN $S=\{s_1,\dots,s_n\}$3 to aggregate node and relational features, producing a graph-level code $S=\{s_1,\dots,s_n\}$4. It then decodes relation matrices $S=\{s_1,\dots,s_n\}$5 and $S=\{s_1,\dots,s_n\}$6, replaces the traditional ViT module inside a multimodal LLM with graph-derived features, and has the LLM emit a JSON record for each node containing category, coordinates, and relations (Jin et al., 26 May 2025). The crucial design choice is that all LLM cross-attention layers operate on hierarchical graph features rather than raw pixels.

COHO takes a masked autoencoding route. A graph-based masked autoencoder (GMAE) hides only the 512-D building-layout codes $S=\{s_1,\dots,s_n\}$7, keeping shape/location $S=\{s_1,\dots,s_n\}$8 and edges $S=\{s_1,\dots,s_n\}$9 visible, and reconstructs the masked codes with a cross-entropy objective over quantized dimensions (He et al., 2024). Generation is not purely one-shot: COHO uses $s_i=(x_i,y_i,w_i,h_i,l_i)$0 scheduled iterative sampling steps and admits the top $s_i=(x_i,y_i,w_i,h_i,l_i)$1 fraction of confident blocks at each step, so semantically important regions are fixed early and later predictions inherit their style.

HiCo embeds hierarchy inside a diffusion conditioning architecture. It augments a pretrained latent diffusion UNet with $s_i=(x_i,y_i,w_i,h_i,l_i)$2 side branches: one global branch for background and $s_i=(x_i,y_i,w_i,h_i,l_i)$3 local branches for foreground objects. Fusion is mask-based, with branch outputs gated by binary masks and summed into a conditioning signal (Cheng et al., 2024). This makes the hierarchy object-separable rather than graph-structured, and the intended effect is spatial disentanglement.

Graphist and PosterO both reframe HLG as sequence generation, but with different structured outputs. Graphist serializes layout as JSON tokens and trains a large multimodal model on an autoregressive objective $s_i=(x_i,y_i,w_i,h_i,l_i)$4, using an RGBA encoder and a “Visual Shrinker” to reduce visual token length (Cheng et al., 2024). PosterO instead converts layouts into SVG trees and uses in-context learning: examples are selected by nearest intent embedding, then an LLM predicts a new layout tree aligned to the test image’s intent region (Hsu et al., 6 May 2025).

SlideCoder and MM-WebAgent represent another shift: hierarchy becomes task decomposition for agentic code generation. SlideCoder uses three cascading levels—coarse visual segmentation, block-level description and snippet generation, and global assembly—under a Hierarchical Retrieval-Augmented Generation pipeline with a Shape Type Knowledge Base and an Operation Function Knowledge Base (Tang et al., 9 Jun 2025). MM-WebAgent is explicitly training-free: it produces a global layout plan $s_i=(x_i,y_i,w_i,h_i,l_i)$5, local placeholder plans, a code skeleton, and then runs a three-level self-reflection loop over local assets, HTML/CSS context, and full-page global coherence (Li et al., 16 Apr 2026).

In 3D, generation is commonly coupled with optimization. HLG for room construction first extracts scene information, creates a coarse room, then applies Fine-Grained Layout Alignment and a Trainable Layout Optimization Network with ownership and pose heads (Wang et al., 25 Aug 2025). HOG-Layout interleaves LLM planning, VLM-based group-wise placement, retrieval-augmented asset grounding, and a hierarchical force-directed optimization objective over collision, support, boundary, adjacency, and alignment terms (Jiang et al., 12 Apr 2026). HDSL similarly follows recursive subtree generation with bounded verification, multimodal retrieval, and force-directed layout optimization (Li et al., 8 Jun 2026).

4. Controllability, editing, and human-centric interaction

A defining feature of many HLG systems is that their intermediate states are directly editable.

ASR makes the intermediate relation matrices $s_i=(x_i,y_i,w_i,h_i,l_i)$6 and $s_i=(x_i,y_i,w_i,h_i,l_i)$7 human-editable. Its pipeline accepts a user-provided or randomly masked skeleton graph, decodes relation matrices, optionally allows the user to edit them, and then feeds the edited structure into the MLLM to generate the final node list (Jin et al., 26 May 2025). The paper states that designers can progressively “push” new contain, parallel, or positional constraints and obtain LLM-driven reflows that respect the edited hierarchy.

Hong et al. exposed the same principle in an earlier form by separating layout from rendering: because boxes $s_i=(x_i,y_i,w_i,h_i,l_i)$8 and masks $s_i=(x_i,y_i,w_i,h_i,l_i)$9 are explicit, users can add, remove, or move objects and rerun the later stages (Hong et al., 2018). PosterO extends editability to SVG realization, where the predicted layout tree is already valid SVG and can be further realized into posters by editing the chat with LLMs (Hsu et al., 6 May 2025).

In webpage generation, MM-WebAgent uses placeholders $l_i$0 and inherited style attributes $l_i$1 to coordinate downstream asset generation. Editing is indirect but systematic: local refine fixes individual assets, context refine patches integration errors such as clipping or overflow, and global refine adjusts layout and style consistency after inspecting the full rendered page (Li et al., 16 Apr 2026). The hierarchy is therefore operationalized as a refinement schedule.

In 3D editing, hierarchy becomes a localization device. HOG-Layout supports Add, Move, and Delete operations by updating only the relevant parent–child relations and then re-running optimization (Jiang et al., 12 Apr 2026). HDSL formalizes localized editing more aggressively through Hierarchical Retrieval-Augmented Generation: it retrieves a relevant subtree by path-aware embeddings, asks the LLM to rewrite only that fragment, and merges it back with a deterministic three-way merge before reapplying force-directed refinement (Li et al., 8 Jun 2026).

This body of work rejects the misconception that hierarchy is only a regularizer for training. In HLG, hierarchy is often the mechanism by which control is exposed to human designers or to downstream editing agents.

5. Evaluation protocols and empirical patterns

HLG systems are evaluated with domain-specific metrics, and the diversity of those protocols reflects the diversity of tasks.

In UI generation on RICO, ASR reports for direct UI generation $l_i$2, $l_i$3, $l_i$4, and $l_i$5. On completion, it reports $l_i$6 and $l_i$7, and on graph-editing $l_i$8 and $l_i$9. In a crowd-sourced AMT study with $\pi_1$0, reasonableness scores were $\pi_1$1 for easy layouts, $\pi_1$2 for medium layouts, and $\pi_1$3 for hard layouts (Jin et al., 26 May 2025).

COHO evaluates urban layouts with Context Score (CTS), WD-5D, WD-CO, Overlap, O-Blk, FID, KID, and LPIPS. It reports the lowest WD-5D at $\pi_1$4, the lowest WD-CO at $\pi_1$5, and $\pi_1$6, with best FID/KID/LPIPS among the listed baselines across 330 heterogeneous US cities (He et al., 2024). The inclusion of CTS is notable because it directly measures harmonization with neighboring blocks rather than only marginal realism.

Graphist introduced HLG-specific metrics for graphic composition, especially the Inverse Order Pair Ratio (IOPR) for overlapping layer-order inversions and GPT-4V-based ratings for structural balance, synergy, innovation, and text readability. On the Crello HLG task, Graphist* reports SDL $\pi_1$7, SGI $\pi_1$8, SIO $\pi_1$9, STV $\pi_0$0, and $\pi_0$1 (Cheng et al., 2024). PosterO instead evaluates overlay, alignment, underlay effectiveness, intent coverage, conflict, saliency, readability, and a combined average; on the PKU unannotated split it reports $\pi_0$2 versus $\pi_0$3 for the prior state of the art, and on the CGL unannotated split $\pi_0$4 versus $\pi_0$5 (Hsu et al., 6 May 2025).

In 3D indoor generation, HLG for room construction reports OOB $\pi_0$6, ORI $\pi_0$7, CLIP-Sim $\pi_0$8, and FID $\pi_0$9, while HSM reports SceneEval scores $G=(V,E)$0 for CNT, ATR, OOR, and OAR, as well as support-region extraction IoU $G=(V,E)$1 and $G=(V,E)$2 (Wang et al., 25 Aug 2025, Pun et al., 21 Mar 2025). HOG-Layout reports highest GPT-5 semantic-plausibility mean of about $G=(V,E)$3, CLIPsim $G=(V,E)$4, average generation time $G=(V,E)$5 s, and editing under $G=(V,E)$6 s (Jiang et al., 12 Apr 2026). HDSL reports average object coverage $G=(V,E)$7 increasing from $G=(V,E)$8 to $G=(V,E)$9, CLIP from $G=(B,E)$0 to $G=(B,E)$1, time decreasing from $G=(B,E)$2 s to $G=(B,E)$3 s, and editing reductions of $G=(B,E)$4 in token use and $G=(B,E)$5 in runtime (Li et al., 8 Jun 2026).

These results suggest a recurrent empirical pattern: hierarchical formulations are typically evaluated not only on global fidelity but also on structural correctness, editability, or physical plausibility. That emphasis distinguishes HLG from flat generation benchmarks that focus only on end-task perceptual quality.

6. Conceptual ambiguities, limitations, and research directions

The main conceptual ambiguity in HLG is that the term “hierarchy” is used in several non-equivalent ways. In some papers it means coarse-to-fine latent stages, as in box-then-mask generation (Hong et al., 2018). In others it means inferred z-order over unordered elements (Cheng et al., 2024), typed containment graphs (Jin et al., 26 May 2025), intent-enriched layout trees (Hsu et al., 6 May 2025), background/object branch separation (Cheng et al., 2024), or support-region recursion in 3D (Pun et al., 21 Mar 2025, Li et al., 8 Jun 2026). The literature therefore supports treating HLG as a design principle rather than a single formalism.

The reported limitations are similarly heterogeneous. Graphist notes design homogeneity, the environmental cost of multi-stage pretraining, and the fact that the system still requires user-supplied assets (Cheng et al., 2024). The 3D HLG room-construction framework depends on the initial GPT-4o instruction, assumes a single rectangular floor plan, has finite small-object banks, and adds about $G=(B,E)$6 ms per object for refinement (Wang et al., 25 Aug 2025). The hierarchical RL formulation is restricted to two furniture items and only axis-aligned primitive moves (Di et al., 2022). MM-WebAgent explicitly has no gradient-based joint loss; its “objective” is implemented through evaluator-driven penalties inside a training-free reflection loop (Li et al., 16 Apr 2026).

Several papers also indicate likely directions for extension. COHO states that its graph pipeline is agnostic to node semantics and can be reapplied to indoor scene generation, document layouts, and game-level or terrain maps, provided that local layouts are quantized and the graph structure is preserved (He et al., 2024). Hong et al. suggest that stronger segmentation-to-image decoders or differentiable layout renderers could enable end-to-end training and that richer text embeddings could support more complex multi-sentence conditioning (Hong et al., 2018). Graphist identifies joint generation of both content and layout as future work (Cheng et al., 2024).

A plausible implication is that future HLG research will converge less on a universal architecture than on reusable interfaces between levels: editable graphs, support trees, SVG or DSL subtrees, and placeholder plans that can be generated, retrieved, verified, and revised independently. Across current work, the persistent contribution of HLG is not merely hierarchical depth, but the conversion of layout generation into a sequence of structured, inspectable decisions.