---
title: Hierarchical Layout Generation
url: https://www.emergentmind.com/topics/hierarchical-layout-generation-hlg
type: topic
---

# Hierarchical Layout Generation

Hierarchical Layout Generation (HLG) denotes a family of generative formulations in which layout prediction is decomposed across multiple structural levels rather than learned as a flat, one-shot mapping. In published work, those levels have been instantiated as semantic layout stages for text-to-image synthesis, unordered graphic elements with inferred layer order, typed graphs for user interfaces and cities, SVG trees for posters, branch-structured conditioning for diffusion, and room–support–object hierarchies for 3D scenes [1801.05091, 2404.14368, 2407.11294, 2410.14324, 2505.19554, 2508.17832]. The unifying idea is that explicit intermediate structure—such as boxes, masks, relation matrices, support regions, or placeholders—can preserve semantics, improve controllability, and reduce geometric or physical violations while exposing interpretable states for editing and partial specification.

## 1. Emergence of the hierarchical formulation

One early and influential decomposition appears in text-to-image synthesis, where semantic layout is inferred before pixel rendering. Hong et al. model the latent layout as bounding boxes and object masks, factorizing
$$
p(L \mid T) = p(B_{1:T} \mid T)\cdot p(M_{1:T} \mid B_{1:T}, T),
$$
so that the generator first predicts object boxes and then refines each box into a binary mask [1801.05091]. This formulation separates “what and where” from later image synthesis, and it explicitly supports user intervention by editing the generated scene layout before rerendering.

The term HLG was later made explicit in graphic design. Graphist defines HLG over an unordered set of RGBA design elements \(M=\{M_1,\dots,M_n\}\), predicting a composition \(S=\{s_1,\dots,s_n\}\) with
\(s_i=(x_i,y_i,w_i,h_i,l_i)\), where \(l_i\) is the layer index or z-order [2404.14368]. That formalization marks a decisive shift from earlier Graphic Layout Generation (GLG), which assumed that the input order was already correct. In HLG, the system must infer both placement and hierarchy from an unordered set.

A parallel line of work treats hierarchy as multi-level decision making rather than sequence factorization. In virtual indoor scenes, a two-furniture layout problem was cast as an MDP and solved with hierarchical actor-critic, using a manager policy \(\pi_1\) that proposes sub-goals and a worker policy \(\pi_0\) that executes primitive actions [2210.10431]. This establishes that HLG is not tied to a single model family: it can be framed as latent-variable generation, structured prediction, or hierarchical control.

The literature therefore does not present HLG as one canonical architecture. Instead, it uses hierarchy to reduce the complexity of layout reasoning, to impose structure on multimodal conditioning, or to expose controllable intermediate variables. That breadth is visible in later work spanning UI design, urban generation, posters, slides, webpages, and 3D indoor scenes [2407.11294, 2505.07843, 2506.07964, 2604.15309].

## 2. Structural representations used in HLG

The primary technical divergence across HLG systems lies in how hierarchy is represented.

| Domain | Representation | Hierarchical unit |
|---|---|---|
| UI layout [2505.19554] | Directed attributed graph \(G=(V,E)\) | `contain`, `parallel`, `top`, `left` edges |
| City-scale urban layout [2407.11294] | Canonical graph \(G=(B,E)\) | blocks, communities, city context |
| Graphic design [2404.14368] | JSON sequence over unordered RGBA patches | spatial transforms plus layer index \(l_i\) |
| Poster design [2505.07843] | SVG-style tree \(T\) | parent–child enclosure and intent nodes |
| Layout-to-image [2410.14324] | \(K+1\) conditioning branches | background level and per-object levels |
| 3D indoor scenes [2508.17832, 2503.16848, 2606.09738] | scene graphs, support trees, XML/CSS trees | room, zones/surfaces, objects |

In UI generation, ASR models each layout as a directed attributed graph \(G=(V,E)\). Each node carries a category label from \(\{\text{BACKGROUND}, \text{IMAGE}, \text{TEXT}, \text{INPUT}, \text{SLIDING\_BAR}, \text{ICON}\}\) and a 4-D coordinate vector \([x_i,y_i,w_i,h_i]\); edges encode positional relations (“top”, “left”) and semantic relations (“contain”, “parallel”) [2505.19554]. The hierarchy is induced by `contain` edges, while sibling structure and local order are represented by `parallel`, `top`, and `left`.

In COHO, the hierarchy is city-scale and contextual rather than containment-based. The city is represented as a graph over city-block nodes, each with a 4-D shape/location descriptor \(s_i\) and a 512-D quantized code \(q_i\) describing the internal building layout, while edges encode adjacency and distance between blocks [2407.11294]. The same graph simultaneously captures neighboring blocks, communities, and city-level context.

PosterO uses an SVG tree whose leaves include both element nodes and design-intent nodes. Every node carries a shape tag from \(\{\text{rect}, \text{v-rect}, \text{rot-rect}, \text{ellipse}, \text{path}\}\), normalized geometric parameters, and, for intent nodes, a latent embedding from a U-Net-style intent model [2505.07843]. Parent–child grouping is derived by enclosure tests, and absolute coordinates are recovered by accumulating parent offsets down the tree.

In 3D indoor generation, hierarchy is usually support-centric. HLG for room construction defines a scene \(S=(O,D,C)\), assigns each object to a parent, and decomposes constraints across levels \(C_\ell\) such that \(C_i \cap C_j = \emptyset\) for \(i\neq j\) [2508.17832]. HSM similarly models a rooted tree \(H=(\mathcal{N},\mathcal{E})\) of support regions and object-placement nodes, with factorization over levels and parents [2503.16848]. HDSL turns that principle into an explicit XML/CSS-style DSL in which nesting simultaneously represents containment and support, and coordinates are local to the parent frame [2606.09738].

These formalisms indicate that “hierarchy” in HLG is not restricted to depth order. It may refer to containment, support, spatial scale, semantic intent, or disentangled object-conditioned branches.

## 3. Core generation mechanisms

Once a hierarchy is defined, HLG systems differ in how they propagate information through it.

ASR uses a 5-layer GNN \(\mathscr{F}_G\) to aggregate node and relational features, producing a graph-level code \(h_a\in\mathbb{R}^{1024}\). It then decodes relation matrices \(M_{sem}\) and \(M_{pos}\), replaces the traditional ViT module inside a multimodal LLM with graph-derived features, and has the LLM emit a JSON record for each node containing category, coordinates, and relations [2505.19554]. The crucial design choice is that all LLM cross-attention layers operate on hierarchical graph features rather than raw pixels.

COHO takes a masked autoencoding route. A graph-based masked autoencoder (GMAE) hides only the 512-D building-layout codes \(Q=\{q_i\}\), keeping shape/location \(S\) and edges \(E\) visible, and reconstructs the masked codes with a cross-entropy objective over quantized dimensions [2407.11294]. Generation is not purely one-shot: COHO uses \(T=12\) scheduled iterative sampling steps and admits the top \(\beta(t)=1-\cos(t/T)\) fraction of confident blocks at each step, so semantically important regions are fixed early and later predictions inherit their style.

HiCo embeds hierarchy inside a diffusion conditioning architecture. It augments a pretrained latent diffusion UNet with \(K+1\) side branches: one global branch for background and \(K\) local branches for foreground objects. Fusion is mask-based, with branch outputs gated by binary masks and summed into a conditioning signal [2410.14324]. This makes the hierarchy object-separable rather than graph-structured, and the intended effect is spatial disentanglement.

Graphist and PosterO both reframe HLG as sequence generation, but with different structured outputs. Graphist serializes layout as JSON tokens and trains a large multimodal model on an autoregressive objective \(\log p(Y\mid M)\), using an RGBA encoder and a “Visual Shrinker” to reduce visual token length [2404.14368]. PosterO instead converts layouts into SVG trees and uses in-context learning: examples are selected by nearest intent embedding, then an LLM predicts a new layout tree aligned to the test image’s intent region [2505.07843].

SlideCoder and MM-WebAgent represent another shift: hierarchy becomes task decomposition for agentic code generation. SlideCoder uses three cascading levels—coarse visual segmentation, block-level description and snippet generation, and global assembly—under a Hierarchical Retrieval-Augmented Generation pipeline with a Shape Type Knowledge Base and an Operation Function Knowledge Base [2506.07964]. MM-WebAgent is explicitly training-free: it produces a global layout plan \(L_g\), local placeholder plans, a code skeleton, and then runs a three-level self-reflection loop over local assets, HTML/CSS context, and full-page global coherence [2604.15309].

In 3D, generation is commonly coupled with optimization. HLG for room construction first extracts scene information, creates a coarse room, then applies Fine-Grained Layout Alignment and a Trainable Layout Optimization Network with ownership and pose heads [2508.17832]. HOG-Layout interleaves LLM planning, VLM-based group-wise placement, retrieval-augmented asset grounding, and a hierarchical force-directed optimization objective over collision, support, boundary, adjacency, and alignment terms [2604.10772]. HDSL similarly follows recursive subtree generation with bounded verification, multimodal retrieval, and force-directed layout optimization [2606.09738].

## 4. Controllability, editing, and human-centric interaction

A defining feature of many HLG systems is that their intermediate states are directly editable.

ASR makes the intermediate relation matrices \(M_{sem}\) and \(M_{pos}\) human-editable. Its pipeline accepts a user-provided or randomly masked skeleton graph, decodes relation matrices, optionally allows the user to edit them, and then feeds the edited structure into the MLLM to generate the final node list [2505.19554]. The paper states that designers can progressively “push” new contain, parallel, or positional constraints and obtain LLM-driven reflows that respect the edited hierarchy.

Hong et al. exposed the same principle in an earlier form by separating layout from rendering: because boxes \(B_t\) and masks \(M_t\) are explicit, users can add, remove, or move objects and rerun the later stages [1801.05091]. PosterO extends editability to SVG realization, where the predicted layout tree is already valid SVG and can be further realized into posters by editing the chat with LLMs [2505.07843].

In webpage generation, MM-WebAgent uses placeholders \(p_{j,i}\) and inherited style attributes \(\phi_j\) to coordinate downstream asset generation. Editing is indirect but systematic: local refine fixes individual assets, context refine patches integration errors such as clipping or overflow, and global refine adjusts layout and style consistency after inspecting the full rendered page [2604.15309]. The hierarchy is therefore operationalized as a refinement schedule.

In 3D editing, hierarchy becomes a localization device. HOG-Layout supports Add, Move, and Delete operations by updating only the relevant parent–child relations and then re-running optimization [2604.10772]. HDSL formalizes localized editing more aggressively through Hierarchical Retrieval-Augmented Generation: it retrieves a relevant subtree by path-aware embeddings, asks the LLM to rewrite only that fragment, and merges it back with a deterministic three-way merge before reapplying force-directed refinement [2606.09738].

This body of work rejects the misconception that hierarchy is only a regularizer for training. In HLG, hierarchy is often the mechanism by which control is exposed to human designers or to downstream editing agents.

## 5. Evaluation protocols and empirical patterns

HLG systems are evaluated with domain-specific metrics, and the diversity of those protocols reflects the diversity of tasks.

In UI generation on RICO, ASR reports for direct UI generation \( \text{mIoU}=0.63\), \( \text{RE}=0.27\), \( \text{OL}=52.43\), and \( \text{FID}=23.08\). On completion, it reports \( \text{mIoU}=0.70\) and \( \text{RE}=0.24\), and on graph-editing \( \text{mIoU}=0.68\) and \( \text{RE}=0.09\). In a crowd-sourced AMT study with \(N=50\), reasonableness scores were \(4.30\) for easy layouts, \(3.66\) for medium layouts, and \(3.39\) for hard layouts [2505.19554].

COHO evaluates urban layouts with Context Score (CTS), WD-5D, WD-CO, Overlap, O-Blk, FID, KID, and LPIPS. It reports the lowest WD-5D at \(2.28\), the lowest WD-CO at \(1.91\), and \( \text{CTS}\approx 0.21\), with best FID/KID/LPIPS among the listed baselines across 330 heterogeneous US cities [2407.11294]. The inclusion of CTS is notable because it directly measures harmonization with neighboring blocks rather than only marginal realism.

Graphist introduced HLG-specific metrics for graphic composition, especially the Inverse Order Pair Ratio (IOPR) for overlapping layer-order inversions and GPT-4V-based ratings for structural balance, synergy, innovation, and text readability. On the Crello HLG task, Graphist* reports SDL \(5.85\), SGI \(6.90\), SIO \(5.10\), STV \(5.24\), and \( \text{IOPR}_{avg}=0.97\) [2404.14368]. PosterO instead evaluates overlay, alignment, underlay effectiveness, intent coverage, conflict, saliency, readability, and a combined average; on the PKU unannotated split it reports \( \text{Avg}\downarrow = 0.0470\) versus \(0.1434\) for the prior state of the art, and on the CGL unannotated split \(0.0203\) versus \(0.4118\) [2505.07843].

In 3D indoor generation, HLG for room construction reports OOB \(16.2\), ORI \(89.2\), CLIP-Sim \(29.3\), and FID \(38.2\), while HSM reports SceneEval scores \((48.3, 47.1, 22.6, 41.1)\) for CNT, ATR, OOR, and OAR, as well as support-region extraction IoU \(62.5\) and \(F_1@0.5 = 54.2\) [2508.17832, 2503.16848]. HOG-Layout reports highest GPT-5 semantic-plausibility mean of about \(70\), CLIPsim \(69.7\), average generation time \(18.6\) s, and editing under \(25\) s [2604.10772]. HDSL reports average object coverage \(NObj\) increasing from \(43.15\) to \(58.19\), CLIP from \(19.81\) to \(20.99\), time decreasing from \(552\) s to \(383\) s, and editing reductions of \(5.22\times\) in token use and \(6.19\times\) in runtime [2606.09738].

These results suggest a recurrent empirical pattern: hierarchical formulations are typically evaluated not only on global fidelity but also on structural correctness, editability, or physical plausibility. That emphasis distinguishes HLG from flat generation benchmarks that focus only on end-task perceptual quality.

## 6. Conceptual ambiguities, limitations, and research directions

The main conceptual ambiguity in HLG is that the term “hierarchy” is used in several non-equivalent ways. In some papers it means coarse-to-fine latent stages, as in box-then-mask generation [1801.05091]. In others it means inferred z-order over unordered elements [2404.14368], typed containment graphs [2505.19554], intent-enriched layout trees [2505.07843], background/object branch separation [2410.14324], or support-region recursion in 3D [2503.16848, 2606.09738]. The literature therefore supports treating HLG as a design principle rather than a single formalism.

The reported limitations are similarly heterogeneous. Graphist notes design homogeneity, the environmental cost of multi-stage pretraining, and the fact that the system still requires user-supplied assets [2404.14368]. The 3D HLG room-construction framework depends on the initial GPT-4o instruction, assumes a single rectangular floor plan, has finite small-object banks, and adds about \(50\) ms per object for refinement [2508.17832]. The hierarchical RL formulation is restricted to two furniture items and only axis-aligned primitive moves [2210.10431]. MM-WebAgent explicitly has no gradient-based joint loss; its “objective” is implemented through evaluator-driven penalties inside a training-free reflection loop [2604.15309].

Several papers also indicate likely directions for extension. COHO states that its graph pipeline is agnostic to node semantics and can be reapplied to indoor scene generation, document layouts, and game-level or terrain maps, provided that local layouts are quantized and the graph structure is preserved [2407.11294]. Hong et al. suggest that stronger segmentation-to-image decoders or differentiable layout renderers could enable end-to-end training and that richer text embeddings could support more complex multi-sentence conditioning [1801.05091]. Graphist identifies joint generation of both content and layout as future work [2404.14368].

A plausible implication is that future HLG research will converge less on a universal architecture than on reusable interfaces between levels: editable graphs, support trees, SVG or DSL subtrees, and placeholder plans that can be generated, retrieved, verified, and revised independently. Across current work, the persistent contribution of HLG is not merely hierarchical depth, but the conversion of layout generation into a sequence of structured, inspectable decisions.

Source: https://www.emergentmind.com/topics/hierarchical-layout-generation-hlg