---
title: 'SceneHGN: Hierarchical 3D Scene Synthesis'
url: https://www.emergentmind.com/topics/scenehgn
type: topic
---

# SceneHGN: Hierarchical 3D Scene Synthesis

SceneHGN denotes a hierarchical graph network framework for the synthesis of complete 3D indoor scenes, encompassing both the global room layout and the fine-grained part geometry of constituent furniture objects. Its principal innovation is the joint, end-to-end modeling of the full spatial and semantic hierarchy found in furnished interiors, including explicit handling of hierarchical structure, object adjacency, symmetry, and multi-object combinatorial groupings, thereby addressing the limitations of generation approaches that operate at only a single level of abstraction or fail to enforce plausible spatial and geometric constraints [2302.10237].

## 1. Hierarchical Scene Representation

SceneHGN encodes each room as a four-level hierarchy augmented by both vertical (parent-child) and horizontal (sibling) edges, as well as higher-order hyper-edges to reflect complex geometric relations. The four levels are:

- **Level 1: Room root** — a single node with a latent feature vector $h_\mathrm{room}\in\mathbb{R}^d$.
- **Level 2: Functional regions** — spatial clusters ($K$ per room, via DBSCAN on object centroids), each labeled by the class of its largest object (e.g., living_region, dining_region). This intermediates between the room and object levels, facilitating scalable learning.
- **Level 3: Objects** — within each region, nodes for each furniture object. Node features include a pretrained shape embedding from a part-aware VAE (DSG-Net), a semantic class label, and placement parameters $(c_i, s_i, \theta_i)$ (center, scale, and yaw).
- **Level 4: Object parts** — each object node is itself rooted to a hierarchy of geometric parts, encoded following StructureNet, with fine-grained semantic and structural grouping.

**Vertical edges:**  
- Room $\rightarrow$ Region  
- Region $\rightarrow$ Object  
- Object $\rightarrow$ Part  
- Skip links Room $\rightarrow$ Object to enforce $in\_room$ (object lies within boundaries) and $align\_room$ (object yaw aligned to room axes).

**Horizontal (sibling) edges:**  
- At the object level: adjacency, translational/reflective/2-fold rotational symmetry.
- Hyper-edges: parallel collinearity (all axes parallel and centers collinear), $N$-fold rotational symmetry.

These relations are leveraged both at encoding (by message-passing and aggregation) and decoding (explicit prediction and loss).

## 2. Graph Neural Network Architecture

SceneHGN is formulated as a conditional recursive variational autoencoder (VAE). The architecture is decomposed as follows:

- **FloorNet**: A standalone VAE encodes the 2D room boundary (596-vertex closed loop, mapped to a unit-square ring) into a condition vector $h_b$. Boundary features include ACAP descriptors per vertex, processed by a GCN and fully connected layers to output $(\mu_b,\sigma_b)$, with a KL-divergence objective to $N(0,I)$.
- **Scene Encoder:**  
  - **Part$\to$object**: Uses DSG-Net’s part encoder to aggregate per-part features into object-level embeddings.
  - **Object$\to$region$\to$room**: For each object $O_i$, constructs
    \[
    x_i = \left[f^{\rm shape}_i \;|\; l_i\;|\; p_i\right], \quad h_i^{(0)} =  \text{MLP}_{\rm obj}(x_i)
    \]
    Sibling message-passing for each binary and hyper-edge type, using edge-specific weights $W^{(e)}$ and hyper-edge transforms. Node updates:
    \[
    h_i^{(1)} = \sigma(W_{\text{self}}h_i^{(0)} + m_i + b)
    \]
    Region nodes aggregate child object features; the room root node aggregates region features, yielding the global latent code $(\mu_s, \sigma_s)$.
- **Scene Decoder:**  
  Given latent sample $z\sim N(\mu_s,\sigma_s)$ concatenated with $h_b$ (boundary), decoding proceeds top-down. Each parent predicts:
  - Child existence masks (up to 10 per node)
  - Child semantic classes
  - For objects: placement $(\hat{c}, \hat{s}, \hat{\theta})$
  - Sibling binary/hyper-edge structure
  - Final features forwarded for part-gen by DSG-Net.

Each graph level is parameterized by dedicated weights per edge/hyper-edge type. Root and boundary conditions ensures plausibility and floor alignment.

## 3. Fine-Grained Geometry and Loss Formulation

The geometric detail is encoded at the part level, where each part stores an ACAP deformation gradient $X_p \in \mathbb{R}^{3 \times 3}$, center $c_p\in\mathbb{R}^3$, and part-tree structure. DecpG induces part features $f_p$ and reconstructs mesh geometry. Part meshes are synthesized by applying predicted deformations to part templates.

Total loss for end-to-end training combines:

- **KL Losses** for both scene and boundary latent codes:
  \[
  \mathcal{L}_{KL} = KL(\mathcal{N}(\mu_s, \sigma_s)\|\mathcal{N}(0,I)) + KL(\mathcal{N}(\mu_b, \sigma_b)\|\mathcal{N}(0,I))
  \]
- **Reconstruction losses:**
  - L2 on part gradients/centers, cross-entropy on existence/class, CE on binary/hyper-edge structure;
  - Room-object placement:
    \[
    \mathcal{L}_{\text{place}} = \|c - \hat{c}\|_2 + \|s - \hat{s}\|_2 + \|\mathrm{Angle}[\arg\max p] + b - \theta\|_2
    \]
  - Room-object skip-link loss: Chamfer matching of OBB normals post-rotation
- **Hyper-edge losses** comprise group symmetry/parallelism.
- **Aggregate loss**:
  \[
  \mathcal{L} = \mathbb{E}_{z,b}\left[\mathcal{L}_{\text{rec}}(\hat{S}, S)\right] + \lambda_{KL} \mathcal{L}_{KL} + \lambda_{\text{hyp}} \mathcal{L}_{\text{hyp}}
  \]

## 4. Training Protocol and Datasets

- **Datasets:**  
  - **3D-FRONT**: 18,797 furnished rooms with room/furniture annotation.
  - **PartNet:** Part hierarchies for 24 object classes; used to substitute finer parts into 3D-FRONT.
- Partition: 80% train, 20% test per room type.
- **Two-stage training:**  
  1. Object-part modules (DSG-Net) trained as a VAE on PartNet independently.
  2. Full scene encoder and decoder are trained, freezing or lightly fine-tuning DSG-Net.
  3. FloorNet is trained separately.
- **Optimization:** Adam optimizer with learning rate $10^{-3}$, batch size 64, $~500$ epochs. KL regularization, skip-link, hyper-edge consistency losses are active throughout.

## 5. Experimental Evaluation

**Quantitative metrics:**
- **FID** on rendered room layouts versus ground truth
- EMD of object-category global histograms ($o_1$) and per-room type ($o_2$)
- Pairwise co-occurrence EMD ($o_3$)
- XY-plane heat-map correlation $(o_4)$ for object offsets
- **Orientation score:** $\frac{1}{M} \sum \cos^2\left(2(\theta - \hat{\theta})\right)$
- **Reconstruction:** Chamfer Distance (CD; 100k pts), Earth Mover’s Distance (EMD; 10k pts).

**Comparative performance:**
- Outperforms GRAINS, DeepPriors, and ATISS on FID, $o_1$, $o_2$, $o_3$, and matches or exceeds orientation score.
- Reconstruction: SceneHGN CD $\approx 3.11 \times 10^{-3}$ vs. OccNet $5.95 \times 10^{-3}$.

**User Study:**  
On 100 random room boundaries, SceneHGN is preferred as best layout/coordinated/overall in $\sim70\%$ of cases.

**Ablation findings:**  
- Removing the functional-region level yields scenes missing objects.
- Dropping skip-links causes misaligned furniture.
- Removing object-object or hyper-edges degrades symmetry and parallelism.
- Two-stage (DSG-Net pretraining) yields better convergence and lower CD than end-to-end from scratch.

## 6. Applications and Limitations

### Applications
- **Room editing at any semantic level:**  
  Given a scene, local edits (e.g. translating, rotating, deforming an object or part) can be propagated via encoding, gradient-based latent code adjustment:
  \[
  \min_z \|z-z_0\|^2 + \alpha\,d_\mathrm{CD}\left(T(B_e)U,T(B_t)U\right) + \beta\,\mathcal{L}_{\rm struc}(D(z))
  \]
  so the scene globally adapts, retaining plausibility.
- **Scene interpolation:**  
  Interpolating in $(z, h_b)$ yields smooth morphing between rooms, object layouts, and part geometries.
- **Conditional generation:**  
  Given only a set of 3D object bounding boxes (OBBs) and a room boundary, SceneHGN can infer semantic labels and mesh geometries for a complete room.
- **Scene completion:**  
  After removal of objects or parts, encoding and decoding reconstructs plausible replacements.

### Limitations and Open Directions
- Requires rich part annotations from 3D-FRONT and PartNet; coverage of small objects (e.g. cups, books) remains low.
- Doors and windows are not explicitly handled (vertical edges not specialized for openings).
- No explicit collision-free loss, resulting in occasional interpenetrations.
- Texture and appearance modalities are absent; model operates on pure geometry.
- Prospective directions include explicit handling of openings, material/texture synthesis, and tighter physics-aware collision constraints.

## 7. Significance and Context

SceneHGN is the first generative model that can synthesize the entirety of a 3D furnished room—including joint modeling of room layout, object placement, and part-level mesh geometry—from arbitrary 2D boundary input. The hierarchical graph formalism supports semantically-meaningful structural manipulations, scene completion, and interpolation in a unified VAE framework. The functional-region abstraction enables scalable synthesis for large rooms; binary/hyper-edge constraints effectively capture the physical and design principles underlying plausible interiors [2302.10237]. This comprehensive approach substantially advances generative modeling in 3D scene synthesis, particularly in fidelity, structural correctness, and flexibility.

Source: https://www.emergentmind.com/topics/scenehgn