SceneHGN: Hierarchical 3D Scene Synthesis

• SceneHGN is a hierarchical graph network that models 3D indoor scenes using a four-level semantic and geometric hierarchy from room layout to object parts.
• It employs a recursive VAE with vertical and horizontal message passing, leveraging hyper-edges to enforce spatial plausibility, symmetry, and object adjacency.
• Evaluation metrics including FID, Chamfer Distance, and user studies demonstrate SceneHGN's superior performance compared to existing 3D scene synthesis methods.

SceneHGN denotes a hierarchical graph network framework for the synthesis of complete 3D indoor scenes, encompassing both the global room layout and the fine-grained part geometry of constituent furniture objects. Its principal innovation is the joint, end-to-end modeling of the full spatial and semantic hierarchy found in furnished interiors, including explicit handling of hierarchical structure, object adjacency, symmetry, and multi-object combinatorial groupings, thereby addressing the limitations of generation approaches that operate at only a single level of abstraction or fail to enforce plausible spatial and geometric constraints (Gao et al., 2023).

1. Hierarchical Scene Representation

SceneHGN encodes each room as a four-level hierarchy augmented by both vertical (parent-child) and horizontal (sibling) edges, as well as higher-order hyper-edges to reflect complex geometric relations. The four levels are:

• Level 1: Room root — a single node with a latent feature vector $h_\mathrm{room}\in\mathbb{R}^d$.
• Level 2: Functional regions — spatial clusters ($K$ per room, via DBSCAN on object centroids), each labeled by the class of its largest object (e.g., living_region, dining_region). This intermediates between the room and object levels, facilitating scalable learning.
• Level 3: Objects — within each region, nodes for each furniture object. Node features include a pretrained shape embedding from a part-aware VAE (DSG-Net), a semantic class label, and placement parameters $(c_i, s_i, \theta_i)$ (center, scale, and yaw).
• Level 4: Object parts — each object node is itself rooted to a hierarchy of geometric parts, encoded following StructureNet, with fine-grained semantic and structural grouping.

Vertical edges:

• Room $\rightarrow$ Region
• Region $\rightarrow$ Object
• Object $\rightarrow$ Part
• Skip links Room $\rightarrow$ Object to enforce $in\_room$ (object lies within boundaries) and $align\_room$ (object yaw aligned to room axes).

Horizontal (sibling) edges:

• At the object level: adjacency, translational/reflective/2-fold rotational symmetry.
• Hyper-edges: parallel collinearity (all axes parallel and centers collinear), $N$-fold rotational symmetry.

These relations are leveraged both at encoding (by message-passing and aggregation) and decoding (explicit prediction and loss).

2. Graph Neural Network Architecture

SceneHGN is formulated as a conditional recursive variational autoencoder (VAE). The architecture is decomposed as follows:

• FloorNet: A standalone VAE encodes the 2D room boundary (596-vertex closed loop, mapped to a unit-square ring) into a condition vector $K$0. Boundary features include ACAP descriptors per vertex, processed by a GCN and fully connected layers to output $K$1, with a KL-divergence objective to $K$2.
• Scene Encoder:
  • Part$K$3object: Uses DSG-Net’s part encoder to aggregate per-part features into object-level embeddings.
  • Object$K$4region$K$5room: For each object $K$6, constructs

  $K$7

  Sibling message-passing for each binary and hyper-edge type, using edge-specific weights $K$8 and hyper-edge transforms. Node updates:

  $K$9

  Region nodes aggregate child object features; the room root node aggregates region features, yielding the global latent code $(c_i, s_i, \theta_i)$0.

• Scene Decoder:

  • Child existence masks (up to 10 per node)
  • Child semantic classes
  • For objects: placement $(c_i, s_i, \theta_i)$3
  • Sibling binary/hyper-edge structure
  • Final features forwarded for part-gen by DSG-Net.

Each graph level is parameterized by dedicated weights per edge/hyper-edge type. Root and boundary conditions ensures plausibility and floor alignment.

3. Fine-Grained Geometry and Loss Formulation

The geometric detail is encoded at the part level, where each part stores an ACAP deformation gradient $(c_i, s_i, \theta_i)$4, center $(c_i, s_i, \theta_i)$5, and part-tree structure. DecpG induces part features $(c_i, s_i, \theta_i)$6 and reconstructs mesh geometry. Part meshes are synthesized by applying predicted deformations to part templates.

Total loss for end-to-end training combines:

• KL Losses for both scene and boundary latent codes:

$(c_i, s_i, \theta_i)$7

• Reconstruction losses:
  • L2 on part gradients/centers, cross-entropy on existence/class, CE on binary/hyper-edge structure;
  • Room-object placement:

  $(c_i, s_i, \theta_i)$8 - Room-object skip-link loss: Chamfer matching of OBB normals post-rotation

• Hyper-edge losses comprise group symmetry/parallelism.

• Aggregate loss:

$(c_i, s_i, \theta_i)$9

4. Training Protocol and Datasets

• Datasets:

  • 3D-FRONT: 18,797 furnished rooms with room/furniture annotation.
  • PartNet: Part hierarchies for 24 object classes; used to substitute finer parts into 3D-FRONT.
• Partition: 80% train, 20% test per room type.
• Two-stage training:

1. Object-part modules (DSG-Net) trained as a VAE on PartNet independently.
2. Full scene encoder and decoder are trained, freezing or lightly fine-tuning DSG-Net.
3. FloorNet is trained separately.

• Optimization: Adam optimizer with learning rate $\rightarrow$0, batch size 64, $\rightarrow$1 epochs. KL regularization, skip-link, hyper-edge consistency losses are active throughout.

5. Experimental Evaluation

Quantitative metrics:

• FID on rendered room layouts versus ground truth
• EMD of object-category global histograms ($\rightarrow$2) and per-room type ($\rightarrow$3)
• Pairwise co-occurrence EMD ($\rightarrow$4)
• XY-plane heat-map correlation $\rightarrow$5 for object offsets
• Orientation score: $\rightarrow$6
• Reconstruction: Chamfer Distance (CD; 100k pts), Earth Mover’s Distance (EMD; 10k pts).

Comparative performance:

• Outperforms GRAINS, DeepPriors, and ATISS on FID, $\rightarrow$7, $\rightarrow$8, $\rightarrow$9, and matches or exceeds orientation score.
• Reconstruction: SceneHGN CD $\rightarrow$0 vs. OccNet $\rightarrow$1.

User Study:

On 100 random room boundaries, SceneHGN is preferred as best layout/coordinated/overall in $\rightarrow$2 of cases.

Ablation findings:

• Removing the functional-region level yields scenes missing objects.
• Dropping skip-links causes misaligned furniture.
• Removing object-object or hyper-edges degrades symmetry and parallelism.
• Two-stage (DSG-Net pretraining) yields better convergence and lower CD than end-to-end from scratch.

6. Applications and Limitations

Applications

• Room editing at any semantic level:

Given a scene, local edits (e.g. translating, rotating, deforming an object or part) can be propagated via encoding, gradient-based latent code adjustment:

$\rightarrow$3

so the scene globally adapts, retaining plausibility.

• Scene interpolation:

Interpolating in $\rightarrow$4 yields smooth morphing between rooms, object layouts, and part geometries.

• Conditional generation:

Given only a set of 3D object bounding boxes (OBBs) and a room boundary, SceneHGN can infer semantic labels and mesh geometries for a complete room.

• Scene completion:

After removal of objects or parts, encoding and decoding reconstructs plausible replacements.

Limitations and Open Directions

• Requires rich part annotations from 3D-FRONT and PartNet; coverage of small objects (e.g. cups, books) remains low.
• Doors and windows are not explicitly handled (vertical edges not specialized for openings).
• No explicit collision-free loss, resulting in occasional interpenetrations.
• Texture and appearance modalities are absent; model operates on pure geometry.
• Prospective directions include explicit handling of openings, material/texture synthesis, and tighter physics-aware collision constraints.

7. Significance and Context

SceneHGN is the first generative model that can synthesize the entirety of a 3D furnished room—including joint modeling of room layout, object placement, and part-level mesh geometry—from arbitrary 2D boundary input. The hierarchical graph formalism supports semantically-meaningful structural manipulations, scene completion, and interpolation in a unified VAE framework. The functional-region abstraction enables scalable synthesis for large rooms; binary/hyper-edge constraints effectively capture the physical and design principles underlying plausible interiors (Gao et al., 2023). This comprehensive approach substantially advances generative modeling in 3D scene synthesis, particularly in fidelity, structural correctness, and flexibility.