Hierarchical Scene Graph Encoder (HSGE)

• HSGE is an approach that encodes perceptual observations into hierarchical, multiscale representations capturing semantic, geometric, and functional relations.
• It employs methods like transformer encoders, relational GCNs, and spectral clustering to progressively refine scene structure for improved task performance.
• Empirical evidence shows HSGE enhances room/region classification, planning efficiency, and graph quality in diverse 3D navigation and localization applications.

A Hierarchical Scene Graph Encoder (HSGE) is a family of scene-representation mechanisms that encodes perceptual observations, or an already constructed scene graph, into multi-level, structure-aware representations organized across scales such as room, region, object, functional part, or navigation topology. In the recent literature, the term is used explicitly in navigation, most notably in SAGE-Nav, and is also used implicitly as a design pattern in transformer-based 3D scene understanding, online hierarchical 3D scene graph construction, hyperbolic scene-graph embedding, and functional 3D scene graph inference (Su et al., 24 Jun 2026). This suggests that HSGE is best understood as an architectural role—hierarchical encoding of scene structure—rather than a single fixed module.

1. Genealogy and scope

Earlier scene-graph work established several ingredients later absorbed into HSGE-style systems. Hierarchical relation heads factorized predicate prediction into super-categories and fine relations (Jiang et al., 2023), hierarchical commonsense validation added a coarse-to-fine relation hierarchy plus language-model filtering (Jiang et al., 2023), and robust scene graph generation introduced hierarchical knowledge graphs that refine predictions from coarse initial estimates to detailed predictions (Zhang et al., 2024). Human-mimetic scene graph generation encoded a Hierarchical Entity Tree with a Hybrid-LSTM to model hierarchy and sibling context (Wang et al., 2020), while HIG extended hierarchy to video by encoding frame-wise graphs into progressively more abstract temporal graph cells (Nguyen et al., 2023). HOSE-Net moved hierarchy into the classifier space through structure-aware embedding-to-classifier mapping and hierarchical semantic aggregation (Wei et al., 2020).

In 3D robotics and embodied AI, the concept became more operational. TB-HSU describes a transformer that encodes object-level inputs into a room–region–object 3DHSG and can be read as a concrete HSGE design pattern (Xu et al., 2024). SAGE-Nav names its module explicitly and uses it to convert a hierarchical scene graph plus an active waypoint into a goal-conditioned embedding for low-level control (Su et al., 24 Jun 2026). Other systems play the same role algorithmically rather than with an explicit neural module: object–zone–region HSG construction for semantic navigation (Wu et al., 30 Jun 2026), hierarchical graph matching for localization (Ndulue et al., 30 Apr 2026), hierarchical traversable 3DSGs for navigation among movable obstacles (Wang et al., 17 Dec 2025), and open-vocabulary functional 3D scene graph inference with temporal graph optimization and hierarchy shaping (Hu et al., 15 May 2026). This suggests that the term now spans learned encoders, optimization pipelines, and hybrid symbolic–neural systems.

2. Representational hierarchy

The central design choice in HSGE is the hierarchy itself. TB-HSU defines a 3-layer directed acyclic graph with room nodes $\mathcal{N}_3$, region nodes $\mathcal{N}_2$, and object nodes $\mathcal{N}_1$, where each object belongs to exactly one region and each region belongs to exactly one room (Xu et al., 2024). In online semantic navigation, the hierarchy is object–zone–region, with $V_t = O_t \cup Z_t \cup R_t$, intra-layer adjacency edges, and inter-layer containment edges (Wu et al., 30 Jun 2026). SAGE-Nav uses object-level nodes together with higher-level cluster nodes representing rooms, regions, or semantic clusters, linked by intra-level spatial edges and inter-level hierarchical edges (Su et al., 24 Jun 2026).

Other works broaden the notion of hierarchy beyond room-scale semantics. HERO organizes a traversable 3DSG across building, floor, room, object, and navigation layers, then augments the graph with traversability edges through movable obstacles (Wang et al., 17 Dec 2025). Functional indoor graphs define $V = O \cup C \cup U$, where $O$ are objects, $C$ functional carriers, and $U$ interactive units, and then recover either $O \leftarrow C \leftarrow U$ or $O \leftarrow U$ structures through global hierarchy shaping (Hu et al., 15 May 2026). Hi-Dyna Graph decomposes the world into a global static graph $\mathcal{N}_2$0 and a local dynamic graph $\mathcal{N}_2$1, then anchors the dynamic subgraphs to the static topology using semantic and spatial constraints (Hou et al., 30 May 2025). A different form of internal hierarchy appears in object-centric 3DSG geometry, where each object node carries a four-layer geometric stack: point cloud, dense mesh, superquadrics, and inflated superquadrics (Zhang et al., 1 Jun 2026).

The literature therefore uses hierarchy in at least four senses: semantic containment, functional decomposition, geometric abstraction, and traversability structure. This suggests that HSGE is less a commitment to one ontology than a commitment to explicitly encoding cross-scale dependencies.

3. Encoder architectures and mathematical mechanisms

One influential HSGE pattern is the transformer encoder over object tokens. TB-HSU embeds each object with a semantic embedding and a positional embedding,

$\mathcal{N}_2$2

prepends a learnable room token, and forms

$\mathcal{N}_2$3

It then applies a standard transformer stack,

$\mathcal{N}_2$4

followed by

$\mathcal{N}_2$5

with the class token decoded for room classification and the object tokens decoded for region-specific affordance labels. Training uses a multi-task loss

$\mathcal{N}_2$6

with adaptive $\mathcal{N}_2$7 defined from the relative room and region losses (Xu et al., 2024). Here, the hierarchy is induced by token roles and supervision, not by explicit graph message passing.

SAGE-Nav implements an explicitly named HSGE with a relational GCN. Each node starts from

$\mathcal{N}_2$8

where $\mathcal{N}_2$9 is a 512-D CLIP semantic embedding and $\mathcal{N}_1$0 a 32-D sinusoidal positional encoding of the 3D coordinate. At each abstraction level $\mathcal{N}_1$1, HSGE applies

$\mathcal{N}_1$2

then conditions the representation on the active waypoint $\mathcal{N}_1$3 via

$\mathcal{N}_1$4

Average pooling across levels and an MLP yield the final waypoint embedding

$\mathcal{N}_1$5

which is 64-dimensional in the implementation (Su et al., 24 Jun 2026).

A second HSGE family is algorithmic rather than parametric. In online semantic navigation, the encoder builds an affinity matrix over object adjacency,

$\mathcal{N}_1$6

then applies spectral clustering first to obtain zones and again to obtain regions (Wu et al., 30 Jun 2026). In functional 3D scene graphs, node association is performed across frames in 3D using multiple cues, while edge association is cast as temporal graph optimization with evidence accumulation, entropy regularization, and temporal smoothing before hierarchy shaping (Hu et al., 15 May 2026).

A third family changes the geometry of the embedding space itself. HSG embeds scene-graph nodes in the Lorentz model of hyperbolic space,

$\mathcal{N}_1$7

uses geodesic distance

$\mathcal{N}_1$8

and trains with hyperbolic contrastive learning plus entailment cones that enforce place $\mathcal{N}_1$9 object hierarchy (Wang et al., 19 Apr 2026). This reframes HSGE as a geometry-aware representation learner rather than only a message-passing network.

4. Integration with grounding, planning, localization, and manipulation

In open-vocabulary indoor grounding, OVIGo-3DHSG represents an extensive indoor environment over a Hierarchical Scene Graph derived from RGB-D frame sequences, explicitly modeling relations across floors, rooms, locations, and objects, and integrates the hierarchy with a LLM for multistep reasoning using inter-layer and intra-layer connections (Linok et al., 16 Jul 2025). The available abstract specifies the hierarchical representation and LLM reasoning but does not specify a distinct HSGE module. This suggests that, in some work, HSGE is implicit in the scene-graph construction and downstream reasoning pipeline.

In navigation, the role is more explicit. SAGE-Nav uses an asynchronous LLM planner to generate semantic waypoints, an HSGE to turn the current hierarchical scene graph and active waypoint into a structure-aware embedding, and a Goal-aware Alignment-Fusion Network to fuse that embedding with egocentric visual features for a recurrent Actor–Critic controller (Su et al., 24 Jun 2026). A related but nonparametric formulation maintains an online object–zone–region HSG and plans over it with hierarchical beliefs and POUCT rollouts, using region- and zone-level priors from an LLM combined with exploration evidence (Wu et al., 30 Jun 2026). HERO further extends navigation by redefining traversability: operable obstacles are modeled as pathways, and the hierarchical graph is updated so that interaction with these objects changes the navigation topology (Wang et al., 17 Dec 2025).

In localization, a learned hierarchical scene graph matching pipeline encodes both a prior architectural graph and an online situational graph, each containing room and wall-surface nodes with typed edges, with a shared MLP plus GATv2 encoder. Cross-graph affinity is computed by

$V_t = O_t \cup Z_t \cup R_t$0

then normalized by Sinkhorn and discretized by the Hungarian algorithm to obtain correspondences for BIM-assisted localization (Ndulue et al., 30 Apr 2026). In robotic autonomy, Hi-Dyna Graph builds a global topological graph from posed RGB-D and dynamic relation subgraphs from video, anchors them using spatial or semantic constraints, and exposes the unified graph to an LLM agent that infers latent task triggers and generates navigation and manipulation commands (Hou et al., 30 May 2025). In functional manipulation-centric scene understanding, the open-vocabulary pipeline anchors fine-grained functional edges from 2D visual evidence, associates nodes across frames in 3D, optimizes temporal edge consistency, and finally performs global hierarchy shaping to recover the $V_t = O_t \cup Z_t \cup R_t$1, $V_t = O_t \cup Z_t \cup R_t$2, and $V_t = O_t \cup Z_t \cup R_t$3 levels (Hu et al., 15 May 2026).

5. Empirical evidence

Empirical results show that hierarchical encoding can improve both structural quality and downstream task performance. TB-HSU introduces a 3DHSG dataset with 120 hierarchical scene graphs, 3967 nodes, and 3847 edges, spanning 12 room types, 27 region-specific affordances, 194 object labels, and approximately 200 object-specific affordances. On room classification, TB-HSU reports 62.19% accuracy on Matterport3D versus 57.67% for Hydra, and on the custom 3DHSG dataset it reports 91.67% accuracy and 74.60% mIoU for room classification, plus 87.27% accuracy and 78.55% mIoU for region classification with position embedding (Xu et al., 2024).

SAGE-Nav reports that adding HSGE alone improves ALL SR from 74.21% to 75.62% and long-horizon $V_t = O_t \cup Z_t \cup R_t$4 SR from 65.53% to 67.15%. In the full system, removing HSGE reduces ALL SR from 82.47% to 80.15% and $V_t = O_t \cup Z_t \cup R_t$5 SR from 77.22% to 75.10%, with worse DTS, while the abstract reports state-of-the-art performance in i-THOR and RoboTHOR with gains in navigation efficiency and zero-shot generalization (Su et al., 24 Jun 2026).

Hyperbolic encoding improves graph-structure metrics even when retrieval remains strong. HSG reports PP IoU of 33.17 and Graph IoU of 33.51, outperforming the best AoMSG variant’s Graph IoU of 25.37 by 8.14, while maintaining Recall@1 comparable to Euclidean baselines (Wang et al., 19 Apr 2026). Learned hierarchical graph matching for localization reports, on a real LiDAR plus BIM environment, F1 of 84% versus 67% for the combinatorial baseline, with 0.021 s runtime versus 0.191 s, after being trained only on synthetic floor-plan graphs (Ndulue et al., 30 Apr 2026).

Hierarchical structure also benefits dynamic and embodied settings. Hi-Dyna Graph reports, on OpenPVSG, in-vocabulary $V_t = O_t \cup Z_t \cup R_t$6 of 10.56 / 8.90 and open-vocabulary 12.24 / 9.07, and maintains competitive vertex and edge accuracy under dynamic updates over long time horizons (Hou et al., 30 May 2025). HERO reports a 35.1% reduction in PL in partially obstructed environments and a 79.4% increase in SR in fully obstructed ones relative to its baseline (Wang et al., 17 Dec 2025). In geometry-heavy object-centric hierarchy, the superquadric-based representation improves map alignment over ROMAN and yields 100% navigation success with 0.885 efficiency and 0.271 s occupancy-map build time, compared with 209.192 s for dense mesh-based occupancy construction (Zhang et al., 1 Jun 2026). These results indicate that HSGE is valuable not only for semantic labeling but also for graph quality, planning efficiency, and geometric reasoning.

6. Limitations, ambiguities, and open directions

The literature does not yet converge on a single formal definition of HSGE. In SAGE-Nav the name refers to a concrete R-GCN module (Su et al., 24 Jun 2026); in TB-HSU the paper explicitly functions as such without naming the module that way (Xu et al., 2024); and in OVIGo-3DHSG the available abstract describes a hierarchical scene graph plus LLM reasoning but does not specify whether a distinct HSGE exists (Linok et al., 16 Jul 2025). This suggests that HSGE remains a unifying concept rather than a standardized component.

Several recurrent technical limitations remain. SAGE-Nav identifies dependence on perception quality, scalability issues in very large or dynamic environments, limited cross-level message passing, and the absence of explicit auxiliary supervision on graph embeddings (Su et al., 24 Jun 2026). Online HSG-based semantic navigation assumes perfect pose, is bottlenecked by VLM/LLM calls, and rebuilds hierarchy periodically with spectral clustering, with the paper explicitly noting that incremental or online graph neural encoders could replace this procedure (Wu et al., 30 Jun 2026). Hi-Dyna Graph similarly notes dynamic update latency and the absence of a learned graph encoder over the hierarchical graph (Hou et al., 30 May 2025). Hyperbolic HSG is sensitive to curvature initialization, and fixed curvature $V_t = O_t \cup Z_t \cup R_t$7 degrades the structural benefits of hyperbolic representation learning (Wang et al., 19 Apr 2026).

A second unresolved issue is where hierarchy should reside. In some scene graph generation work, hierarchy is primarily in the output or label space—for example hierarchical relation heads, hierarchical knowledge graphs, or higher-order structure embedded classifiers—rather than in explicit multi-level graph message passing (Jiang et al., 2023, Jiang et al., 2023). By contrast, later 3D and embodied systems increasingly encode hierarchy in the graph topology itself, in multigranular node types, or in the geometry of the latent space (Su et al., 24 Jun 2026, Wang et al., 19 Apr 2026). A plausible implication is that future HSGE research will continue to combine these strands: explicit multilevel graph topology, learned cross-level message passing, geometry-aware embedding spaces, temporal optimization, and structured interfaces to LLMs or VLMs.