HMSG: Hierarchical Multi-modal Scene Graphs

• Hierarchical Multi-modal Scene Graph (HMSG) is a structured representation that organizes spatial entities over multiple abstraction levels while incorporating multi-modal attributes such as visual, geometric, and linguistic data.
• It enables diverse applications including open-vocabulary grounding, robotic navigation, and dynamic scene understanding through advanced message passing and cross-modal fusion strategies.
• Empirical studies show HMSG enhances task performance with scalable retrieval methods, reinforcement learning integration, and efficient multi-modal data aggregation.

Hierarchical Multi-modal Scene Graph (HMSG) denotes a family of scene representations in which a graph $G=(V,E)$ organizes entities across multiple abstraction levels and attaches multi-modal attributes to nodes and edges. In recent work, the hierarchy may encode floors, rooms, views, and objects; rooms, furniture, compartments, and objects; or object-level nodes and cluster-level nodes, while the modalities may combine RGB or RGB-D observations, point clouds, language, CLIP or GloVe embeddings, captions, and task instructions (Linok et al., 16 Jul 2025, Kurenkov et al., 2020, Su et al., 24 Jun 2026, Zhou et al., 17 Sep 2025). HMSGs are used for open-vocabulary indoor grounding, hierarchical mechanical search, Object-Goal Navigation, vision-language navigation, dynamic robotic autonomy, operating-room scene graph generation, and multi-scale image–transcriptome alignment, which suggests that the term functions less as a single canonical architecture than as a structured design pattern for hierarchical reasoning over spatially organized multi-modal data (Hou et al., 30 May 2025, Guo et al., 2024, Jeong et al., 19 Nov 2025).

1. Graph formalisms and hierarchical organization

Representative HMSG formulations begin with an explicit graph definition but differ in node taxonomy and edge semantics. In Hierarchical Mechanical Search, the graph is a directed acyclic graph $G=(V,E)$ with $V=C\cup O$, where $C$ is the set of container-nodes and $O$ is the set of object-nodes, and the only edges are parent$\to$child links $E\subseteq C\times (C\cup O)$; the hierarchy may be described as rooms, furniture or shelves, and leaf objects (Kurenkov et al., 2020). In SAGE-Nav, the graph is $G=(V,E)$ with object-level nodes $V_{object}$, cluster-level nodes $V_{cluster}$, intra-level edges $G=(V,E)$0, and inter-level hierarchical edges $G=(V,E)$1, so that multiple abstraction levels $G=(V,E)$2 capture rooms$G=(V,E)$3regions$G=(V,E)$4objects in a single multi-relational graph (Su et al., 24 Jun 2026). In FSR-VLN, the graph is directed over four node sets $G=(V,E)$5, with Floor$G=(V,E)$6Room, Room$G=(V,E)$7View, Room$G=(V,E)$8Object, and View$G=(V,E)$9Object edges (Zhou et al., 17 Sep 2025).

Other systems expand the same principle beyond a fixed indoor layout hierarchy. Hi-Dyna Graph separates a global topological graph $V=C\cup O$0, containing region-vertices and static-object vertices, from dynamic subgraphs $V=C\cup O$1, whose vertices encode tracked instances and whose edges encode pairwise relations over a recent time window; these subgraphs are then anchored to the global topology using semantic and spatial constraints (Hou et al., 30 May 2025). Sigmma constructs three nested subgraphs—micro, meso, and macro—over cell-level spatial transcriptomic data, so that graph receptive field grows with the spatial scale of aligned image regions (Jeong et al., 19 Nov 2025). HiGS, while framed around a Progressive Hierarchical Spatial-Semantic Graph (PHiSSG) rather than canonical HMSG, similarly defines a directed attributed graph with global and local refinement levels and explicit cross-level dependency edges (Hong et al., 31 Oct 2025).

Taken together, these formulations indicate that hierarchy in HMSG is not restricted to simple containment. It may encode containment, visibility, semantic inclusion, same-level spatial adjacency, temporal interaction, or cross-level dependency, depending on the target task and sensing regime.

2. Multi-modal attributes on nodes and edges

The defining feature of HMSG is that graph nodes and edges are not purely symbolic labels. They carry geometric and semantic attributes derived from heterogeneous perceptual or linguistic channels. In Hierarchical Mechanical Search, every node $V=C\cup O$2 is mapped to an initial feature vector $V=C\cup O$3, while the target text description is mapped to $V=C\cup O$4. Container nodes combine a 300-dimensional GloVe vector $V=C\cup O$5 with normalized volume; object nodes combine GloVe, normalized bounding-box center, normalized bounding-box size, and a ResNet-18 feature extracted from an image crop (Kurenkov et al., 2020).

FSR-VLN makes the geometric-semantic factorization explicit by storing for each node a feature $V=C\cup O$6, where $V=C\cup O$7 contains geometry and $V=C\cup O$8 semantics. Floor nodes store point-cloud and bounding heights together with a text embedding of floor name via GPT-4o; room nodes store 2D footprint polygon, room point-cloud, CLIP embedding of aggregated room-views, and a text tag from GPT-4o; view nodes store camera pose, CLIP image embedding, and VLM-generated caption embedding; object nodes store 3D bounding box, mean depth, and CLIP embedding of the object crop (Zhou et al., 17 Sep 2025). SAGE-Nav initializes each node with $V=C\cup O$9, where $C$0 is a CLIP or other semantic embedding and $C$1 is a sinusoidal 3D positional encoding, thereby coupling semantic content to spatial topology from the outset (Su et al., 24 Jun 2026).

Hi-Dyna Graph uses posed RGB-D frames and open-vocabulary image embeddings from a vision-language encoder for the static branch, and FC-CLIP plus UniTrack for the dynamic branch, where vertices correspond to tracked instances with semantic features and class labels, and edges correspond to relation instances aggregated over time spans (Hou et al., 30 May 2025). TriTemp-OR broadens the modality set further by combining 2D multi-view RGB streams, 3D point clouds, and language embeddings from LLaVA-Med; its scene graph generation pipeline performs tri-modal feature aggregation to predict relations in operating rooms (Guo et al., 2024). Sigmma shows that the same HMSG logic can be transferred outside classical robotic scenes: whole-slide HE images are tiled at three spatial resolutions, while Xenium spatial transcriptomes are converted into a cell graph with node embeddings from an ST foundation model and edges formed by spatial proximity (Jeong et al., 19 Nov 2025).

A plausible implication is that “multi-modal” in HMSG should be read operationally rather than narrowly. In the cited literature, it may mean semantic language plus geometry, visual semantics plus spatial topology, video plus depth plus language, or image tiles plus molecular measurements.

3. Hierarchical encoding, message passing, and fusion

Once a hierarchical graph has been built, HMSG systems differ primarily in how they propagate information across levels and fuse modalities. Hierarchical Mechanical Search performs one round of parent-child message passing before each classification step. For a container node $C$2 with children $C$3, it computes messages $C$4, overwrites each child feature with that message, averages the updated child features, and updates the parent as $C$5, where $C$6. Readout then ties the updated node feature to the target description by elementwise product, $C$7, followed by a classifier $C$8 and sigmoid (Kurenkov et al., 2020). The same model states explicitly that semantic/geometric fusion is done by simple concatenation when building node features and by elementwise multiplication at readout, with no learned gating or attention beyond the learned fully connected layers.

SAGE-Nav replaces this comparatively shallow message passing with a Hierarchical Scene Graph Encoder (HSGE) based on R-GCN layers, residual projection from the raw anchor $C$9, waypoint-conditioned infusion, multi-level average pooling, and a final MLP that produces a waypoint-conditioned embedding $O$0. That graph encoding is then aligned with egocentric observations through Goal-aware Alignment-Fusion Network (GAFN): CLIP ViT-B/32 visual features are cross-attended with $O$1, an alignment score $O$2 is computed, and a gated fusion produces the final fused feature $O$3 (Su et al., 24 Jun 2026).

FSR-VLN adopts a different computational strategy. Rather than emphasizing graph neural propagation, it uses HMSG for progressive retrieval. A text instruction $O$4 is embedded as $O$5, room scores are computed from the maximum CLIP similarity over views in the room, view scores from cosine between text and the image part of the view feature, and object scores from cosine between text and object features. Top-$O$6 candidates are retained, after which GPT-4o is used as a frozen VLM to score candidate view-object pairs $O$7, and the final pair is $O$8 (Zhou et al., 17 Sep 2025).

TriTemp-OR organizes fusion hierarchically across modalities rather than graph layers alone. ViewTemp performs scale-adaptive multi-scale 2D convolution and local temporal cross-attention over frames; PointTemp performs 4D convolution over $O$9 followed by global temporal self-attention; relation-aware unification merges 2D and 3D features at predicted subject-object locations; and cross-attention then fuses these bi-modal pair embeddings with distilled LLaVA-Med text embeddings before relation classification (Guo et al., 2024).

These mechanisms show that HMSG does not prescribe one encoder family. It accommodates message-passing GNNs, relational graph convolutions, progressive retrieval pipelines, and staged cross-modal fusion schemes.

4. Reasoning loops and downstream uses

The principal attraction of HMSG is that it turns hierarchical structure into a substrate for task-conditioned reasoning. OVIGo-3DHSG represents an extensive indoor environment over a Hierarchical Scene Graph derived from sequences of RGB-D frames using open-vocabulary foundation models and sensor data processing. Its hierarchical representation explicitly models spatial relations across floors, rooms, locations, and objects, and it integrates the hierarchical scene graph with a LLM for multistep reasoning that leverages inter-layer and intra-layer connections for complex queries involving spatial reference to other objects (Linok et al., 16 Jul 2025).

Hierarchical Mechanical Search applies a 3D scene graph to object search in indoor organized environments such as homes or offices. Its HMSG-based policy reasons jointly about rooms and containers that are likely to contain a target object and about which objects are best to move if the target is hidden from view, all from a natural language description of the target (Kurenkov et al., 2020). SAGE-Nav uses HMSG for Object-Goal Navigation by decoupling asynchronous global semantic planning from the high-frequency reactive control loop: an LLM decomposes abstract instructions into semantically grounded waypoints, HSGE encodes the graph or task-relevant subgraph, and the low-level policy consumes fused visual-topological guidance (Su et al., 24 Jun 2026). FSR-VLN uses HMSG in vision-language navigation to support progressive retrieval from coarse room-level localization to fine-grained goal view and object identification, then combines fast matching with slow VLM-driven refinement for final goal selection (Zhou et al., 17 Sep 2025).

Hi-Dyna Graph extends the reasoning loop to dynamic, human-centric environments. It constructs a global topological graph from posed RGB-D inputs, populates dynamic subgraphs from environmental and egocentric video with object position relations and human-object interaction patterns, anchors those subgraphs to the global topology, serializes the unified graph to text, and uses an LLM to infer latent task triggers and generate executable instructions grounded in robotic affordances (Hou et al., 30 May 2025). TriTemp-OR uses hierarchical tri-modal aggregation to predict relations for scene graph generation in operating rooms, where long-term temporal structure and class imbalance are central constraints (Guo et al., 2024). Sigmma uses hierarchical graph construction and multi-scale contrastive alignment to learn cross-modal representations of tissue organization from histopathology images and spatial transcriptomes (Jeong et al., 19 Nov 2025). HiGS treats a progressively built hierarchical graph as the organizing structure for multi-step associative semantic spatial composition in 3D scene generation (Hong et al., 31 Oct 2025).

This range of uses suggests that HMSG is best understood as a task interface between structured scene abstraction and downstream reasoning modules, including classifiers, retrieval systems, LLM planners, actor-critic policies, and generative pipelines.

5. Optimization objectives and supervision regimes

HMSG-based systems are trained under markedly different objectives, and some are not trained end-to-end as scene-graph models at all. Hierarchical Mechanical Search is fully supervised with per-node binary cross-entropy on “contains-target” or “occludes-target” predictions. Given sampled node-label pairs $\to$0, it computes $\to$1 after message passing and minimizes

$\to$2

updating the ResNet-18 backbone and all learned projections jointly via Adam (Kurenkov et al., 2020).

SAGE-Nav couples hierarchical graph encoding to reinforcement learning. The fused feature $\to$3 is passed to a two-layer LSTM actor-critic head, and optimization uses the standard A3C objective with advantage $\to$4, value weight $\to$5, entropy weight $\to$6, and discount $\to$7. Only HSGE parameters, the GAFN MLPs, and the actor-critic LSTM are learned end-to-end; both the LLM and the scene-graph builder are frozen at training time (Su et al., 24 Jun 2026).

FSR-VLN relies primarily on pretrained modules: CLIP encoders are pretrained with the standard contrastive InfoNCE loss, GPT-4o is used as a frozen inference engine, and the system notes that if one wished to learn intra-HMSG pooling weights one could minimize a cross-entropy on a held-out grounding dataset, but in the current system simple mean or concatenation pooling is used (Zhou et al., 17 Sep 2025). Sigmma, by contrast, is explicitly optimized for hierarchical cross-modal consistency: at each of the micro, meso, and macro scales it minimizes a bidirectional InfoNCE loss between paired image and graph embeddings, and the total loss is the sum across scales (Jeong et al., 19 Nov 2025). TriTemp-OR uses a composite loss $\to$8, where $\to$9 is an $E\subseteq C\times (C\cup O)$0 distillation loss aligning bi-modal pair embeddings to LLaVA-Med triplet embeddings, and $E\subseteq C\times (C\cup O)$1 is a focal classification loss over relations (Guo et al., 2024).

Other HMSG-related systems treat the graph primarily as an inference-time or partially learned structure. Hi-Dyna Graph uses a cross-entropy loss for the transformer-based relation classifier but states that there are no additional trainable modules for anchoring or LLM prompting (Hou et al., 30 May 2025). HiGS states explicitly that PHiSSG is a symbolic/graph structure built at inference time and that there is no end-to-end training of PHiSSG in that work (Hong et al., 31 Oct 2025).

One consequence is that HMSG should not be equated with a single training paradigm. Supervision may be per-node BCE, actor-critic RL, multi-scale contrastive alignment, relation classification, or fixed-module inference with a symbolic graph backbone.

6. Empirical behavior, misconceptions, and research trajectory

The empirical record is heterogeneous but consistently task-oriented. FSR-VLN reports, on 87 real-world instructions, average query latency and retrieval statistics showing $E\subseteq C\times (C\cup O)$2 strict success-rate and $E\subseteq C\times (C\cup O)$3 for RSR@1m/2m/3m/4m, while reducing the response time by $E\subseteq C\times (C\cup O)$4 compared to VLM-based methods on tour videos by activating slow reasoning only when fast intuition fails; its ablation reports RSR@1m of $E\subseteq C\times (C\cup O)$5 without Spatial-Target instructions and No Reasoning, $E\subseteq C\times (C\cup O)$6 with Spatial-Target but without No Reasoning, and $E\subseteq C\times (C\cup O)$7 for the full system (Zhou et al., 17 Sep 2025). Sigmma reports an improvement of avg. $E\subseteq C\times (C\cup O)$8 in the gene-expression prediction task and avg. $E\subseteq C\times (C\cup O)$9 in the cross-modal retrieval task across datasets (Jeong et al., 19 Nov 2025). TriTemp-OR reports Average Precision $G=(V,E)$0, Recall $G=(V,E)$1, and $G=(V,E)$2 on the 4D-OR test set, versus $G=(V,E)$3, $G=(V,E)$4, and $G=(V,E)$5 for LABRAD-OR, with ablations indicating that ViewTemp alone adds $G=(V,E)$6 points F1, PointTemp adds $G=(V,E)$7 points, and LLaVA-Med text alignment adds $G=(V,E)$8 points (Guo et al., 2024). OVIGo-3DHSG reports efficient scene comprehension and robust object grounding compared to existing methods on Habitat Matterport 3D Semantic multi-floor scenes, while SAGE-Nav reports state-of-the-art performance in i-THOR and RoboTHOR with substantial gains in navigation efficiency and zero-shot generalization while maintaining low control latency (Linok et al., 16 Jul 2025, Su et al., 24 Jun 2026).

Several simplifying readings of HMSG are contradicted by the literature. First, HMSG is not necessarily static. Hi-Dyna Graph reconstructs a new dynamic subgraph every $G=(V,E)$9 seconds and anchors it into a persistent global graph, while HiGS contrasts progressively built and merged PHiSSG with “canonical HMSG” understood as a single static parse of a scene or image (Hou et al., 30 May 2025, Hong et al., 31 Oct 2025). Second, HMSG is not necessarily equivalent to end-to-end graph learning. FSR-VLN uses frozen CLIP and GPT-4o components with simple mean or concatenation pooling, Hi-Dyna Graph uses no additional trainable modules for anchoring or LLM prompting, and HiGS builds its graph at inference time (Zhou et al., 17 Sep 2025, Hou et al., 30 May 2025, Hong et al., 31 Oct 2025). Third, HMSG is not restricted to conventional indoor object maps: TriTemp-OR applies hierarchical multi-modal scene graph generation to surgical scenes, and Sigmma applies the same broad design logic to histopathology image and spatial transcriptome alignment (Guo et al., 2024, Jeong et al., 19 Nov 2025).

The current trajectory points toward richer coupling between hierarchy, language-guided reasoning, and nontrivial temporal or multi-scale structure. OVIGo-3DHSG emphasizes inter-layer and intra-layer reasoning for open-vocabulary grounding, SAGE-Nav combines LLM planning with graph-conditioned reactive control, FSR-VLN uses room-to-view-to-object retrieval for latency-aware navigation, and Sigmma enforces cross-modal consistency at multiple scales rather than a single scale (Linok et al., 16 Jul 2025, Su et al., 24 Jun 2026, Zhou et al., 17 Sep 2025, Jeong et al., 19 Nov 2025). This suggests a broader consolidation of HMSG as a representational backbone for systems that must preserve both global context and local referential precision under heterogeneous sensing and task constraints.