Hierarchical Object-Zone Graphs

• Hierarchical Object-Zone Graphs are structured representations that capture multi-layer spatial, semantic, and relational information in 3D environments.
• They integrate sensor fusion, segmentation, and neural message passing to construct accurate graphs that support efficient navigation and object localization.
• These graphs enable open-vocabulary grounding and language-conditioned queries by combining graph convolution techniques with large language model integrations.

Hierarchical object-zone graphs are structured representations that encode the spatial, semantic, and relational organization of environments, particularly relevant in 3D scene understanding, navigation, and object localization tasks. These graphs formalize multi-level containment and adjacency relationships between rooms, zones, containers, and objects, supporting reasoning over hierarchical and compositional structures. Their utility has been demonstrated in indoor and outdoor robotics, embodied AI navigation, open-vocabulary object grounding, and multi-modal query answering, with numerous recent advances in graph construction, feature embedding, neural message passing, and integration with LLMs.

1. Formal Structure and Typology

Hierarchical object-zone graphs are typically modeled as directed graphs or trees, with nodes stratified by semantic and spatial scope:

• Nodes encode hierarchical units such as floors, rooms, containers/zones, furniture, and objects. For example, $$V = V_\text{floor} \cup V_\text{room} \cup V_\text{zone} \cup V_\text{object}$$ (Werby et al., 2024, Linok et al., 16 Jul 2025, Werby et al., 1 Oct 2025, Kurenkov et al., 2020).
• Edges capture containment, part-whole, or relational ties. Most frameworks restrict edges to parent-child ("contains") or cross-layer connections; lateral/adjoining or functional edges may also be included (e.g., doorways, adjacency, spatial relations) (Linok et al., 16 Jul 2025, Deng et al., 2024, Lingelbach et al., 2023).
• Features on nodes include visual (e.g., CLIP or ResNet embeddings), semantic (label distributions, word embeddings), geometric (3D position, bounding boxes), and contextual/linguistic information, supporting open-vocabulary and multi-modal reasoning (Kurenkov et al., 2020, Werby et al., 2024, Werby et al., 1 Oct 2025, Linok et al., 16 Jul 2025).

A canonical hierarchy may extend through four or more layers (floor → zone/room → location/furniture → object) (Linok et al., 16 Jul 2025, Werby et al., 1 Oct 2025), and can be formalized as: $$G = (V, E) \quad \text{with} \quad V = V_\text{floor} \cup V_\text{zone} \cup V_\text{object} (\cup \, V_\text{location}, V_\text{furniture} )$$ with $E$ partitioned into inter- and intra-layer edge sets (Linok et al., 16 Jul 2025). In robotics, containment-based trees are often combined with additional graphs for navigation (e.g., Voronoi graphs across/within floors (Werby et al., 2024)).

2. Construction and Graph Embedding Methodologies

Graph construction pipelines proceed in several stages, fusing sensor data and semantic cues:

• Semantic and geometric mapping is performed with RGB-D or LiDAR input, fusing data into volumetric or point cloud representations (e.g., via TSDF integration (Linok et al., 16 Jul 2025)) and extracting surface meshes or occupancy grids (Werby et al., 2024).
• Zone/room/floor segmentation involves floor-wise slicing, clustering (K-means, DBSCAN, Watershed) on geometric or appearance cues, and polygonal region fitting (Werby et al., 1 Oct 2025, Werby et al., 2024, Linok et al., 16 Jul 2025).
• Object detection and anchoring leverages open-vocabulary detectors (e.g., OWL-ViT, Grounding DINO), with back-projection to 3D and matching across views to build aggregated instance nodes (Linok et al., 16 Jul 2025, Deng et al., 2024).
• Edge construction encodes explicit containment and, optionally, proximity, adjacency, functional, or logical relations (e.g., "onTop," "inRoom") (Lingelbach et al., 2023, Linok et al., 16 Jul 2025).

Feature embedding pipelines typically combine:

• Visual representations (ResNet, CLIP, SBERT embeddings)
• Geometric attributes (centroids, bounding box parameters, room polygons)
• Semantic distributions (label histograms or open-set class probabilities)
• Node aggregation or graph convolution (GCN, Heterogeneous Graph Transformers) for embedding propagation and summarization (Kurenkov et al., 2020, Zhang et al., 2021, Lingelbach et al., 2023, Werby et al., 2024).

Node and edge attributes are updated online as new sensor data is acquired, with policies for fusing or overwriting features in the face of dynamic observations (Zhang et al., 2021, Werby et al., 2024).

3. Neural Message Passing and Hierarchical Reasoning

Neural message passing over hierarchical object-zone graphs enables context-sensitive and cross-layer inference:

• Recursive message passing: Node features are recursively updated with neighborhood and parent context (e.g., with MLPs over concatenated parent-child features, followed by aggregation) (Kurenkov et al., 2020).
• Typed GNNs: Dedicated transformations per relation type (e.g., "onTop," "adjacent") allow flexible handling of heterogeneous edges (Lingelbach et al., 2023).
• Attention-based summarization: Goal-conditioned or query-conditioned attention mechanisms weight node contributions for downstream reasoning or policy outputs (Lingelbach et al., 2023, Werby et al., 1 Oct 2025).
• LLM integration: Scene graph APIs or embedding functions serve as memory for LLMs, supporting multi-step query decomposition and hierarchical reasoning across layers (e.g., floor-room-object tuples, spatial predicates) (Linok et al., 16 Jul 2025, Werby et al., 1 Oct 2025).

End-to-end pipelines leverage binary cross-entropy or RL objectives (PPO, A3C) for learning, propagating gradients through node embedding layers, GNN modules, and vision/language backbones (Kurenkov et al., 2020, Lingelbach et al., 2023, Zhang et al., 2021).

4. Application Domains and Empirical Performance

Hierarchical object-zone graphs yield strong empirical performance in several domains:

• Mechanical and hierarchical search: Top-down greedy search based on per-node probabilities assigned via neural message passing enables efficient discovery of occluded targets in multi-room, multi-container scenarios. Dynamic thresholding and priority-driven exploration guarantee completeness (Kurenkov et al., 2020).
• Goal-directed navigation: Coarse-to-fine planning with graph-based sub-goal selection, zone-aware embeddings, and DRL policies achieves superior navigation success (e.g., +13% SR, +6% SPL over A3C in AI2-Thor (Zhang et al., 2021)), with similar gains in SLAM-based and RoboTHOR environments.
• Open-vocabulary object grounding: Integration of foundation models for object recognition (e.g., CLIP, SBERT) with hierarchical graphs enables robust object localization and spatial query answering in Habitat Matterport3D and Replica. OVIGo-3DHSG achieves 71.5% object grounding vs. 60.5–67.8% for non-hierarchical baselines, and 82.1% zone IoU (Linok et al., 16 Jul 2025).
• Language-conditioned navigation and retrieval: HOV-SG reduces memory footprint by 75% over dense VLMaps, with 56.1% navigation success (robot within 1 m of target) and AUC$_k^\text{top}$ = 84.9% on large-scale multi-floor scenes (Werby et al., 2024).
• Functional and compositional reasoning: KeySG’s keyframe-augmented graph supports complex and ambiguous language queries, with competitive or superior Recall@K and segmentation metrics against state-of-the-art baselines (Werby et al., 1 Oct 2025).

A summary of core applications and quantitative results is provided below:

Framework	Main Application	Object Grounding (%)	Zone IoU (%)	Navigation SR (%)
OVIGo-3DHSG (Linok et al., 16 Jul 2025)	Open-vocab grounding	71.5	82.1	-
HOV-SG (Werby et al., 2024)	Navigation, retrieval	-	-	56.1
KeySG (Werby et al., 1 Oct 2025)	Hierarchical retrieval, QA	30.4 (IoU≥0.10)	-	34.0 (R@1)
HMS (Kurenkov et al., 2020)	Mechanical search	close to oracle (median actions)	-	-
HOZ (Zhang et al., 2021)	Object navigation	-	-	+13% over A3C

5. Comparative Insights and Design Considerations

Multiple ablation studies highlight that hierarchical structuring is critical for robust performance:

• Removing zone or floor layers results in large drops in grounding and localization accuracy (e.g., –12.3 pp for no zones in OVIGo-3DHSG (Linok et al., 16 Jul 2025)).
• Explicit intra-zone and object-object connectivity supports fine-grained disambiguation in cluttered or ambiguous environments (Linok et al., 16 Jul 2025, Lingelbach et al., 2023).
• Keyframe-driven summaries enable efficient scaling without explicit pairwise relation labeling, maintaining context for query answering even under prompt-length constraints (Werby et al., 1 Oct 2025).
• Online adaptation of node features enables rapid generalization to novel or rearranged environments without costly pre-mapping (Zhang et al., 2021).

This suggests that zone/room-level segmentation and containment edges both constrain the combinatorial search space and provide structural inductive bias, particularly beneficial for long-horizon search, navigation, and spatial reasoning tasks in complex scenes.

6. Extensions to Open-Vocabulary and Large-Scale Settings

Recent frameworks extend hierarchical object-zone graphs to:

• Open-vocabulary domains via vision-LLM embeddings, facilitating zero-shot and open-set class support for both object and zone queries (Deng et al., 2024, Werby et al., 2024, Werby et al., 1 Oct 2025).
• Large-scale, cross-floor, and outdoor environments with multi-layer hierarchies and navigation graph integration (e.g., Voronoi graphs, lane graphs, and cross-floor connectivity) (Deng et al., 2024, Werby et al., 2024).
• Language interaction through graph-augmented LLM planning, supporting multi-step, compositional, and abstract queries across scene layers (Werby et al., 1 Oct 2025, Linok et al., 16 Jul 2025).
• Task-driven attention pooling and transformer GNNs that enable goal- or instruction-specific summarization of scene graphs in embodied navigation agents (Lingelbach et al., 2023).

A plausible implication is that as open-vocabulary scene representations and LLMs become more capable, hierarchical object-zone graphs will increasingly serve as structured, differentiable memory for embodied reasoning and planning at scale.

7. Summary and Outlook

Hierarchical object-zone graphs provide a principled, multi-layered formalism for organizing the semantics, geometry, and topology of complex 3D environments. Through a combination of principled graph construction, efficient feature embedding, neural message passing—including attention and open-vocabulary fusion—and integration with LLMs, these structures have enabled state-of-the-art performance in navigation, mechanical search, grounding, and language-conditioned reasoning. Current research emphasizes scalability (both spatial and semantic), open-vocabulary generalization, and robust querying, suggesting that such hierarchical representations will remain central to embodied AI and spatially grounded multimodal reasoning (Kurenkov et al., 2020, Zhang et al., 2021, Linok et al., 16 Jul 2025, Deng et al., 2024, Werby et al., 2024, Werby et al., 1 Oct 2025, Lingelbach et al., 2023).