Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation

Abstract: Semantic navigation is a fundamental task for embodied agents operating in unseen environments, requiring both semantic understanding and long-term decision-making. Recent foundation models have empowered agents with rich semantic priors for this task. However, without structured global representations, decision-making often falls back on local observations and greedy strategies, resulting in inefficient exploration and myopic behaviors, especially in long-distance navigation. To address these challenges, we propose a zero-shot semantic navigation framework. Our method incrementally maintains an online Hierarchical 3D Scene Graph (HSG) to form a multi-granular semantic topology over objects, zones, and regions, serving as a compact state abstraction for global planning. Building on this memory, we introduce a hierarchical belief-based planning framework that fuses semantic priors with exploration evidence on the HSG, and performs finite-horizon rollouts on an HSG-based simulator to explicitly estimate the long-term expected returns of candidate macro-actions. This enables globally consistent decisions and reduces redundant backtracking. Extensive experiments in high-fidelity simulation environments across multiple tasks and datasets demonstrate that our method outperforms existing state-of-the-art methods, particularly in long-distance scenarios, where our approach improves SR and SPL by an average of 9.4\% and 5.0\%, respectively.

• The paper presents a novel framework that incrementally builds hierarchical 3D scene graphs and uses belief-based planning to enable long-horizon, zero-shot semantic navigation.
• It combines real-time RGB-D processing, spectral clustering for semantic abstraction, and POUCT-based simulation to optimize exploration and decision-making.
• Empirical evaluations on benchmarks like Matterport3D and HSSD demonstrate significant improvements in success rate and SPL, especially in long-range navigation tasks.

Hierarchical 3D Scene Graph Construction and Belief-Based Planning for Semantic Navigation

Introduction

This work presents a zero-shot semantic navigation framework built on the online, incremental construction of Hierarchical 3D Scene Graphs (HSGs) tailored to enable embodied agents to efficiently explore and reason about unseen environments. The motivation addresses the deficits of contemporary foundation-model-based navigation agents, particularly their reliance on local, unstructured observations and myopic exploration policies, which inhibit long-horizon, semantically guided planning. The core contribution is a methodology that incrementally maintains an HSG at multiple semantic granularities and couples this with a hierarchical, belief-based planning architecture that fuses foundation model priors with active exploration evidence for globally consistent decision making.

Framework Overview

The framework is two-fold. The perception module processes RGB-D observations to incrementally maintain both an occupancy map and a multi-layer HSG, composing object instances, spatial topologies, and object-centric cell partitions. The decision module maintains a belief state over this graph, synthesizes semantic priors via LLM queries with evidence from information-theoretic frontier analysis, and employs a hierarchical Partially Observable Monte Carlo Tree Search (POUCT) within an HSG-based simulator to explicitly evaluate long-horizon returns of candidate macro-actions. Local planning decomposes region/zone selections into efficient exploration tours, solved as asymmetric traveling salesman problems.

Figure 1: Overview of the full pipeline showing perception, belief updating, macro-action planning via HSG, and local path execution.

This system enables the agent to abstract low-level sensory streams into semantically meaningful, globally structured representations used for planning at multiple decision horizons—a key factor in overcoming inefficiencies typical of prior zero-shot navigation approaches.

Hierarchical 3D Scene Graph (HSG) Construction

The HSG incrementally integrates three semantic layers:

• Object Layer: Nodes correspond to currently detected 3D object instances, estimated via open-vocabulary segmentation models projected into the world frame. Object adjacency is established via generalized Voronoi decomposition on the occupancy grid.
• Zone and Region Layers: Higher-level nodes are induced by spectral clustering over the object-level graph, combining geometric proximity and semantic similarity (from VLM embeddings), with cross-level containment edges forming the hierarchical topology.

  Figure 2: The construction of the Object–Zone–Region hierarchical scene graph via bottom-up clustering and Voronoi-based spatial partitioning.

Each node is enriched with multi-modal semantic descriptions: objects are labeled with stable category predictions, zones receive text captions generated by VLMs, and region-level summaries are produced by LLM aggregation over constituent zone captions. The graph updates incrementally, with local neighborhood recomputation and hierarchical reclustering at fixed intervals, maintaining efficient online operation.

Hierarchical Belief-Based Planning

A central feature is maintaining a hierarchical belief state over the HSG, modeling the distribution of target existence across regions and zones. The belief integrates:

• Semantic Priors: Computed via LLM queries conditioned on node descriptions, dynamically weighted to prevent overconfidence in explored regions.
• Exploration Evidence: Derived from information gain metrics on unexplored frontiers in each region/zone, modulating the belief based on observed expansion potential.

  Figure 3: Computation of exploration evidence via frontier information gain and local planning based on high-gain areas within selected HSG partitions.

This belief is updated at each decision step and forms the input distribution for planning.

The planner instantiates an HSG-based simulator for explicit POUCT rollouts, evaluating the expected long-term returns of visiting candidate nodes. The design supports macro-action selection at both region and zone levels with variable planning horizons, directly optimizing expected path efficiency while penalizing redundant backtracking and myopia.

Figure 4: (a) Illustration of planning over the HSG—enabling non-myopic, globally efficient trajectories via rollouts; (b) Performance gains compared to leading baselines on long-distance navigation tasks.

Empirical Evaluation

The method is evaluated across multiple benchmarks, including Matterport3D, HM3Dv1/2, HSSD, and the InstanceNav text-instructed navigation dataset, emphasizing both zero-shot ObjectNav and TextInstanceNav tasks.

• ObjectNav: Outperforms strong baselines (e.g., ApexNav) with an average SR improvement of 5.4% and SPL improvement of 2.3%, and shows a pronounced benefit in long-range settings (over 10m initial distance), where the gains rise to 9.4% SR and 5.0% SPL.
• TextInstanceNav: Achieves 17.9% SR and 4.6% SPL increase versus the state-of-the-art (UniGoal), highlighting the advantage of granular semantic abstraction in complex query settings.

Ablation studies systematically dissect the impact of the exploration evidence, hierarchical granularity, planning horizon, and VLM quality:

Figure 5: Level and planning horizon ablation, demonstrating how hierarchy depth and simulation horizon affect performance on long-range object navigation.

• Removing exploration evidence or hierarchical layers substantially reduces efficiency, confirming the necessity of multi-granularity reasoning and active feedback in belief updates.
• Extensions of planning horizon (POUCT rollouts) yield initially improved returns until model approximation errors accumulate, confirming a task-dependent optimal horizon-depth.

Qualitative analyses visualize navigation trajectories and graph evolutions, demonstrating a significant reduction in redundant revisiting and improved spatial coverage efficiency compared to myopic baselines.

Figure 6: Comparative visualizations of agent trajectories and HSG evolution for a representative episode, highlighting macro-action efficiency and memory-driven avoidance of revisit loops.

Implications and Future Directions

This method demonstrates that online, hierarchical structuring of semantic and geometric information—coupled with hierarchical, evidence-driven planning—is critical for scaling zero-shot semantic navigation in complex, previously unseen environments. Practically, the approach suggests a pathway to integrating modular, foundation-model-powered semantic inference with continual world modeling for long-horizon, goal-oriented sequential decision processes.

However, several limitations are noted:

• The system’s current deployment assumes accurate RGB-D pose and is subject to VLM/LLM inference latency, precluding real-time application in unconstrained real-world settings.
• Scene-graph and occupancy fusion performance under realistic SLAM error profiles and partial observability remains unaddressed.

Future research directions include integration with semantic SLAM for robust localization, acceleration of semantic inference pipelines, and full real-robot validation.

Conclusion

The proposed online HSG construction and belief-based planning framework advances the state-of-the-art in zero-shot semantic navigation by endowing embodied agents with structured global memory and explicit long-horizon planning. The method achieves strong numerical gains over existing baselines on multiple benchmarks, especially for challenging long-distance navigation. Its hierarchical abstraction and tight coupling of semantic priors with observation-driven feedback offer a robust paradigm for embodied sequential decision systems in semantic-rich, unstructured environments.

---

Reference: "Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation" (2606.31071)