SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation

Abstract: Object-Goal Navigation (ObjNav) requires embodied agents to autonomously locate specified targets using only egocentric visual observations. Existing monolithic methods struggle with long-horizon reasoning and generalize poorly to novel environments. To address these limitations, we propose SAGE-Nav, a novel hierarchical framework that integrates the reasoning capabilities of LLMs with dynamic scene graphs. Crucially, it decouples asynchronous global semantic planning from the high-frequency reactive control loop. The LLM serves as a global planner, decomposing abstract instructions into a sequence of semantically grounded waypoints. To translate these plans into dense multi-modal guidance, we design a Hierarchical Scene Graph Encoder (HSGE) that leverages relational graph convolutions to produce structure-aware embeddings preserving both semantic and spatial topology. Furthermore, we develop the Goal-aware Alignment-Fusion Network (GAFN) to dynamically fuse real-time perception with these structural priors. Using an adaptive gating mechanism with an explicit inductive bias, GAFN ensures robust visual-topological alignment for the low-level policy. Extensive evaluations in the i-THOR and RoboTHOR environments demonstrate that SAGE-Nav achieves state-of-the-art performance, delivering substantial gains in navigation efficiency and zero-shot generalization while maintaining the low control latency required for physical robotic deployment.

• The paper introduces a hierarchical framework that decouples global semantic planning from local reactive control using LLM-guided planning, scene graph encoding, and goal-aware fusion.
• It employs multi-layer R-GCNs and CLIP-based features to encode spatial-semantic relationships, leading to improved success rates on benchmarks like iTHOR and RoboTHOR.
• Reinforcement learning via an A3C Actor-Critic policy validates the framework's efficiency in long-horizon reasoning and robust navigation under challenging conditions.

SAGE-Nav: Hierarchical Navigation via LLM Planning and Scene Graph Alignment

Framework Overview and Motivation

SAGE-Nav addresses the Object-Goal Navigation (ObjNav) task, wherein an embodied agent must locate a specified object using solely egocentric observations. Prevailing end-to-end DRL methods lack explicit scene-level semantics, resulting in suboptimal generalization and long-horizon reasoning failures. SAGE-Nav introduces a hierarchical architecture composed of three core modules: an LLM-Guided Hierarchical Global Planner (H-GP); a Hierarchical Scene Graph Encoder (HSGE) for semantic-spatial grounding; and a Goal-aware Alignment-Fusion Network (GAFN) for multi-modal policy alignment. By decoupling global semantic planning from local reactive control, SAGE-Nav leverages both commonsense reasoning and structured priors to maximize navigation performance.

Figure 1: The SAGE-Nav pipeline: (i) LLM-Guided Global Planner produces semantic waypoints; (ii) HSGE grounds the plan in structured spatial-semantic representations; (iii) GAFN dynamically fuses aligned perception and graph priors for efficient RL-based navigation.

Hierarchical Global Planning via LLMs and Scene Graphs

SAGE-Nav constructs a hierarchical scene graph $\mathcal{G}$ with multi-level object and cluster nodes, encoding both semantic and spatial relationships through heterogeneous edges and relational graph convolutions. During navigation, a retrieval-augmented instruction-relevant subgraph $\mathcal{G}_Q$ is generated, where each candidate node's semantic prior is boosted by its neighborhood and hierarchy. The LLM planner operates asynchronously over $\mathcal{G}_Q$, solving a Greedy-TSPP to yield coherent sequences of waypoints, balancing semantic desirability and geometric cost via a hierarchical coherence weight:

$$W(v_i, v_j) = \lambda_{\text{sem}} S_{\text{multi-level}}(v_i, v_j) - \lambda_{\text{dist}} D_{\text{geo}}(v_i, v_j)$$

Unseen object reasoning is handled by LLM-induced probabilistic inference, dynamically integrating virtual target nodes into the planning substrate.

Structural Encoding and Waypoint Conditioning

The HSGE module translates abstract plans into actionable embeddings $\mathbf{E}_w$, using multi-layer R-GCNs to encode hierarchical scene structure and leveraging CLIP-based semantic and sinusoidal positional features. Object-level and cluster-level nodes are processed with intra-level relational encoding:

$$h_{v}^{(\ell)} = \mathcal{F}_{\text{R-GCN}}^{(\ell)}(\mathbf{x}_v, \mathcal{E}^{(\ell)}) + W_{\text{res}} \mathbf{x}_v$$

Waypoint-conditioned attention refines structural relevance, followed by level-wise pooling and MLP projection to produce the structure-aware embedding for the current plan's waypoint.

Figure 2: HSGE architecture: Multi-layer R-GCN for structural encoding, residual attention for task alignment, and level-wise pooling for final structure-aware waypoint embedding.

Goal-Aware Alignment-Fusion Mechanism

GAFN aligns real-time egocentric visual features with structural priors from the HSGE via cross-attention. The module computes a semantic alignment score $\alpha_t$ and adaptively modulates fusion through an explicit gating mechanism:

$$\lambda(\alpha_t) = \sigma(\text{MLP}([\mathbf{F}_v; \mathbf{F}_a; \mathbf{E}_w; \alpha_t])) \cdot (1 - \eta\sigma(\kappa \alpha_t))$$

This design structurally incentivizes the agent to rely more on abstract guidance as visual and topological alignment increases, facilitating robust progress in long-horizon and occluded scenarios, and balancing obstacle avoidance against goal pursuit.

Reinforcement Learning Optimization

The fused representation $\mathbf{F}_f$ is passed to a two-layer LSTM-based A3C Actor-Critic policy. The reward formulation simultaneously encourages goal-reaching, efficient trajectory, exploration, and penalizes redundant actions:

$$R_{\text{nav}} = 5 \cdot \mathbb{I}_{vis} + 0.01 \cdot \mathbb{I}_{fwd} + \max(\lambda \Delta d_t, 0) - 0.01 \cdot (1 - \mathbb{I}_{vis})$$

Optimization is performed over 6 million episodes, with frozen CLIP visual encoders and Qwen2.5-VL-7B LLM for planning. Training is strictly decoupled from LLM runtime overhead, focusing on HSGE and the policy network.

Figure 3: Asynchronous A3C training architecture. Parallel workers collect trajectories, with frozen modules and shared optimizer for network updates.

Experimental Results

Quantitative Performance

SAGE-Nav exhibits strong numerical gains in iTHOR (SR $\mathcal{G}_Q$0 overall, $\mathcal{G}_Q$1 for $\mathcal{G}_Q$2) and RoboTHOR (SR $\mathcal{G}_Q$3, SPL $\mathcal{G}_Q$4, DTS $\mathcal{G}_Q$5), consistently outperforming prior SOTA methods (e.g., TSOG, CGI-GAIL, AKGVP-CI) across all major navigation benchmarks. The efficiency analysis demonstrates SAGE-Nav achieves the lowest control latency ($\mathcal{G}_Q$6 s) and sparse LLM queries ($\mathcal{G}_Q$7 calls/episode), surpassing CogNav in SPL and SG-Nav in computational cost.

Zero-shot and Ablation Results

SAGE-Nav achieves $\mathcal{G}_Q$8 SR, $\mathcal{G}_Q$9 SPL, and $\mathcal{G}_Q$0 m DTS on zero-shot categories, significantly outperforming baselines. Ablation indicates GAFN is crucial for cross-domain adaptation and alignment; removing HSGE results in pronounced SR and DTS deterioration, emphasizing the importance of structural encoding.

Qualitative Analysis

Navigation trajectory visualizations reveal that SAGE-Nav avoids common failure modes, such as redundant paths and premature terminations, exhibited by previous baselines. Waypoint-guided planning enables superior path efficiency and target localization.

Figure 4: Agent trajectory visualizations in unfamiliar scenes: SAGE-Nav consistently selects efficient paths and avoids rotational loops present in prior baselines.

Limitations and Future Directions

Failure case analysis highlights four main limitations: target visibility failures, detector errors, entrapment in constrained spaces, and premature termination. Addressing these issues will involve integrating robust open-vocabulary detectors, active vision for perceptual adjustment, and 3D volumetric representations for extrication and obstacle avoidance.

Figure 5: Failure case visualizations: Representative errors include missed targets, false positives, entrapment, and premature episode termination.

Conclusion

SAGE-Nav establishes a hierarchical paradigm in embodied navigation, integrating LLM-powered global planning with dynamic scene graph priors and explicit perceptual alignment. Empirical results validate substantial performance improvements in efficiency, long-horizon reasoning, and zero-shot generalization. The explicit structural cascade facilitates robust adaptation in complex unseen environments. Future work will extend SAGE-Nav with generative active perception and scalable 3D representations to enhance real-world deployment resilience.