SAGE-Nav: Hierarchical Scene Graph Navigation

• SAGE-Nav is a hierarchical framework for ObjNav that decouples high-level semantic planning from low-level control using dynamic scene graphs and LLMs.
• It employs a dual-loop design where an LLM-based global planner computes semantically grounded waypoints while a reactive Actor-Critic controller executes fast, precise motions.
• Evaluations on iTHOR and RoboTHOR demonstrate state-of-the-art zero-shot generalization and low latency, outperforming existing methods in success rate and efficiency.

SAGE-Nav is a hierarchical framework for Object-Goal Navigation (ObjNav) that integrates LLMs with dynamic scene graphs in order to address long-horizon reasoning and generalization to novel environments. In the formulation introduced in "SAGE-Nav: Leveraging LLM Planning and Alignment Fusion for Hierarchical Scene Graph-Guided Navigation," the system decouples asynchronous global semantic planning from the high-frequency reactive control loop: an LLM produces semantically grounded waypoints over a hierarchical scene graph, while a reactive Actor-Critic controller executes motions using fused visual and structural features. The reported evaluations in i-THOR and RoboTHOR position SAGE-Nav as a state-of-the-art ObjNav system with strong zero-shot generalization and low control latency (Su et al., 24 Jun 2026).

1. Problem setting and conceptual scope

ObjNav requires an embodied agent to autonomously locate specified targets using only egocentric visual observations. The motivating claim behind SAGE-Nav is that existing monolithic methods struggle with long-horizon reasoning and generalize poorly to novel environments. The framework therefore imposes an explicit hierarchy: high-level semantic reasoning is separated from low-level closed-loop actuation, and the interface between the two is a waypoint representation grounded in a hierarchical scene graph (Su et al., 24 Jun 2026).

The system is defined around two asynchronously coupled loops. The Global Planner, denoted H-GP, is an LLM that reasons over a hierarchical scene graph to produce a sequence of high-level, semantically grounded waypoints. The low-level controller is a reactive A3C policy with an LSTM backbone that executes motions at high frequency using dense multi-modal guidance. This design directly targets a common misconception in embodied navigation research, namely that stronger semantic reasoning must imply a slower control loop. In SAGE-Nav, the slow deliberative component is explicitly decoupled from the fast reactive component rather than inserted into every action-selection step (Su et al., 24 Jun 2026).

A second defining feature is the use of an incrementally growing hierarchical scene graph $G=(V,E)$ built from the egocentric RGB-D stream and VO/VLM summaries. The graph is not merely a memory store: it is the substrate for retrieval-augmented LLM planning, waypoint grounding, and structural feature extraction. This suggests that SAGE-Nav is best understood not as a monolithic policy, but as a graph-mediated planning-and-control architecture.

2. Hierarchical architecture and control flow

The runtime pipeline consists of five stages. First, the agent builds or updates the hierarchical scene graph $G=(V,E)$ from its egocentric RGB-D stream and VO/VLM summaries. Second, whenever the agent reaches a waypoint or deadlocks, H-GP queries the LLM using a compact subgraph $G_Q$ as context and decomposes the instruction into a sequence of semantic regions or landmarks. Third, the Hierarchical Scene Graph Encoder (HSGE) converts each planned waypoint into a fixed-size embedding $E_w$ that preserves semantic and spatial topology. Fourth, at every step, the Goal-aware Alignment-Fusion Network (GAFN) fuses $E_w$ with the current visual feature $F_v$ to produce $F_f$. Fifth, $F_f$ is flattened and passed into the LSTM-based Actor-Critic, yielding the action distribution $\pi(a_t \mid h_t,q)$ (Su et al., 24 Jun 2026).

The architectural split is therefore between semantic decomposition and reactive execution. H-GP does not output primitive actions; it outputs waypoints. The low-level controller does not perform open-ended symbolic reasoning; it consumes a dense fused representation and acts at high frequency. The explicit waypoint interface is central, because it transforms the LLM output from free-form text into a semantically grounded control target.

The low-level policy is specified as an Actor-Critic network with an A3C objective and an LSTM backbone. This places SAGE-Nav in the class of hierarchical embodied agents that rely on recurrent state for partial observability while using an explicit semantic scaffold for long-range planning. The framework’s contribution is not merely adding an LLM, but coupling the LLM to a dynamic scene graph and then aligning the resulting waypoint semantics with perception through a dedicated fusion mechanism.

3. Retrieval-augmented LLM planning over scene graphs

The formal planning stage constructs a compact subgraph $G_Q \subset G$ by ranking each node $G=(V,E)$0 according to

$G=(V,E)$1

$G=(V,E)$2

with $G=(V,E)$3 empirically. The top-$G=(V,E)$4 nodes, with $G=(V,E)$5, together with their cluster summaries $G=(V,E)$6, form the retrieval-augmented context supplied to the LLM (Su et al., 24 Jun 2026).

The prompt template is explicitly semantic. An example context includes cluster summaries such as kitchen or living room content, followed by a task like “Find a mug,” and asks the LLM to return an ordered list of sub-regions to visit. The output is parsed into a waypoint list such as $G=(V,E)$7. These regions are then grounded to object-level waypoints via top-down edges: for each region $G=(V,E)$8, the procedure appends $G=(V,E)$9 or a virtual node.

Ordering is associated with an internal objective

$G_Q$0

with $G_Q$1 and $G_Q$2. The LLM’s summaries $G_Q$3 influence $G_Q$4. The practical effect is that semantic relevance and geometric cost are both present in waypoint ordering, but neither is treated as sufficient in isolation.

This planning stage also clarifies the role of the graph hierarchy. The LLM does not plan over raw frames; it plans over a compact graph context that already compresses semantic and relational structure. A plausible implication is that the retrieval step regularizes LLM reasoning by constraining it to a structured and task-relevant subgraph rather than the full accumulated world model.

4. HSGE and GAFN: structural encoding and alignment fusion

The hierarchical scene graph is defined with nodes

$G_Q$5

where $G_Q$6 is a semantic label, $G_Q$7 is a 3D centroid, $G_Q$8 is a CLIP embedding, $G_Q$9 is the abstraction level, and $E_w$0 is a VLM-generated text summary that is empty at $E_w$1. Edges are typed, including spatial adjacency and hierarchical inclusion, and carry metric distances (Su et al., 24 Jun 2026).

HSGE initializes each node with

$E_w$2

where $E_w$3 is a 32-D sinusoidal 3D encoding. Each intra-level $E_w$4 is then processed by an R-GCN: $E_w$5 For one layer,

$E_w$6

After 3 layers, a residual anchor is added.

Waypoint conditioning is implemented by computing alignment weights

$E_w$7

followed by

$E_w$8

The final waypoint embedding is

$E_w$9

GAFN then fuses this structure-aware waypoint embedding with the visual stream. Its inputs are a flattened visual feature map $E_w$0, described as CLIP-ViT projected to 128-D, and the waypoint embedding $E_w$1. First,

$E_w$2

Then it computes the semantic alignment score

$E_w$3

and a gate

$E_w$4

with $E_w$5 and $E_w$6. The fused feature is

$E_w$7

The stated inductive bias is explicit: the factor $E_w$8 drives $E_w$9 as $F_v$0, smoothly shifting control from obstacle-avoidance $F_v$1 to goal-directed guidance $F_v$2 (Su et al., 24 Jun 2026).

5. Optimization protocol and empirical profile

The low-level policy is trained with A3C. For a rollout of length $F_v$3, the total loss is

$F_v$4

where $F_v$5, $F_v$6, and $F_v$7. The navigation reward is

$F_v$8

with $F_v$9 and $F_f$0 (Su et al., 24 Jun 2026).

The reported dataset splits are: iTHOR with 120 scenes, partitioned into 80 train, 20 val, and 20 test across 4 room types and 22 target categories; RoboTHOR with 75 train+val scenes, partitioned into 60 train and 5 val, plus 14 test scenes, of which 10 are test. Training uses 6 M episodes of A3C, Adam with learning rate $F_f$1, and $F_f$2. Scene graphs are pre-built offline to focus on HSGE+policy learning. All experiments were conducted on a single RTX 3090.

The evaluation metrics are Success Rate (SR), SPL, and Distance to Success (DTS), reported on ALL episodes and on long-horizon episodes with optimal path $F_f$3 m. Selected results are as follows (Su et al., 24 Jun 2026):

Setting	SAGE-Nav	Comparator
iTHOR (ALL)	SR 82.47%, SPL 42.34%, DTS 0.32 m	TSOG: SR 80.04%, SPL 41.44%, DTS 0.40 m
RoboTHOR (ALL)	SR 52.35%, SPL 30.12%, DTS 0.72 m	CGI-GAIL: SR $F_f$4, SPL $F_f$5
Zero-shot novel targets	SR 75.05%, SPL 34.05%, DTS 0.38 m	AKGVP-CI: SR 69.51%, SPL 28.86%, DTS 0.41 m

The ablations isolate the contribution of each module. In the zero-shot setting, removing Unseen Inference changes SR from $F_f$6 to $F_f$7, removing LLM Reasoning changes SR to $F_f$8, and removing Alignment Fusion changes SR to $F_f$9. In iTHOR ALL / $F_f$0, the Base system scores $F_f$1, $F_f$2H-GP only scores $F_f$3, $F_f$4HSGE only scores $F_f$5, $F_f$6H-GP scores $F_f$7, $F_f$8HSGE scores $F_f$9, $\pi(a_t \mid h_t,q)$0GAFN scores $\pi(a_t \mid h_t,q)$1, and the Full system scores $\pi(a_t \mid h_t,q)$2. These numbers indicate that the full gain is not attributable to a single component.

The efficiency trade-off is explicit in RoboTHOR: CogNav reports SR $\pi(a_t \mid h_t,q)$3, latency $\pi(a_t \mid h_t,q)$4 s/step, and 15 LLM calls/episode; SG-Nav reports SR $\pi(a_t \mid h_t,q)$5, latency $\pi(a_t \mid h_t,q)$6 s, and 32 calls; SAGE-Nav reports SR $\pi(a_t \mid h_t,q)$7, latency $\pi(a_t \mid h_t,q)$8 s, and 9.1 calls. This is consistent with the claim that asynchronous semantic planning can coexist with low action latency.

6. Terminological overlap and adjacent systems

The name “SAGE-Nav” is not unique across the broader arXiv literature. In the most direct usage, it denotes the hierarchical scene graph-guided ObjNav framework described above (Su et al., 24 Jun 2026). However, the same label or closely related labels also appear in distinct technical settings.

In "Improving Diffusion Planners by Self-Supervised Action Gating with Energies," the paper introduces SAGE as an inference-time re-ranking method for diffusion planners and includes a pseudocode block titled “Algorithm SAGE-Nav.” There, the method samples candidate futures, computes a latent consistency energy over the first $\pi(a_t \mid h_t,q)$9 steps, keeps the lowest-energy fraction, and selects the action by maximizing a value score penalized by energy. This use of “SAGE-Nav” belongs to offline reinforcement learning and diffusion planning rather than embodied ObjNav (Lu et al., 3 Mar 2026).

The label also appears in trajectory generation for spacecraft under the name SAGES, described as “also called SAGE-Nav,” where a language encoder, semantic interpreter, and sequential convex programming module translate natural-language commands into constraint-satisfying rendezvous trajectories (Takubo et al., 9 Dec 2025). Related but distinct navigation paradigms include SAGE-3D, which upgrades 3D Gaussian Splatting into an executable, semantically and physically aligned environment for Visual-Language Navigation (Miao et al., 24 Oct 2025), and S-Nav, a semantic-geometric planner built on S-Graphs for mobile robots (Kremer et al., 2023).

This suggests a broader naming pattern rather than a single lineage: across these systems, “SAGE” or “SAGE-Nav” is repeatedly associated with coupling high-level semantic structure to executable navigation or control. For technical precision, however, the term should be disambiguated by paper title or arXiv identifier whenever it is used in scholarly discussion.