Goal-aware Alignment-Fusion Network (GAFN)

• The paper presents GAFN as a module that fuses egocentric visual features with structure-aware waypoint embeddings to create goal-aware representations for reactive navigation.
• It employs structure-infused cross-attention and alignment-aware gating to balance local perception with global structural priors.
• GAFN enhances navigation performance by reducing dithering and improving approach stability, as evidenced by lower DTS and higher success rates.

Goal-aware Alignment-Fusion Network (GAFN) is the fusion module within SAGE-Nav that connects high-level structural semantics to a low-latency reactive navigation policy for Object-Goal Navigation (ObjNav). Introduced as the bridge between LLM-guided waypoint planning and embodied control, GAFN dynamically aligns egocentric visual perception with hierarchical scene graph priors derived from the active waypoint, then produces a fused, goal-aware representation for a two-layer LSTM Actor-Critic controller. Its defining characteristics are structure-infused cross-attention, an explicit visual-topological alignment signal, and an alignment-aware gate with an inductive bias that increasingly trusts structural priors as alignment rises (Su et al., 24 Jun 2026).

1. Position within the SAGE-Nav hierarchy

SAGE-Nav decomposes ObjNav into three asynchronous-to-reactive layers: an LLM-Guided Hierarchical Global Planner (H-GP), a Hierarchical Scene Graph Encoder (HSGE), and GAFN. The planner outputs a sequence of semantically grounded waypoints over a compact subgraph; HSGE converts that subgraph into structure-aware embeddings preserving semantics and spatial topology; GAFN performs visual-topological alignment and fusion for downstream control. This organization is central to the system’s decoupling of sparse global planning from a high-frequency reactive loop.

Layer	Function	Output relevant to GAFN
H-GP	Sequence of semantically grounded waypoints over a compact subgraph	Active waypoint in $P^*$
HSGE	Structure-aware encoding of hierarchical scene graphs	Waypoint embedding $E_w$
GAFN	Visual-topological alignment and fusion	Fused representation $F_f$

GAFN consumes three signals: egocentric visual features $F_v$ from a frozen CLIP visual backbone, the structure-aware waypoint embedding $E_w$ produced by HSGE for the active waypoint, and an alignment scalar $\alpha_t$ computed from $F_v$ and $E_w$. It outputs the fused state representation $F_f$, which is passed to the two-layer LSTM Actor-Critic policy. In operational terms, this means that long-horizon planning remains asynchronous and sparse, while control remains reactive and low latency.

A common misunderstanding is to treat GAFN as a planner. It is not. Its role begins after waypoint selection and graph encoding, where it conditions reactive control on plan-relevant structural context rather than replacing the planner itself.

2. Input representations and reactive dataflow

GAFN is defined by the types of representations it fuses. CLIP ViT-B/32 produces 512-D visual features per frame, and flattened feature maps are linearly projected to a 128-D latent for fusion. HSGE, in turn, encodes 544-D node features—512-D CLIP semantics concatenated with 32-D sinusoidal 3D position—through a 3-layer R-GCN to a 64-D waypoint embedding $E_w$. For fusion, $E_w$0 and $E_w$1 are projected to 128-D, and the final fused representation $E_w$2 is 64-D.

The active waypoint from the LLM sequence $E_w$3 conditions both HSGE and GAFN at time $E_w$4. The graph representation supplied to HSGE preserves topological priors through multi-relational intra-level and inter-level edges, hierarchical pooling, and residual attention. These priors provide the structure that GAFN later injects into the reactive visual stream.

The reactive loop proceeds in a fixed sequence. Perception $E_w$5 is encoded by CLIP into $E_w$6. H-GP selects the active waypoint, and HSGE encodes the relevant graph to produce $E_w$7. GAFN cross-attends $E_w$8 with $E_w$9 to obtain $F_f$0, computes the alignment scalar $F_f$1, gates the fusion, and outputs $F_f$2. The LSTM policy then consumes $F_f$3 together with its recurrent state to produce $F_f$4 and $F_f$5. This dataflow is the mechanism by which a sparse waypoint plan is translated into dense control guidance (Su et al., 24 Jun 2026).

3. Alignment-aware fusion mechanism

GAFN performs fusion in two stages. The first stage is structure-infused cross-attention:

$F_f$6

This operation grounds the visual latent in the waypoint’s geometry and semantics, rather than relying on per-frame appearance alone.

The second stage is alignment-aware gated fusion. Visual-topological alignment at time $F_f$7 is defined as the cosine similarity between the egocentric visual latent and the graph-derived waypoint embedding:

$F_f$8

The gate is then computed as

$F_f$9

with $F_v$0 and $F_v$1, and fusion is performed by

$F_v$2

The explicit inductive bias is encoded in the multiplicative penalty term $F_v$3. As $F_v$4 increases, the penalty reduces $F_v$5, biasing the fused representation toward the structure-infused feature $F_v$6. In the paper’s interpretation, this adaptively balances obstacle avoidance, associated with local visual evidence in $F_v$7, against goal-directed structural guidance in $F_v$8. The stated effect is to stabilize final approach and reduce dithering near the goal (Su et al., 24 Jun 2026).

This design also distinguishes GAFN from prior ObjNav fusion strategies that often use concatenation or fixed attention without explicit goal-aware gating. The comparison drawn in the paper is not merely architectural: it frames GAFN as a module that quantifies alignment explicitly and uses that scalar to regulate reliance on structure versus immediate perception.

A further point of clarification is that alignment is not optimized via a standalone supervised objective in the reported system. The alignment scalar modulates fusion inside the gate; no separate alignment loss is reported.

4. Dependence on the Hierarchical Scene Graph Encoder

GAFN’s behavior depends on the form of the waypoint embedding supplied by HSGE. For each node, the input feature is

$F_v$9

where $E_w$0 is a CLIP embedding and $E_w$1 is a positional encoding. HSGE applies intra-level R-GCN encoding according to

$E_w$2

where $E_w$3 comprises relational convolution, LN, ReLU, and Dropout, and $E_w$4 is a residual projection introduced to mitigate over-smoothing and preserve raw anchors.

The active waypoint’s intrinsic embedding $E_w$5 conditions the level-$E_w$6 encodings through attention weights

$E_w$7

followed by

$E_w$8

Hierarchical aggregation then produces the final waypoint embedding through multi-level average pooling, channel-wise concatenation, and an MLP:

$E_w$9

These operations are the basis for the claim that $\alpha_t$0 preserves multi-scale spatial topology and semantics. GAFN therefore does not fuse raw graph descriptors; it fuses a waypoint-conditioned structural summary whose inductive biases are already shaped by relational message passing, residual connections, and waypoint-centric attention. The paper also notes that HSGE’s $\alpha_t$1 weights bias the representation toward graph nodes that are structurally proximal or semantically relevant to the waypoint, which propagates waypoint context into the topological neighborhood.

This suggests that GAFN should be understood as a second-stage alignment mechanism layered on top of a first-stage graph-conditioning mechanism: HSGE constructs the structure-aware prior, and GAFN decides how strongly that prior should influence the current control state.

5. Policy integration, optimization context, and empirical effect

The low-level controller is a two-layer LSTM Actor-Critic network that consumes $\alpha_t$2 and recurrent hidden states to produce the action distribution $\alpha_t$3 and the value $\alpha_t$4. Reward shaping uses the composite navigation reward

$\alpha_t$5

where $\alpha_t$6, $\alpha_t$7, $\alpha_t$8 indicates target visibility, and $\alpha_t$9 indicates forward movement. Optimization uses the standard A3C objective with value and entropy regularization.

The training context is defined precisely. Experiments are conducted in iTHOR and RoboTHOR with standard splits; iTHOR uses 22 target categories, while RoboTHOR contains apartments with larger areas and longer trajectories. CLIP ViT-B/32 provides frozen 512-D features. The online 3D scene graph fuses multi-frame OpenSEED segmentations and VLM-driven relations; nodes are aggregated by distance $F_v$0 and semantic affinity threshold $F_v$1. HSGE maps 544-D node features to a 64-D $F_v$2. Reinforcement learning uses A3C with Adam, $F_v$3, $F_v$4, and 6 million episodes. During training, hierarchical scene graphs for training environments are pre-constructed offline to accelerate learning; the learnable components include HSGE and the LSTM-based policy.

The reported metrics are Success Rate (SR), Success weighted by Path Length (SPL), and Distance to Success (DTS). Full-system performance is reported as follows: iTHOR (ALL), SR 82.47%, SPL 42.34%, DTS 0.32 m; iTHOR ($F_v$5), SR 77.22%, SPL 43.73%, DTS 0.43 m; RoboTHOR (ALL), SR 52.35%, SPL 30.12%, DTS 0.72 m; RoboTHOR ($F_v$6), SR 40.82%, SPL 22.95%, DTS 1.10 m (Su et al., 24 Jun 2026).

Ablations isolate GAFN’s contribution. Removing GAFN and replacing it with naive concatenation reduces SR from 82.47% to 78.65% on ALL iTHOR and from 77.22% to 73.95% on iTHOR $F_v$7, while DTS worsens from 0.32 to 0.39 and from 0.43 to 0.50. SPL in the long-horizon setting slightly increases from 43.73 to 44.94. The paper interprets this as evidence that GAFN’s more conservative approach stabilizes success and approach quality at a small cost to shortest-path adherence. In zero-shot generalization, removing alignment fusion drops SR/SPL/DTS from 75.05%/34.05%/0.38 m to 70.65%/30.80%/0.47 m, indicating that the module is especially important when structural priors must compensate for unfamiliar local appearance.

6. Efficiency, robustness, limitations, and non-reported extensions

SAGE-Nav reports an average per-step latency of 0.42 s with sparse LLM calls averaging 9.1 per episode, compared with CogNav at 2.20 s and 15 calls and SG-Nav at 0.85 s and 32 calls. The control loop invokes LLM planning asynchronously on subgoal attainment or deadlocks, rather than at every step. Within this system-level design, GAFN is the component that allows the control loop to remain reactive while still incorporating global structural guidance (Su et al., 24 Jun 2026).

The robustness claims attributed to GAFN are specific. Cross-attention grounding aligns perception to structural priors, making the agent less prone to local visual noise when the goal is out of view. The adaptive gate mitigates overreliance on either local perception or priors; as alignment grows, reliance shifts to structural context, which is reported to reduce last-meter dithering and premature termination.

Observed failure modes remain tied to perception and state estimation. The paper lists target visibility failure, such as high shelves, detector false positives, starting-position entrapment, and premature Done decisions when the agent is within an estimated but incorrect proximity. Future work identified in the same source includes improved open-vocabulary detection, active viewpoint control, and 3D volumetric representations to enhance obstacle handling.

Several mathematical formulations appear in the source as reconstructed possibilities rather than official training components. A plausible auxiliary alignment loss is an InfoNCE-style objective,

$F_v$8

which would pull $F_v$9 toward $E_w$0 and push it away from non-target waypoints. A margin-based alternative is

$E_w$1

Likewise, a plausible gate regularizer is

$E_w$2

These expressions are explicitly presented as reconstructions consistent with the mechanism, not as reported losses in the paper. The official training losses include policy gradient, value, and entropy regularization, and no separate alignment loss is reported.

The same distinction applies to the canonical A3C loss sometimes associated with the system:

$E_w$3

with $E_w$4 and $E_w$5. This formulation is reconstructed for completeness and consistency with A3C, not printed as the full loss formula in the paper. The same section also suggests possible extensions, including explicit alignment losses, gate regularization, mixture-of-experts or multi-head alignment gates, and predictive modeling for dynamic scenes. Such proposals are best understood as forward-looking implications of the GAFN design rather than established components of the reported model.