AwareVLN: Self-Aware Navigation

• AwareVLN is a vision-and-language navigation framework that integrates a self-aware reasoning mechanism with a monocular-RGB policy for continuous environments.
• It employs a unified reason-act loop, assessing scene context, progress, and deviations without relying on explicit 3D maps or additional sensors.
• Empirical evaluations demonstrate enhanced navigation accuracy and interpretability, outperforming several sensor-rich methods in VLN-CE settings.

AwareVLN is a vision-and-language navigation framework for continuous environments that equips an end-to-end monocular-RGB navigation model with an explicit self-aware reasoning mechanism. It is designed for VLN-CE settings in which an embodied agent must follow a natural-language instruction in an unknown 3D environment using only egocentric visual observations, while avoiding the explicit map-building, depth sensing, and heuristic planning machinery used by many sensor-rich navigation systems. Its stated contribution is to bridge black-box VLM-based action prediction and map-centric planning by making the model reason about its own state, task progress, deviations, and next-step intentions inside a unified reason-act policy (Guo et al., 21 May 2026).

1. Task setting and conceptual scope

AwareVLN is formulated for Vision-and-Language Navigation in Continuous Environments. The instruction is

$\mathcal{I}=\{w_1, \dots, w_l\},$

and at time step $t$, the agent has observed

$$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$

where each $\mathbf{x}_i$ is a monocular RGB observation. The low-level action space is

$$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$

Each action corresponds to a motion primitive such as moving $25$ cm forward or rotating $15^\circ$. The model produces a textual action description, for example “move forward 75 cm,” which is then parsed into one or more low-level actions $\mathbf{a}_{t+1:t+k}$ (Guo et al., 21 May 2026).

The motivating claim is that recent VLM-based VLN systems benefit from large-scale pretraining but often lack an explicit and explainable understanding of the relation among the agent’s current state, the instruction’s active subgoal, and the observed scene. The paper associates this limitation with weak subtask planning, brittle recovery after wrong turns, unreliable stopping behavior, and limited interpretability. Conversely, methods based on explicit maps, topological graphs, SLAM, depth, odometry, or heuristic planning expose spatial structure more clearly but reduce compatibility with purely monocular, large-scale vision-language pretraining and require additional sensing infrastructure (Guo et al., 21 May 2026).

Within this framing, AwareVLN uses the term self-awareness in a restricted technical sense. It does not denote consciousness or open-ended introspection. Rather, it denotes the model’s ability to understand the current visual scene, assess task progress relative to the instruction, identify path deviations or stopping mistakes, and produce a high-level plan for the next phase of navigation (Guo et al., 21 May 2026).

2. Unified reason-act architecture

AwareVLN is implemented as a unified model that performs both reasoning and action generation. At each step it receives the instruction $\mathcal{I}$, the observation history $\mathcal{O}_t$, the most recent reasoning output $t$0, and a temporal cue indicating how many steps have elapsed since the previous reasoning event. The previous reasoning is fused with a relative temporal offset: $t$1 where $t$2 is the last time step at which reasoning was generated and $t$3 denotes concatenation or fusion (Guo et al., 21 May 2026).

The core policy is written as

$t$4

where $t$5 tokenizes text, $t$6 encodes visual input, $t$7 is a decision logit, and $t$8 is a textual output. For efficiency, the model uniformly samples 8 frames from the observation history as visual input (Guo et al., 21 May 2026).

A special-token decision selects either reasoning mode or action mode: $t$9 If $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$0 is selected, the model emits a structured reasoning string; if $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$1 is selected, it emits an action command that is parsed into executable low-level controls (Guo et al., 21 May 2026).

This design makes language itself the main intermediate state representation. AwareVLN does not construct an explicit geometric map, symbolic planner state, or graph memory. Instead, it maintains a textual self-representation through the previous reasoning output $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$2, its temporal recency, and the current visual context. This yields a reason-act loop in which reasoning is not merely explanatory output but part of the future input context for later decisions (Guo et al., 21 May 2026).

3. Structural reasoning and the notion of self-awareness

The structural reasoning module is the paper’s central mechanism for inducing self-awareness. Its “structure” is not a graph-theoretic state or a separate symbolic planner. Rather, it lies in the format and timing of reasoning. When reasoning is triggered, the model outputs a triplet:

1. Scene description: a concise depiction of the current visual context and surrounding environment.
2. Progress assessment: an analysis of which parts of the instruction have been completed and whether deviations have occurred.
3. Plan for the next step: a high-level intention or strategy for the next navigation phase (Guo et al., 21 May 2026).

The paper characterizes this reasoning process as multi-dimensional and causal: first describing the current scene, then assessing task status, then planning what should happen next. This triplet is intended to provide both spatial self-awareness and task-oriented self-awareness. Spatial self-awareness comes from grounding the scene description in observation history and key-node context; task-oriented self-awareness comes from explicitly tying the current state to completed instruction segments, possible mistakes, and remaining subgoals (Guo et al., 21 May 2026).

Reasoning is sparse rather than dense. AwareVLN triggers reasoning only at key navigation points instead of at every frame. The paper identifies three such scenarios:

• Subtask completion: when a sub-instruction is judged to have been fulfilled.
• Path deviation: when the current visual context is inconsistent with the expected route.
• Stopping error: when the agent is near the end of navigation but the visual context does not yet sufficiently match the goal (Guo et al., 21 May 2026).

This sparse schedule is a defining design choice. The paper argues that dense “reason with action” operation degrades performance relative to sparse reasoning. This suggests that AwareVLN treats self-awareness as event-conditioned reflection rather than continuous verbalization (Guo et al., 21 May 2026).

4. Automatic data engine and progress division

AwareVLN couples its structural reasoning module with an automatic data engine that generates large-scale reasoning supervision in Habitat. The purpose of this engine is to avoid manual annotation of progress-aware navigation rationales while still producing supervision that is aligned with meaningful navigation milestones (Guo et al., 21 May 2026).

Trajectory collection uses two complementary strategies. First, the system collects trajectories by strict ground-truth following, producing instruction-aligned successful episodes. Second, it uses a DAgger-based collection process in which an early-stage VLN model navigates using its own predictions and is corrected back to the next waypoint whenever it deviates. This second strategy produces trajectories containing mistakes, deviations, corrections, and recovery behaviors, which are particularly useful for training error-aware reasoning (Guo et al., 21 May 2026).

The engine then identifies key reasoning nodes automatically using room-level semantic annotations from the simulator and ground-truth waypoints from VLN datasets. The node categories are:

• Subtask completion nodes, often aligned with room transitions or the completion of a correction process.
• Path deviation nodes, marked when the executed trajectory diverges from the reference path beyond a threshold.
• Stopping error nodes, marking terminal states that are offset from the intended goal (Guo et al., 21 May 2026).

The phrase progress division refers to this systematic segmentation of long trajectories into meaningful progress events. The engine also supplies an estimated navigation progress signal, described in prose as the ratio of traveled distance to total path length. This suggests a scalar notion of progress, although the paper does not provide a formal LaTeX equation for it (Guo et al., 21 May 2026).

Reasoning labels are generated automatically using Qwen-VL-Max. The generation is multi-turn. In the first round, the VLM receives the full episode observation sequence and the instruction to form a global understanding of the trajectory. In subsequent rounds, for each key node it receives the node type, downsampled observations preceding the node, room-transition information, estimated progress, and—at deviation nodes—visual observations from the correction process. The resulting supervision is the same triplet used at inference: scene description, progress assessment, and next-step plan (Guo et al., 21 May 2026).

5. Training, inference, and evaluation protocol

AwareVLN follows a two-stage training pipeline. In pre-training, it uses common navigation data and large-scale vision-question answering datasets, following the general recipe used by NaVILA. In fine-tuning, it uses the reasoning-augmented navigation trajectories generated by the automatic data engine, together with additional human videos from NaVILA that do not contain reasoning supervision, to improve generalization (Guo et al., 21 May 2026).

The paper does not provide an explicit training-loss equation. It clearly states that the same model is trained to produce the special mode token, the reasoning text when reasoning is triggered, and the action text otherwise, but it does not publish a formula decomposing these objectives into weighted loss terms. This suggests a supervised autoregressive training regime, but the exact objective is not formalized in the reported text (Guo et al., 21 May 2026).

At inference time, the policy starts with $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$3 “None” and $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$4. At each step it forms $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$5, runs the unified policy, decides between $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$6 and $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$7, updates the reasoning memory if reasoning is produced, or executes parsed low-level actions if an action description is produced. The loop continues until the episode terminates, including through the low-level STOP action (Guo et al., 21 May 2026).

The reported implementation details include training on four nodes of NVIDIA H20 GPUs and inference on an NVIDIA RTX 4090 GPU at roughly 1 FPS. The evaluation setting is Habitat-based VLN-CE with monocular RGB only, without panoramic observations, depth, or odometry (Guo et al., 21 May 2026).

6. Empirical results and ablation findings

AwareVLN is evaluated on R2R-CE and RxR-CE in Habitat, using Matterport3D scenes. The reported validation-unseen splits contain 1,839 episodes for R2R-CE and 11,006 episodes for RxR-CE. The main metrics are Navigation Error (NE), Oracle Success (OS), Success Rate (SR), Success weighted by Path Length (SPL), and for RxR-CE also nDTW (Guo et al., 21 May 2026).

On R2R-CE Val-Unseen, AwareVLN reports:

• $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$8
• $$\mathcal{O}_t=\{\mathbf{x}_0, \dots, \mathbf{x}_t\},$$9
• $\mathbf{x}_i$0
• $\mathbf{x}_i$1

On RxR-CE Val-Unseen, it reports:

• $\mathbf{x}_i$2
• $\mathbf{x}_i$3
• $\mathbf{x}_i$4
• $\mathbf{x}_i$5 (Guo et al., 21 May 2026)

The paper positions these results as state of the art among monocular-RGB end-to-end methods and notes that they exceed several waypoint-predictor or map-based methods that use richer sensor suites. It also reports a cross-dataset transfer setting on RxR-CE Val-Unseen without training on RxR-CE, where AwareVLN achieves $\mathbf{x}_i$6, $\mathbf{x}_i$7, $\mathbf{x}_i$8, and $\mathbf{x}_i$9, again outperforming NaVid and NaVILA in that transfer setting (Guo et al., 21 May 2026).

The ablations directly support the self-awareness interpretation. Removing Subtask Completion reasoning causes the largest drop. On R2R-CE Val-Unseen, performance falls from $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$0 and $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$1 to $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$2 and $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$3; on RxR-CE Val-Unseen, it falls from $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$4 and $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$5 to $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$6 and $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$7. Removing Path Deviation reasoning also degrades performance substantially, while removing Stopping Error reasoning affects goal localization and final performance but less severely than removing subtask completion (Guo et al., 21 May 2026).

A second ablation examines the architectural controls. Removing the special $$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$8/$$\mathcal{A}=\{FORWARD, TURN\text{-}LEFT, TURN\text{-}RIGHT, STOP\}.$$9 tokens degrades performance on both R2R-CE and RxR-CE. Forcing the model to reason densely together with action prediction also underperforms the sparse reasoning regime. The paper interprets this as evidence that sparse self-aware reflection at key nodes is more effective than continuous reasoning (Guo et al., 21 May 2026).

Real-world sim-to-real experiments on a quadruped robot in Corridor, Home, and Office environments provide additional evidence. On simple tasks, AwareVLN reports $25$0 in all three environments, with NE values of 1.86, 1.54, and 1.77 respectively. On complex tasks, it reports $25$1 in Corridor, $25$2 in Home, and $25$3 in Office, again outperforming NaVid and NaVILA in the reported comparisons (Guo et al., 21 May 2026).

7. Relation to adjacent VLN research, limitations, and significance

AwareVLN belongs to a broader line of work that tries to make navigation policies more explicit about what they know, what they are doing, and what remains uncertain. Earlier work such as the Object-and-Action Aware Model (OAAM) separated object descriptions from action specifications and aligned them with visual perception and orientation features respectively, introducing a different notion of “awareness” centered on modality-aligned instruction grounding (Qi et al., 2020). More recent methods such as ActiveVLN emphasize active exploration through multi-turn reinforcement learning and environment interaction (Zhang et al., 16 Sep 2025), while AgentVLN frames VLN as a POSMDP with context-aware self-correction, target-visibility-aware stage routing, and query-driven perceptual chain-of-thought (Xin et al., 18 Mar 2026). AwareVLN differs from these by centering self-aware reasoning about scene, progress, deviation, and next-step planning inside a single monocular end-to-end policy rather than through explicit map construction, explicit skill libraries, or RL-based rollout optimization (Guo et al., 21 May 2026).

Its main significance lies in the claim that explicit self-awareness can be learned without explicit 3D maps or extra sensors. The method thereby positions itself between black-box action-only VLM policies and explicit-map planners. This suggests a particular design philosophy for VLN: rather than requiring metric structure externally, the model can be trained to maintain a structured textual account of where it is in the task and when it should reflect before acting (Guo et al., 21 May 2026).

The paper also states clear limitations. Because AwareVLN reasons from monocular observations without an explicit 3D representation, its perception of layout can still be imprecise in real deployment. Reported failure modes include bumping into doors and stopping slightly offset from the intended target. In addition, the automatic data engine depends on simulator-side semantics, room annotations, and waypoint information, which may complicate direct transfer of the same data-generation pipeline to domains lacking comparable metadata (Guo et al., 21 May 2026).

Taken together, AwareVLN is best understood as a self-awareness-oriented VLN framework in which navigation performance and interpretability are both improved by sparse, structured reasoning at key progress events. Its defining claim is not merely that language-guided agents should act, but that they should explicitly know when to reflect on what they see, what they have completed, whether they are off track, and what they should do next (Guo et al., 21 May 2026).