Fast-SmartWay: Zero-Shot VLN-CE Navigation

• The paper introduces Fast-SmartWay, a zero-shot VLN-CE framework that achieves efficient navigation in continuous 3D environments using only three frontal RGB-D images and end-to-end MLLM action prediction.
• It replaces traditional panoramic sensing and waypoint prediction with structured spatial-semantic representations and uncertainty-aware reasoning to handle ambiguous instructions.
• Empirical evaluations demonstrate improved success weighted by path length and reduced sensing overhead, highlighting its practical advantages over panoramic baselines.

Searching arXiv for the cited Fast-SmartWay paper and closely related navigation work to ground the article. Fast-SmartWay is a zero-shot framework for Vision-and-Language Navigation in Continuous Environments (VLN-CE) that eliminates the standard dependence on panoramic observations and waypoint predictors by using only three frontal RGB-D images and natural-language instructions for most navigation steps, while allowing a multimodal LLM (MLLM) to directly predict actions (Shi et al., 2 Nov 2025). It is designed for continuous 3D environments in which an embodied agent must navigate in a space $\mathbf{E}$ toward a goal $\mathbf{x}_{\text{goal}}$ by following an instruction $L=\{l_1,l_2,\dots,l_n\}$, without task-specific fine-tuning for the target benchmark or environment (Shi et al., 2 Nov 2025). The framework is positioned against prior zero-shot VLN-CE systems such as SmartWay, which typically use a two-stage panoramic-plus-waypoint pipeline, and argues that repeated 360° sensing and waypoint prediction introduce substantial latency and a semantic bottleneck that reduce real-world applicability (Shi et al., 2 Nov 2025, Shi et al., 13 Mar 2025).

1. Task definition and conceptual position

Fast-SmartWay addresses VLN-CE in the continuous-space setting rather than graph-constrained navigation. In this setting, the agent acts in continuous 3D space, senses the environment online, and progressively chooses motion actions to reach the target location $\mathbf{x}_{\text{goal}}$ from natural-language instructions (Shi et al., 2 Nov 2025). The framework is explicitly zero-shot in the sense that it performs navigation without task-specific training or fine-tuning for the target navigation benchmark or environment, and instead relies on the general multimodal reasoning ability of a pretrained MLLM plus prompt engineering and structured inference-time reasoning (Shi et al., 2 Nov 2025).

The paper frames Fast-SmartWay as a response to two standard design choices in prior MLLM-based VLN-CE. First, prior methods commonly use a panorama of 12 RGB-D views at headings $0^\circ,30^\circ,\dots,330^\circ$ at every step (Shi et al., 2 Nov 2025). Second, they often use a waypoint predictor that proposes navigable candidates from RGB-D geometry before language-conditioned reasoning selects among them (Shi et al., 2 Nov 2025). Fast-SmartWay rejects both assumptions for most steps. Its central claim is that a practical zero-shot embodied navigator should work with front-facing sensing only, avoid a separate waypoint proposal stage, and predict actions end-to-end from instruction and local observation (Shi et al., 2 Nov 2025).

This design makes Fast-SmartWay a direct methodological descendant of SmartWay rather than an unrelated system. SmartWay is a two-stage zero-shot VLN-CE framework integrating an enhanced waypoint predictor with an MLLM-based navigator (Shi et al., 13 Mar 2025). Fast-SmartWay retains the zero-shot MLLM orientation, but removes the waypoint predictor entirely and uses frontal-view end-to-end action prediction for normal operation (Shi et al., 2 Nov 2025). This suggests a shift from candidate-based decision-making toward direct action synthesis.

2. End-to-end navigation architecture

Fast-SmartWay has three main components: observation acquisition, spatial-semantic textual description generation, and MLLM-based action prediction with uncertainty-aware reasoning (Shi et al., 2 Nov 2025). The observation policy is asymmetric across time. At the initial step $t=0$, the robot performs a single panoramic scan,

$$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$

but for subsequent steps $t>0$, it uses only three frontal RGB-D views,

$I_t=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,2,3\},$

captured at headings $(330^\circ,0^\circ,30^\circ)$, corresponding to left, front, and right (Shi et al., 2 Nov 2025). The standard per-step image count therefore drops from 12 to 3 during normal navigation (Shi et al., 2 Nov 2025).

The framework does not use a learned waypoint predictor. Instead, the MLLM receives raw RGB images, textual semantic descriptions, textual spatial descriptions derived from depth, the instruction, and navigation history, and directly outputs action parameters (Shi et al., 2 Nov 2025). Operationally, the output includes a selected image, an action label, a turning degree when applicable, a safe forward distance, a Boolean confusion flag, and updated navigation history (Shi et al., 2 Nov 2025). The resulting control interface is hybrid: discrete image/action selection is combined with continuous-valued turning angle and forward distance.

At initialization, the system prompts the MLLM with the instruction, semantic descriptions $\mathbf{x}_{\text{goal}}$0, spatial descriptions $\mathbf{x}_{\text{goal}}$1, a task description, and 12 panoramic RGB images, and asks it to output a thought trace, a selected image from 1 to 12, a safe distance, a trajectory summary, and instruction progress (Shi et al., 2 Nov 2025). During subsequent frontal-view navigation, the MLLM receives three RGB images, the instruction $\mathbf{x}_{\text{goal}}$2, semantic descriptions $\mathbf{x}_{\text{goal}}$3, spatial descriptions $\mathbf{x}_{\text{goal}}$4, valid action options, prior observed objects, Previous Selected Image, Instruction Progress, Trajectory Summary, and Previous Thought (Shi et al., 2 Nov 2025). The system uses GPT-4o-2024-08-06 as the MLLM backend, matching the SmartWay baseline (Shi et al., 2 Nov 2025).

A key architectural implication is that Fast-SmartWay replaces the usual panoramic-plus-waypoint decomposition with a prompt-level multimodal policy that reasons directly over local RGB-D and structured memory (Shi et al., 2 Nov 2025). This suggests a different failure mode profile from SmartWay: the challenge shifts from waypoint quality to local ambiguity and history consistency.

3. Spatial-semantic representation from RGB-D observations

Fast-SmartWay converts depth and RGB observations into language-compatible spatial-semantic descriptions before invoking the MLLM (Shi et al., 2 Nov 2025). For depth, the method builds a partial panoramic point cloud from the three frontal depth views $\mathbf{x}_{\text{goal}}$5, corresponding to $\mathbf{x}_{\text{goal}}$6 (Shi et al., 2 Nov 2025). Each depth image is center-cropped, only the bottom half is retained, and the result is projected into 3D using the pinhole camera model (Shi et al., 2 Nov 2025). For a local-frame 3D point with coordinates $\mathbf{x}_{\text{goal}}$7, the ground-plane distance is

$\mathbf{x}_{\text{goal}}$8

and the nearest obstacle distance per image column is computed as the minimum such distance over rows in the bottom half (Shi et al., 2 Nov 2025).

The frontal $\mathbf{x}_{\text{goal}}$9 field of view from $L=\{l_1,l_2,\dots,l_n\}$0 to $L=\{l_1,l_2,\dots,l_n\}$1 is discretized into five bins corresponding to turn left $L=\{l_1,l_2,\dots,l_n\}$2, turn left $L=\{l_1,l_2,\dots,l_n\}$3, go forward, turn right $L=\{l_1,l_2,\dots,l_n\}$4, and turn right $L=\{l_1,l_2,\dots,l_n\}$5 (Shi et al., 2 Nov 2025). For each bin $L=\{l_1,l_2,\dots,l_n\}$6, the mean obstacle distance $L=\{l_1,l_2,\dots,l_n\}$7 is converted into text using thresholds $L=\{l_1,l_2,\dots,l_n\}$8 and $L=\{l_1,l_2,\dots,l_n\}$9 (Shi et al., 2 Nov 2025): $\mathbf{x}_{\text{goal}}$0 The resulting set is denoted

$\mathbf{x}_{\text{goal}}$1

(Shi et al., 2 Nov 2025).

At the initial step and during disambiguation, a 12-bin panoramic spatial description $\mathbf{x}_{\text{goal}}$2 is generated from 12 depth images using central image regions and directional text such as “go forward,” “turn left $\mathbf{x}_{\text{goal}}$3,” “turn right $\mathbf{x}_{\text{goal}}$4,” or “turn around” (Shi et al., 2 Nov 2025). For RGB semantics, Fast-SmartWay uses the RAM model to extract semantic object tags. For frontal-view navigation, the semantic set is

$\mathbf{x}_{\text{goal}}$5

while the panoramic case uses

$\mathbf{x}_{\text{goal}}$6

(Shi et al., 2 Nov 2025).

The role of this representation is not merely descriptive. The paper explicitly argues that waypoint predictors operate mainly from RGB-D geometry and may generate candidates that are geometrically traversable but semantically irrelevant to the language goal (Shi et al., 2 Nov 2025). Fast-SmartWay therefore substitutes waypoint generation with a direct language-oriented representation of geometry and semantics. A plausible implication is that the method trades explicit candidate enumeration for prompt-time structured scene abstraction.

4. Uncertainty-aware reasoning and bidirectional planning consistency

Fast-SmartWay augments its end-to-end navigator with an Uncertainty-Aware Reasoning module comprising a Disambiguation Module and a Future-Past Bidirectional Reasoning mechanism (Shi et al., 2 Nov 2025). This module is designed to address ambiguous instructions, conflicting visual cues, and local optima induced by egocentric perception (Shi et al., 2 Nov 2025).

The Disambiguation Module is triggered when the MLLM outputs Confuse = true (Shi et al., 2 Nov 2025). The paper states that confusion may arise when the instruction is ambiguous, the goal is unclear, or visual cues conflict with expected progression (Shi et al., 2 Nov 2025). When confusion is detected, the robot performs a full 360° rotation, collects 12 RGB images, semantic descriptions $\mathbf{x}_{\text{goal}}$7, and panoramic spatial descriptions $\mathbf{x}_{\text{goal}}$8, together with the current Trajectory Summary and Instruction Progress (Shi et al., 2 Nov 2025). The MLLM is then prompted to identify completed steps in the instruction, detect misalignment between current heading and intended route, and recommend a re-orientation direction and safe distance (Shi et al., 2 Nov 2025). This mechanism selectively reintroduces panoramic sensing only when local frontal evidence is insufficient.

Future-Past Bidirectional Reasoning is a prompt-level consistency mechanism rather than a separate planner (Shi et al., 2 Nov 2025). “Future” refers to simulating likely visual consequences of candidate actions before execution, such as expecting a hallway leading to a kitchen after turning left (Shi et al., 2 Nov 2025). “Past” refers to reflecting on the previous decision using Previous Selected Image and Previous Thought, and comparing current observations against previous expectations (Shi et al., 2 Nov 2025). History is represented textually through Trajectory Summary, Instruction Progress, Previous Thought, Previous Selected Image, and prior observed objects (Shi et al., 2 Nov 2025). The paper attributes globally coherent planning and consistency improvements to this mechanism (Shi et al., 2 Nov 2025).

The module also includes an anti-stuck heuristic: if the current spatial descriptions $\mathbf{x}_{\text{goal}}$9 are unchanged from the previous ones, the robot performs a slight rightward shift (Shi et al., 2 Nov 2025). This is a minimal but explicit recovery strategy. More broadly, the combination of frontal operation, confusion-triggered panoramic recovery, and history-conditioned reasoning distinguishes Fast-SmartWay from SmartWay’s explicit backtracking-based navigator (Shi et al., 13 Mar 2025, Shi et al., 2 Nov 2025). SmartWay adds a backtrack action option to a waypoint-based MLLM navigator (Shi et al., 13 Mar 2025), whereas Fast-SmartWay addresses comparable failure modes through panoramic disambiguation and prompt-level future-past consistency (Shi et al., 2 Nov 2025).

5. Empirical performance in simulation and on a real robot

Fast-SmartWay is evaluated in Habitat-based simulation on R2R-CE and in real-world deployment on Hello Robot (Shi et al., 2 Nov 2025). The simulator evaluation follows the same 100 episodes used by Open-Nav for direct comparability, and because MLLM outputs are stochastic, each experiment is run four times and averaged (Shi et al., 2 Nov 2025). The reported metrics are Success Rate (SR), Success weighted by Path Length (SPL), normalized Dynamic Time Warping (nDTW), Trajectory Length (TL), and Navigation Error (NE) (Shi et al., 2 Nov 2025).

On R2R-CE, Fast-SmartWay reports

$0^\circ,30^\circ,\dots,330^\circ$0

(Shi et al., 2 Nov 2025). Relative to the panoramic SmartWay baseline,

$0^\circ,30^\circ,\dots,330^\circ$1

Fast-SmartWay is slightly lower on SR but higher on SPL and substantially higher on nDTW while using frontal views instead of panoramas during normal navigation (Shi et al., 2 Nov 2025). It also far exceeds the reproduced frontal-view SmartWay baseline, which reports $0^\circ,30^\circ,\dots,330^\circ$2 and $0^\circ,30^\circ,\dots,330^\circ$3 (Shi et al., 2 Nov 2025).

The ablation results isolate the contribution of the uncertainty-aware components. Without Disambiguation and FPBR, the model achieves $0^\circ,30^\circ,\dots,330^\circ$4 and $0^\circ,30^\circ,\dots,330^\circ$5 (Shi et al., 2 Nov 2025). Adding Disambiguation alone raises performance to $0^\circ,30^\circ,\dots,330^\circ$6 and $0^\circ,30^\circ,\dots,330^\circ$7 (Shi et al., 2 Nov 2025). The full model with both Disambiguation and FPBR reaches $0^\circ,30^\circ,\dots,330^\circ$8 and $0^\circ,30^\circ,\dots,330^\circ$9 (Shi et al., 2 Nov 2025). This supports the paper’s claim that disambiguation helps recover from uncertain local choices while FPBR improves long-horizon consistency.

A compact comparison of the main simulated and real-robot results clarifies the system’s operating point.

Setting	Method	Key reported results
R2R-CE	Fast-SmartWay	SR $t=0$0, SPL $t=0$1, nDTW $t=0$2
R2R-CE	SmartWay	SR $t=0$3, SPL $t=0$4, nDTW $t=0$5
Hello Robot	Fast-SmartWay	Total time $t=0$6 s, SR $t=0$7, NE $t=0$8
Hello Robot	SmartWay	Total time $t=0$9 s, SR $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$0, NE $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$1

On Hello Robot, Fast-SmartWay reports total per-step time $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$2 s, SR $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$3, and NE $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$4, compared with SmartWay’s $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$5 s total time, SR $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$6, and NE $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$7 (Shi et al., 2 Nov 2025). The timing breakdown is especially important. SmartWay’s panoramic setup requires $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$8 s of perception time and $$I_{0}=\{(I^{rgb}_i,I^{depth}_i)\mid i=1,\dots,12\},$$9 s of inference time, whereas Fast-SmartWay requires $t>0$0 s of perception time and $t>0$1 s of inference time, for a total that is $t>0$2 of SmartWay’s (Shi et al., 2 Nov 2025). The paper therefore attributes the speedup primarily to reduced sensing overhead rather than lower MLLM inference latency (Shi et al., 2 Nov 2025).

6. Relation to SmartWay, limitations, and interpretation

Fast-SmartWay is best understood in relation to SmartWay. SmartWay uses an enhanced RGB-D waypoint predictor and an MLLM-based navigator with history-aware reasoning and adaptive path planning with backtracking (Shi et al., 13 Mar 2025). Fast-SmartWay removes the waypoint predictor, uses only three frontal RGB-D views during normal navigation, and replaces explicit backtracking with uncertainty-triggered panoramic disambiguation plus Future-Past Bidirectional Reasoning (Shi et al., 2 Nov 2025). The two systems therefore occupy adjacent points in the design space of zero-shot VLN-CE: SmartWay is a two-stage panoramic candidate-selection framework, whereas Fast-SmartWay is a panoramic-free end-to-end action predictor with selective recovery (Shi et al., 13 Mar 2025, Shi et al., 2 Nov 2025).

Several common misconceptions are addressed by the paper’s structure. Fast-SmartWay is not purely frontal-view from start to finish; it performs a panoramic scan at initialization and again during disambiguation when confusion is detected (Shi et al., 2 Nov 2025). It is also not a trained end-to-end policy in the conventional sense; the method is zero-shot and relies on prompt-based use of GPT-4o-2024-08-06 plus external perception modules such as RAM (Shi et al., 2 Nov 2025). Finally, its efficiency gains do not come from a smaller MLLM. The reported inference time is slightly higher than SmartWay’s, and the practical acceleration comes mainly from reducing panoramic sensing from 12 views to 3 during normal steps (Shi et al., 2 Nov 2025).

The paper also makes its limitations clear, if mostly implicitly. Frontal-only observations can still be ambiguous, which is why the system occasionally needs panoramic rescans (Shi et al., 2 Nov 2025). Uncertainty estimation is prompt-driven rather than statistically calibrated (Shi et al., 2 Nov 2025). GPT-4o inference remains nontrivial in latency, and the exact prompts, parsing templates, and low-level controller details are not fully disclosed (Shi et al., 2 Nov 2025). A plausible implication is that the framework’s practical strength lies less in minimizing absolute compute and more in aligning perception, history, and action generation around the constraints of real front-facing robotic platforms.

In summary, Fast-SmartWay is a zero-shot VLN-CE framework that replaces panoramic waypoint-based navigation with frontal-view end-to-end MLLM action prediction, structured RGB-D-to-text scene abstraction, and selective uncertainty-triggered recovery (Shi et al., 2 Nov 2025). Its significance lies in showing that panoramic sensing and waypoint prediction are not mandatory design primitives for competitive zero-shot embodied navigation, and that a frontal-view architecture can improve real-world latency while remaining competitive or superior in task performance (Shi et al., 2 Nov 2025).