InstructNav: Zero-Shot Navigation

• InstructNav is an instruction-guided navigation framework that interprets natural language commands to generate real-time motion plans in unexplored settings.
• It integrates online perception with dynamic planning modules, including Dynamic Chain-of-Navigation and Multi-sourced Value Maps, to unify route-following and semantic search.
• Empirical results show zero-shot success on tasks like ObjectNav and R2R-CE, while later evaluations highlight the need for enhanced spatial reasoning and metric-aware prompts.

InstructNav denotes a class of embodied navigation systems in which an agent interprets natural-language instructions with respect to its current perceptual state and converts them into executable motion in previously unseen environments. In its narrow, paper-specific sense, the term refers to the zero-shot system introduced as “InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment,” which targets object-goal navigation, visual-language navigation, and demand-driven navigation without navigation-specific training or pre-built maps (Long et al., 2024). In later usage, the term is also applied more broadly to instruction-conditioned navigation stacks and adjacent systems that combine language reasoning, online perception, and spatial decision making under open-world uncertainty (He et al., 24 Dec 2025).

1. Scope and conceptual framing

The core InstructNav problem is instruction-guided robot navigation in indoor environments where a robot receives a natural-language instruction $I$, observes egocentric RGB, depth, and pose information over time, and must choose low-level actions $a_t$ to reach the desired destination. In the 2024 formulation, the observation history is written as $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$, and the setting is explicitly continuous, unexplored, and map-free at the start of an episode (Long et al., 2024).

A defining claim of the original system is that instruction navigation should not be restricted to one linguistic regime. The paper distinguishes three task families: ObjectNav instructions such as “Find a sofa,” route-like visual-language navigation instructions such as “Walk to the piano. Move forward to arched wooden doors. Turn right to bathtub,” and demand-driven instructions such as “I am thirsty,” which require commonsense inference about appropriate destinations (Long et al., 2024). This broadened task definition is what gives the system its “generic instruction navigation” character.

Later work uses “InstructNav” in a wider sense. Several papers describe their own systems as “InstructNav-style” or “InstructNav-like” when they decompose language into intermediate navigation decisions or combine language grounding with downstream planning modules rather than learning a monolithic policy. This suggests that the term has evolved from the title of a specific system into a broader label for instruction-conditioned embodied navigation pipelines (Duan et al., 11 Jun 2025).

2. Antecedents and intellectual lineage

The broader research lineage of InstructNav begins with grounded instruction following rather than with the 2024 system itself. “A new dataset and model for learning to understand navigational instructions” formalized the task as mapping a word sequence $w = (w_1,\dots,w_I)$ and time-varying perceptual sequence $p_{1:T}$ to an action sequence $\mathbf{a}=(a_1,\dots,a_T)$ in a maze domain with actions $$\{\text{MOVE}, \text{RIGHT}, \text{LEFT}, \text{STOP}\}$$. That work also emphasized that instruction following is neither pure language understanding nor pure planning, but grounded navigation under perceptual state (Can et al., 2018). Its critique of the small and unbalanced SAIL benchmark, together with the introduction of the controllable SAILx generator, foreshadowed later concerns about evaluation breadth, subskill balance, and compositional generalization in instruction navigation (Can et al., 2018).

A second important precursor is sub-instruction decomposition. “Sub-Instruction Aware Vision-and-Language Navigation” introduced FGR2R, aligned instruction fragments to sub-paths, and added sub-instruction attention and shifting modules so that an agent attends to one sub-instruction at a time and predicts when to advance to the next fragment (Hong et al., 2020). This work operationalized long-horizon instruction following as sequential execution of locally grounded subgoals, a pattern that later reappears in systems that convert language into ordered action–landmark pairs or dynamically select currently relevant clauses (Hong et al., 2020).

More recent work further sharpened the argument that static instruction embeddings are mismatched to embodied control. “Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation” models instruction meaning as an evolving state $S_t = f(S_{t-1},\, S_0,\, I,\, V_t)$, with coarse sub-instruction activation and fine token-level semantic refinement under current observation (Liu et al., 20 Apr 2026). This suggests that a strong InstructNav system should not only parse instructions once, but continuously reinterpret them as the scene changes.

3. The original InstructNav system

The 2024 InstructNav architecture is organized around two principal components: Dynamic Chain-of-Navigation (DCoN) and Multi-sourced Value Maps. The system receives an instruction $I$ together with RGB, depth, and pose, constructs online scene representations from RGB-D observations, asks a LLM to infer the next navigation step in a structured format, and grounds that step into spatial maps whose fused maximum defines the next waypoint (Long et al., 2024).

DCoN is the system’s high-level planning abstraction. It is motivated by the claim that many navigation tasks can be expressed as repeated “action–landmark” decisions, yielding a chain such as “Action 1 - Landmark 1 $\to$ Action 2 - Landmark 2.” The planner is dynamic because the next pair is recomputed online from the current instruction, previous plan, and observed scene objects. The output format is explicitly structured as $a_t$0, and the prompt includes robot definition, navigation strategy, prediction format, and episode information (Long et al., 2024). The stated purpose is to unify route following, semantic search, and commonsense demand satisfaction within one planning representation.

The second major component is the Multi-sourced Value Map machinery. InstructNav constructs four maps: a Semantic Value Map $a_t$1, an Action Value Map $a_t$2, a Trajectory Value Map $a_t$3, and an Intuition Value Map $a_t$4 (Long et al., 2024). The Semantic Value Map scores navigable points by distance to the current landmark point cloud: $a_t$5 followed by normalized semantic scores

$a_t$6

The Trajectory Value Map similarly scores points by distance from previously visited locations: $a_t$7

$a_t$8

The Action Value Map encodes directional sectors for actions such as “Move Forward,” “Turn Left,” or “Explore,” with exploration implemented as value on the boundaries of the explored environment. The Intuition Value Map uses a multimodal large model over a panoramic image $a_t$9, the instruction $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$0, and the current DCoN action–landmark pair $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$1: $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$2 These maps are fused by direct summation,

$O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$3

and the next waypoint is the maximizer

$O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$4

A* then plans to that waypoint, and execution is performed by a rotate-then-forward controller in simulation or by Navigation2 on the real robot (Long et al., 2024).

The system’s claim to genericity rests on the fact that DCoN may emit search-oriented plans such as Explore - Television, route-following pairs such as Move Forward - Doorway, or demand-conditioned target inferences such as Approach - Bottle, while the same map-based backend converts those symbolic steps into motion (Long et al., 2024).

4. Empirical performance and subsequent re-evaluation

The original paper reports three headline results. On HM3D ObjectNav, InstructNav is listed with NE $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$5, SR $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$6, and SPL $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$7, outperforming the zero-shot baselines reported in the same table (Long et al., 2024). On R2R-CE, it is described as completing the task in a zero-shot way for the first time under the paper’s fairness criterion excluding Matterport3D waypoint-predictor reliance (Long et al., 2024). On DDN, it reports TL $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$8, SR $O_t = \{V_{1:t}, D_{1:t}, P_{1:t}\}$9, and SPL $w = (w_1,\dots,w_I)$0, substantially above the listed prior methods (Long et al., 2024). The same paper also states that the best setting for panoramic intuition uses $w = (w_1,\dots,w_I)$1 views and presents real-robot results on a Turtlebot 4 with an ORBBEC Astra Pro Plus RGB-D camera and RPLIDAR-A1 lidar in offices, apartments, a library, a gallery, and a teaching building (Long et al., 2024).

A later re-evaluation substantially altered how these results were interpreted. “When Engineering Outruns Intelligence: A Re-evaluation of Instruction-Guided Navigation” removes DCoN, GLEE, and the Intuition map and replaces them with a simple Distance-Weighted Frontier Explorer (DWFE), while keeping the occupancy mapping, frontier computation, affordance-map planning, and A* substrate (Aghaei et al., 26 Jul 2025). On 2,000 HM3D-v1 validation episodes, this geometry-only replacement raises Success from $w = (w_1,\dots,w_I)$2 to $w = (w_1,\dots,w_I)$3 and SPL from $w = (w_1,\dots,w_I)$4 to $w = (w_1,\dots,w_I)$5 (Aghaei et al., 26 Jul 2025). The paper argues that, in this benchmark setting, frontier geometry explains more of the observed performance than emergent LLM reasoning, and that stronger claims about language-driven planning would require metric-aware prompts or offline semantic graphs (Aghaei et al., 26 Jul 2025). This re-evaluation has become central to later discussions of InstructNav because it separates the value of the classical mapping-and-planning substrate from the contribution of the language-heavy selector modules.

5. Extensions, neighboring systems, and broadened usage

Subsequent literature uses InstructNav both as a concrete baseline and as a reference architecture for adjacent problems. In assistive and behavior-constrained navigation, NORM-Nav narrows the problem to goal-directed motion under natural-language behavioral constraints such as “keep right,” “avoid entering the grass,” or “slow down near the car,” using an LLM to parse constraints and multi-layer costmaps to ground them (Huo et al., 16 May 2026). Although this is not full free-form route following, it occupies an InstructNav-like slice of natural-language navigation by injecting language-specified behavioral semantics into a standard planner.

In long-horizon semantic navigation, $w = (w_1,\dots,w_I)$6 shifts attention from explicit route instructions to high-level human intents such as “I want coffee” or “Help me hang the clothes on the balcony.” It decomposes the problem into a global policy that infers a likely target object $w = (w_1,\dots,w_I)$7 and region $w = (w_1,\dots,w_I)$8 from a semantically annotated 3D map, and a local policy, NaviAfford, that performs open-vocabulary spatial grounding for the final object or affordance target (Zhang et al., 6 Aug 2025). This suggests one future direction for InstructNav: replacing explicit language-following with instruction-to-goal inference plus region-level reasoning when human commands are underspecified.

In dynamic environments, ComposableNav tackles a different but related problem: compound instructions specifying motion constraints relative to moving people and regions, such as overtaking while staying on one side of the road. It assumes that an instruction decomposes into binary specifications $w = (w_1,\dots,w_I)$9, learns one diffusion planner per primitive, and composes them through additive denoising guidance at inference time (Hu et al., 22 Sep 2025). This is not a generic zero-shot language stack, but it shows how instruction-conditioned navigation can scale compositionally when the instruction semantics are represented as reusable primitives.

Other works connect to the instruction side of the loop by generating navigation language rather than following it. InstruGen builds path–instruction pairs from YouTube house-tour trajectories, combining room labels, action annotations, and large multimodal generation with consistency verification (Yan et al., 2024). NavRAG instead generates “user demand instructions” from hierarchical scene description trees and simulated user roles, producing more than 2 million instructions across 861 scenes and emphasizing destination semantics and affordances rather than route narration (Wang et al., 16 Feb 2025). These systems imply that one bottleneck for generic InstructNav remains the scarcity of realistic, diverse, well-grounded instruction data.

6. Limitations, controversies, and research directions

Several limitations recur across the InstructNav literature. The original system depends on pretrained closed-source models, explicitly GPT-4 for DCoN and GPT-4V for the Intuition Value Map, and the paper itself notes that semantic map quality degrades under occlusion because the Semantic Value Map depends on segmented observations (Long et al., 2024). The re-evaluation literature adds a sharper criticism: because DCoN is not given metric information and the open-vocabulary detector can be noisy, the heaviest language modules may contribute less than the geometry-first exploration policy beneath them (Aghaei et al., 26 Jul 2025).

Domain transfer is another recurring difficulty. PM-Nav argues that InstructNav performs poorly in functional buildings because it does not use prior maps, treats navigation as targeting specific objects, and cannot robustly disambiguate rooms with identical appearances. In the reported functional-building experiments, InstructNav achieves very low SR and SPL, including zero success on several medium and hard settings, whereas PM-Nav adds priori-map guidance, hierarchical chain-of-thought prompting over annotated prior maps, and multi-model action refinement (Gao et al., 10 Mar 2026). A plausible implication is that generic instruction navigation in complex institutional environments may require stronger topological priors than those assumed by zero-shot open-world systems.

Evaluation itself remains contested. NavSpace argues that prior benchmarks emphasize semantic grounding but do not systematically test spatial intelligence, introducing six task categories—Vertical Perception, Precise Movement, Viewpoint Shifting, Spatial Relationship, Environment State, and Space Structure—over 1,228 trajectory–instruction pairs (Yang et al., 9 Oct 2025). Its results show that both generic MLLMs and many navigation models remain weak on closed-loop spatial instruction execution, particularly for precise movement and structural reasoning (Yang et al., 9 Oct 2025). This suggests that future InstructNav systems will likely need stronger explicit spatial representations, history-aware reasoning, and better perception-to-action consistency than current language-heavy stacks provide.

A broader trend in later work is to treat instruction meaning as dynamic, scene-conditioned, and tightly coupled to intermediate spatial representations. This is visible in state-conditioned instruction semantics (Liu et al., 20 Apr 2026), imagination over unobserved space in InstructNav-like decision stacks (He et al., 24 Dec 2025), and behavior-constrained costmap formulations (Huo et al., 16 May 2026). Taken together, these developments suggest that “InstructNav” has come to denote not merely one 2024 architecture, but a continuing research program: unifying natural-language intent interpretation, online environment modeling, and executable navigation across instruction types, embodiment constraints, and deployment regimes.