VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions

Abstract: Conventional Vision-and-Language Navigation (VLN) benchmarks assume instructions are feasible and the referenced target exists, leaving agents ill-equipped to handle false-premise goals. We introduce VLN-NF, a benchmark with false-premise instructions where the target is absent from the specified room and agents must navigate, gather evidence through in-room exploration, and explicitly output NOT-FOUND. VLN-NF is constructed via a scalable pipeline that rewrites VLN instructions using an LLM and verifies target absence with a VLM, producing plausible yet factually incorrect goals. We further propose REV-SPL to jointly evaluate room reaching, exploration coverage, and decision correctness. To address this challenge, we present ROAM, a two-stage hybrid that combines supervised room-level navigation with LLM/VLM-driven in-room exploration guided by a free-space clearance prior. ROAM achieves the best REV-SPL among compared methods, while baselines often under-explore and terminate prematurely under unreliable instructions. VLN-NF project page can be found at https://vln-nf.github.io/.

• The paper introduces VLN-NF, which extends traditional vision-and-language navigation by explicitly reasoning about false-premise instructions and providing a NOT-FOUND output when targets are absent.
• The proposed methodology uses a two-stage ROAM framework, integrating a DUET-based room locator with LLM/VLM-driven in-room exploration enhanced by the FREE geometric prior.
• Empirical results indicate that ROAM improves the REV-SPL metric by 45% compared to conventional methods, demonstrating robust evidence-based performance in uncertain navigation scenarios.

Feasibility-Aware VLN under False-Premise Instructions: An Expert Analysis of "VLN-NF" (2604.10533)

Introduction and Motivation

The paper "VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions" (2604.10533) introduces a new paradigm in Vision-and-Language Navigation (VLN), addressing the critical gap where conventional benchmarks presume all instructions are feasible and targets necessarily exist. In real-world deployments, agent interaction with human instructions is often error-prone, leading to false-premise goals. VLN agents must not only interpret visual-linguistic instructions but also reason about the absence of described entities under partial observability, outputting an explicit NOT-FOUND when appropriate.

Figure 1: Illustration of agent failure modes in VLN with unreliable instructions—standard VLN lacks NOT-FOUND output, naïve NOT-FOUND action induces premature termination, and ROAM+VLN-NF enables evidence-based exploration and correct abstention.

VLN-NF Benchmark and Dataset Construction

VLN-NF is systematically constructed atop the Matterport3D simulation environment, extending the REVERIE dataset. The construction pipeline features a dual-stage process: instruction rewriting using a LLM to substitute plausible but absent target objects, and automated verification via a Visual LLM (VLM) to ensure the substituted target's absence.

Figure 2: Scalable pipeline for creating NOT-FOUND instructions, combining LLM rewrites (commonsense-driven substitution) and VLM-based absence verification for high-fidelity false-premise instances.

Paired instructions (FOUND and NOT-FOUND) share identical navigation contexts, enabling controlled evaluation of agent behavior under both feasible and infeasible instructions. Manual audit yields less than 2% error, indicating strong reliability in absence labeling. Reference exploration protocols are provided for supervision and evaluation, considering both spatial landmark cue (localized search) and global room coverage scenarios.

Task Definition and Evaluation Metrics

VLN-NF extends standard VLN with explicit NOT-FOUND logic: the agent must (1) reach the targeted room, (2) evidence-gathering search for the object, and (3) declare FOUND or NOT-FOUND. The evaluation is anchored in Reach–Explore–Verify SPL (REV-SPL), which captures:

• Room reaching success,
• Decision accuracy (correct abstention or detection),
• In-room evidence coverage,
• Normalized efficiency relative to reference exploration.

REV-SPL aligns incentives against premature stopping and unsupported NOT-FOUND predictions, reflecting real-world operational requirements for robust navigation under uncertain instructions.

Methodology: ROAM Framework

The paper introduces ROAM (Room-Object Aware Movement), a two-stage hybrid framework integrating weakly supervised room-level navigation and LLM/VLM-driven in-room exploration. The first stage leverages a supervised dual-scale graph transformer (DUET) for robust room localization; the second employs an LLM-planner, semantically informed by VLM captions and a geometric prior derived from the Free-space Raycasting Estimation Engine (FREE).

Figure 3: ROAM architecture—Left: supervised navigator guides to room entry; Right: LLM/VLM explorer uses semantic context and FREE geometric prior for thorough in-room search and NOT-FOUND decision.

FREE augments the agent’s spatial reasoning by raycasting over segmented floor regions and integrating clearance cues (distance to obstacles), guiding exploration toward high coverage and avoiding redundant trajectories—a critical enhancement given the documented geometric reasoning limitations of LLMs.

Figure 4: FREE engine segments floor regions and uses depth-based raycasting to estimate free-space clearance for candidate headings, informing coverage-oriented exploration.

Figure 5: Raycasting halts at door frame detection, preventing erroneous distance estimates beyond room boundaries.

Empirical Results and Analysis

Experimental evaluation on VLN-NF val_unseen split demonstrates that conventional VLN agents (e.g., DUET) achieve strong room-reaching but under-explore in-room, with notably low REV-SPL (4.2). LLM-based baselines (NavGPT, MapGPT, SoM-Gemini-2.0) exhibit unreliable navigation and premature abstention, with REV-SPLs ranging from 1.0 to 3.2. ROAM outperforms all baselines, achieving 6.1 REV-SPL—a 45% improvement over supervised agents, despite requiring only room-level annotations.

Detailed ablation confirms the necessity of the two-stage architecture as well as geometric priors: isolating the LLM explorer yields <1.0 REV-SPL, while adding FREE increases coverage and overall efficiency. Interestingly, ROAM maintains strong performance even when transferred to feasible-only VLN evaluation (REVERIE), indicating that its coverage-aware exploration generalizes beyond false-premise scenarios.

Error Analysis and Robustness

Audit of false NOT-FOUND predictions on feasible instructions reveals that most errors stem from room-reaching failures (55.7%), followed by perception/grounding uncertainty (31.0%), and exploration-control issues (13.3%), underscoring that NOT-FOUND logic amplifies agent calibration and verification demands.

Implications and Future Directions

The practical implication of VLN-NF and ROAM extends to any embodied navigation system facing unreliable, ambiguous, or erroneous instructions—enabling explicit abstention bolsters safety and reliability in real-world environments with partial observability and imperfect human input. The theoretical contribution lies in operationalizing absence verification via efficient exploration, rather than overfitting task priors or assuming full observability.

VLN-NF establishes a tractable benchmark for evidence-grounded NOT-FOUND decisions, paving the way for future research in principled absence reasoning, clarification/recovery policies, and robust embodied dialogue. The scalable pipeline is adaptable to plural false-premise variants (attribute errors, ambiguous referents) and richer behavioral objectives beyond abstention (e.g., interactive disambiguation).

Conclusion

VLN-NF and ROAM jointly advance the field by systematizing feasibility-aware VLN, validating hybrid navigation architectures, and establishing rigorous evaluation grounded in evidence-gathering and explicit abstention decisions. These contributions support safer, more accountable navigation in complex environments and will underpin future developments in embodied AI, robust instruction-following, and uncertainty-aware agent architectures.