SkillNav: Modular VLN with Skill Routing

• SkillNav is a modular vision-and-language navigation framework that decomposes natural language instructions into atomic, reusable navigation skills.
• It uses a zero-shot, vision-language model-based router to select specialized experts for tasks like turning, vertical movement, and landmark detection.
• The framework achieves state-of-the-art performance on R2R and strong generalization on GSA-R2R, validating its skill-based decomposition and routing strategy.

Searching arXiv for papers on SkillNav and adjacent skill-routing/navigation work. SkillNav is a modular vision-and-language navigation (VLN) framework that introduces structured, skill-based reasoning into Transformer-based VLN agents. Rather than using a single end-to-end navigator, it combines a mixture of specialized skill agents with a zero-shot vision-LLM (VLM)-based router. The framework is motivated by the claim that VLN instructions are compositional and require distinct forms of reasoning, including turning, vertical movement, stopping, landmark recognition, region identification, and temporal order. SkillNav therefore decomposes navigation into atomic reusable skills, applies temporal reordering to make instruction structure explicit, localizes the currently relevant subgoal from visual history, and routes control to the most suitable specialist at each step. On the R2R benchmark it reports state-of-the-art performance, and it also shows strong generalization on GSA-R2R, a benchmark with novel instruction styles and unseen environments (Ma et al., 11 Aug 2025).

1. Problem formulation and motivation

In standard VLN, the agent follows a natural-language instruction $I$ in an indoor environment represented as a graph $\mathcal{G}=(V,E)$. At time $t$, the agent is at viewpoint $v_t$, observes panoramas $D_t=\{o_i\}_{i=0}^n$, has access to the navigable subset $O_t\subseteq D_t$, and chooses an action from the local action space $A_t$. SkillNav uses DUET as the base navigator, written as

$a_t^* = \pi(I, O_t, M_t),$

where $M_t \subseteq \mathcal{G}$ is the online built topological map (Ma et al., 11 Aug 2025).

The framework is motivated by a critique of monolithic VLN policies. Such systems are described as often memorizing patterns rather than learning compositional reasoning, which becomes particularly problematic under distribution shift, including new instructions, stylistic variation, unseen scene types, and long-horizon reasoning. SkillNav responds by modeling navigation as a skill-based mixture of experts, so that different sub-problems are handled by different specialists. This reframes VLN from a single action-prediction problem into a structured set of reasoning patterns, with routing treated as a runtime decision over reusable atomic skills rather than as an implicit consequence of one shared policy (Ma et al., 11 Aug 2025).

A plausible implication is that SkillNav is less committed to a single global navigation heuristic than prior end-to-end systems. Instead, it encodes the assumption that instruction-following success depends on repeated transitions between qualitatively different reasoning modes, and that these modes can be learned more reliably when they are isolated, specialized, and routed explicitly.

2. Atomic skill taxonomy

SkillNav adopts four atomic skills from NavNuances and adds two more to address temporal and termination failures. The resulting taxonomy defines the framework’s decomposition of VLN into reusable navigation abilities (Ma et al., 11 Aug 2025).

Skill	Role in navigation	Examples noted in the paper
Direction Adjustment	Turning or changing heading without necessarily changing position	turn left, change heading
Vertical Movement	Moving across floors or elevation changes	stairs, elevator
Landmark Detection	Recognizing and navigating relative to salient objects	sofa, lamp, painting
Area and Region Identification	Recognizing semantic spaces or room transitions	kitchen, hallway, entering/exiting a region
Stop and Pause	Deciding whether to stop, pause briefly, or end navigation at a precise point	stop errors, precise termination
Temporal Order Planning	Reasoning about subgoal sequence and temporal logic	“once,” “until,” “then,” “before”

The paper emphasizes that Stop and Pause is added because stop errors are a major source of SR/OSR mismatch, and Temporal Order Planning is added because direct policy learning often struggles with temporal markers such as “before,” “after,” “until,” and “once.” The temporal category explicitly includes conditional immediacy, bounded duration, forward sequential order, and backward sequential order. This taxonomy is central to the framework’s claim that VLN requires specialized handling of semantic, spatial, temporal, and termination phenomena rather than a single uniform policy (Ma et al., 11 Aug 2025).

An important structural detail is that SkillNav does not instantiate all six categories in the same way. The specialized DUET-based experts correspond to direction adjustment, vertical movement, stop/pause, landmark detection, and area/region identification. Temporal reasoning is handled primarily through an LLM-based temporal reordering module rather than through a dedicated low-level navigation policy. This division indicates that the framework separates procedural sequencing from motion control.

3. Routing architecture and execution pipeline

SkillNav’s runtime can be understood as a four-stage control loop: temporal reordering of the instruction, subgoal localization, skill routing, and expert execution. This pipeline is the framework’s core contribution (Ma et al., 11 Aug 2025).

Before routing, SkillNav applies a temporal reordering module: $$I_{\text{reorder}} = \text{LLM}_{\text{TemporalReorder}}(I).$$ The reordered instruction is a structured sequence of subgoals. The appendix prompt instructs the LLM to break the instruction into minimal goal-directed steps, make implicit relationships explicit, preserve order, and avoid leaving temporal markers in the output. The stated purpose is to make temporal logic explicit so that later routing can operate over executable subgoals rather than over a temporally entangled sentence (Ma et al., 11 Aug 2025).

Given reordered subgoals $\mathcal{G}=(V,E)$0, observation history $\mathcal{G}=(V,E)$1, and previously executed subgoals $\mathcal{G}=(V,E)$2, the localizer selects the next subgoal: $\mathcal{G}=(V,E)$3 followed by

$\mathcal{G}=(V,E)$4

The localizer is instructed to infer which sub-instructions are already completed, use previous viewpoints to detect implicit completion, skip steps already passed, and return the next exact sub-instruction to execute. In effect, it is a grounded step selector over a decomposed plan (Ma et al., 11 Aug 2025).

After subgoal localization, the skill router chooses the best expert from

$\mathcal{G}=(V,E)$5

The selection is written as

$\mathcal{G}=(V,E)$6

The router is conditioned on the original instruction $\mathcal{G}=(V,E)$7, the selected subgoal $\mathcal{G}=(V,E)$8, and the reasoning trace $\mathcal{G}=(V,E)$9. The paper stresses that routing is therefore not based on the subgoal alone. Full instruction context is retained to preserve linguistic cues such as verbs and spatial references, while the reasoning trace provides local context about what has already happened. This is meant to resolve ambiguities in cases where similar subgoals may demand different dominant skills under different contexts (Ma et al., 11 Aug 2025).

Finally, the selected specialist executes the action using the DUET-style policy: $t$0 The paper describes the router as zero-shot and in-context, instantiated with large VLMs such as GPT-4o or Qwen2.5-VL. It also reports ablation comparisons with GLM-4.1V-9B-Thinking. This architecture makes SkillNav a mixture-of-skill-agents system rather than a single policy with latent specialization.

4. Specialized agents, data synthesis, and training procedure

Each atomic navigation ability is handled by a specialized DUET-based agent. To train these experts, the framework constructs skill-specific synthetic datasets by sampling trajectories in Matterport3D and generating instructions that emphasize one skill at a time with GPT-4o. The datasets are designed so that the target skill is the dominant factor for success (Ma et al., 11 Aug 2025).

The reported dataset statistics are specific. The temporal synthetic dataset contains $t$1 instructions, and each atomic skill dataset contains $t$2 instructions. Natural VLN data used for comparison include R2R and GSA-R2R. Trajectory generation is filtered so that the target skill is the main factor: vertical movement paths require substantial elevation changes; direction adjustment requires significant heading changes; stop and pause requires semantically meaningful pauses or stops; landmark detection requires visible, distinctive objects; and area/region identification requires region transitions. Trajectory length is constrained to 4–7 steps to keep the synthetic data comparable to natural VLN data (Ma et al., 11 Aug 2025).

Training proceeds in two stages. In Stage 1, the framework fine-tunes pre-trained DUET on the R2R training split, ScaleVLN augmentation data, and the Temporal Synthetic dataset. In Stage 2, it further fine-tunes on the corresponding skill-specific synthetic dataset. The resulting specialists are denoted

$t$3

The paper reports that, when evaluated individually on NavNuances, each expert performs best on its corresponding category, which is presented as evidence that the decomposition creates genuine specialization rather than merely redundant replicas of the same policy (Ma et al., 11 Aug 2025).

The implementation stack is explicit. SkillNav uses CLIP-B/16 as visual backbone, BERT-base-uncased as language backbone, DUET as the base navigator, GPT-4o for temporal reordering and synthetic instruction generation, Qwen2.5-VL-7B-Instruct as the main action router, and GLM-4.1V-9B-Thinking in ablation comparisons. Stage 1 fine-tuning uses 50,000 iterations, batch size 32, and learning rate $t$4; Stage 2 specialization uses 30,000 iterations and batch size 16. The best temporal backbone is selected by SPL on R2R Val-Unseen (Ma et al., 11 Aug 2025).

5. Benchmarks, quantitative results, and ablations

SkillNav is evaluated on two benchmarks. R2R is the standard Room-to-Room benchmark with unseen validation and test splits. GSA-R2R is the framework’s main generalization benchmark for General Scene Adaptation, including residential and non-residential scenes, more diverse instruction styles, role-specific dialogue styles, and splits such as Test-R-Basic, Test-N-Basic, and Test-N-Scene. The reported metrics are NE, OSR, SR, and SPL (Ma et al., 11 Aug 2025).

On R2R, the main table reports Val-Unseen: SR $t$5, SPL $t$6 and Test-Unseen: SR $t$7, SPL $t$8. The paper states that SkillNav achieves strong state-of-the-art-level performance and notably beats ScaleVLN on both SR and SPL in the reported test setting. On GSA-R2R, the paper reports that SkillNav achieves the best SPL on all GSA-R2R splits, with particularly strong SR/SPL on more challenging non-residential and scene-specific splits. It contrasts this with R2R-specialized systems such as SRDF, which perform very well on R2R but degrade on GSA-R2R, interpreting the difference as evidence that modular skill structure generalizes better than memorized trajectory patterns (Ma et al., 11 Aug 2025).

The paper also evaluates the experts on NavNuances, which isolates the categories Direction Change, Vertical Movement, Landmark Recognition, and Room Recognition. The corresponding experts outperform generic baselines in their matched categories: the Vertical Movement agent achieves the best VM SR and SPL, the Direction Adjustment agent leads DC, the Landmark Detection agent performs best in LR, and the Area/Region Identification agent leads RR. This is presented as skill-level evidence that the specialists have acquired category-specific competence rather than only contributing through ensemble diversity (Ma et al., 11 Aug 2025).

Ablation results support both major architectural additions. With the same router, adding temporal reordering improves results across all GSA-R2R splits, with the strongest benefit on Test-N-Basic. Comparing random skill selection with the VLM router shows that routing improves both SR and SPL, indicating that expert choice is non-trivial and benefits from grounded multimodal reasoning. Router model comparisons further show that Qwen2.5-VL outperforms GLM-4.1V-Thinking, especially on SPL in complex splits such as Test-N-Scene, which the paper interprets as evidence that stronger visual grounding improves skill choice (Ma et al., 11 Aug 2025).

The qualitative examples are aligned with the modular interpretation. The router is shown choosing stop once the agent reaches the target pillars, and choosing vertical movement when the next instruction requires moving toward stairs. These cases are used to illustrate the intended execution pattern: instruction restructuring into subgoals, subgoal localization from visual history, and subsequent routing to the appropriate expert (Ma et al., 11 Aug 2025).

6. Position within skill-based agent research and open issues

SkillNav belongs to a broader research program that treats skills as explicit computational objects for decomposition, routing, retrieval, organization, and evaluation rather than as unstructured prompt text. Within this landscape, SkillNav addresses embodied VLN, but adjacent work studies related problems in LLM-agent systems. Query-conditional skill retrieval has been formalized as a set-selection problem with explicit skill compatibility in R3-Skill, which argues that routing must account for whether retrieved skills can collaborate under a query, not only for independent relevance (Wang et al., 2 Jun 2026). SkillRouter studies routing over a pool of approximately 80K skills and reports that removing the skill body causes 29–44 percentage point degradation across retrieval methods, with 91.7% of cross-encoder attention concentrated on the body field, thereby challenging the assumption that names and descriptions suffice for large-scale routing (Zheng et al., 23 Mar 2026). SelSkill shifts the focus from relevance to selective invocation, formulating use as a skill-or-skip decision and reporting gains of 10.9 percentage points on ALFWorld and 5.7 percentage points on BFCL (Chen et al., 30 May 2026).

Other nearby systems emphasize execution and organization. WebXSkill closes a “grounding gap” in autonomous web agents by pairing parameterized action programs with step-level natural-language guidance and indexing them in a URL-based skill graph for context-aware retrieval (Wang et al., 14 Apr 2026). Formal Skill introduces a runtime-native abstraction with JSON metadata, Python executors, lifecycle hooks, runtime state, and routing metadata, moving reusable procedure into executable state machines and hook policies (Zhang et al., 19 May 2026). SkillJuror isolates how skill organization changes runtime behavior, showing that Progressive Disclosure increases distinct resources touched per trajectory from 1.18 to 3.85 and effective uptake events from 1.33 to 3.92, with 17 additional verifier-passing trials out of 410 matched trials over a normalized flat baseline (Chen et al., 10 Jun 2026).

Against this broader backdrop, SkillNav’s distinctive contribution is to locate skill routing inside an embodied navigation loop. Its experts are not retrieved from a repository at query time; they are learned motion-and-perception specialists that are activated per step from subgoal and observation context. This suggests a complementary relation to the LLM-agent literature: SkillNav studies intra-policy routing among navigation experts, whereas many other skill systems study inter-skill retrieval, progressive disclosure, skill execution control, or repository-scale selection.

The paper does not present a long explicit limitations section, but several failure modes are stated or implied. Stop errors remain a major challenge in VLN generally, which is why a dedicated Stop and Pause skill is added. Temporal reasoning mistakes can still produce subgoal-ordering errors if temporal reordering or localization is wrong. Routing quality depends on the VLM’s grounding ability, and weaker routers reduce SPL. The system is also more complex than a single end-to-end navigator because it requires synthetic skill data, multiple specialized models, and multi-stage routing. Finally, the decomposition depends on a fixed skill taxonomy, so unseen skills or more subtle mixed-skill instructions may remain difficult (Ma et al., 11 Aug 2025).

Taken together, these characteristics define SkillNav as a skill-aware modular VLN system that decomposes an instruction into ordered subgoals, localizes the current subgoal using instruction and history, routes to the appropriate specialist, and executes with that expert. Its significance lies not only in reported benchmark gains, but in presenting VLN as an explicitly navigable composition of reusable skills rather than as a monolithic policy (Ma et al., 11 Aug 2025).