---
title: 'SkillNav: Modular VLN with Skill Routing'
url: https://www.emergentmind.com/topics/skillnav
type: topic
---

# SkillNav: Modular VLN with Skill Routing

Searching arXiv for recent papers on SkillNav and adjacent skill-routing/navigation work.
SkillNav is a modular vision-and-language navigation (VLN) framework that introduces structured, skill-based reasoning into Transformer-based VLN agents. Rather than using a single end-to-end navigator, it combines a mixture of specialized skill agents with a zero-shot vision-language model (VLM)-based router. The framework is motivated by the claim that VLN instructions are compositional and require distinct forms of reasoning, including turning, vertical movement, stopping, landmark recognition, region identification, and temporal order. SkillNav therefore decomposes navigation into atomic reusable skills, applies temporal reordering to make instruction structure explicit, localizes the currently relevant subgoal from visual history, and routes control to the most suitable specialist at each step. On the R2R benchmark it reports state-of-the-art performance, and it also shows strong generalization on GSA-R2R, a benchmark with novel instruction styles and unseen environments [2508.07642].

## 1. Problem formulation and motivation

In standard VLN, the agent follows a natural-language instruction \(I\) in an indoor environment represented as a graph \(\mathcal{G}=(V,E)\). At time \(t\), the agent is at viewpoint \(v_t\), observes panoramas \(D_t=\{o_i\}_{i=0}^n\), has access to the navigable subset \(O_t\subseteq D_t\), and chooses an action from the local action space \(A_t\). SkillNav uses DUET as the base navigator, written as
\[
a_t^* = \pi(I, O_t, M_t),
\]
where \(M_t \subseteq \mathcal{G}\) is the online built topological map [2508.07642].

The framework is motivated by a critique of monolithic VLN policies. Such systems are described as often memorizing patterns rather than learning compositional reasoning, which becomes particularly problematic under distribution shift, including new instructions, stylistic variation, unseen scene types, and long-horizon reasoning. SkillNav responds by modeling navigation as a skill-based mixture of experts, so that different sub-problems are handled by different specialists. This reframes VLN from a single action-prediction problem into a structured set of reasoning patterns, with routing treated as a runtime decision over reusable atomic skills rather than as an implicit consequence of one shared policy [2508.07642].

A plausible implication is that SkillNav is less committed to a single global navigation heuristic than prior end-to-end systems. Instead, it encodes the assumption that instruction-following success depends on repeated transitions between qualitatively different reasoning modes, and that these modes can be learned more reliably when they are isolated, specialized, and routed explicitly.

## 2. Atomic skill taxonomy

SkillNav adopts four atomic skills from NavNuances and adds two more to address temporal and termination failures. The resulting taxonomy defines the framework’s decomposition of VLN into reusable navigation abilities [2508.07642].

| Skill | Role in navigation | Examples noted in the paper |
|---|---|---|
| Direction Adjustment | Turning or changing heading without necessarily changing position | turn left, change heading |
| Vertical Movement | Moving across floors or elevation changes | stairs, elevator |
| Landmark Detection | Recognizing and navigating relative to salient objects | sofa, lamp, painting |
| Area and Region Identification | Recognizing semantic spaces or room transitions | kitchen, hallway, entering/exiting a region |
| Stop and Pause | Deciding whether to stop, pause briefly, or end navigation at a precise point | stop errors, precise termination |
| Temporal Order Planning | Reasoning about subgoal sequence and temporal logic | “once,” “until,” “then,” “before” |

The paper emphasizes that **Stop and Pause** is added because stop errors are a major source of SR/OSR mismatch, and **Temporal Order Planning** is added because direct policy learning often struggles with temporal markers such as “before,” “after,” “until,” and “once.” The temporal category explicitly includes conditional immediacy, bounded duration, forward sequential order, and backward sequential order. This taxonomy is central to the framework’s claim that VLN requires specialized handling of semantic, spatial, temporal, and termination phenomena rather than a single uniform policy [2508.07642].

An important structural detail is that SkillNav does not instantiate all six categories in the same way. The specialized DUET-based experts correspond to direction adjustment, vertical movement, stop/pause, landmark detection, and area/region identification. Temporal reasoning is handled primarily through an LLM-based temporal reordering module rather than through a dedicated low-level navigation policy. This division indicates that the framework separates procedural sequencing from motion control.

## 3. Routing architecture and execution pipeline

SkillNav’s runtime can be understood as a four-stage control loop: temporal reordering of the instruction, subgoal localization, skill routing, and expert execution. This pipeline is the framework’s core contribution [2508.07642].

Before routing, SkillNav applies a temporal reordering module:
\[
I_{\text{reorder}} = \text{LLM}_{\text{TemporalReorder}}(I).
\]
The reordered instruction is a structured sequence of subgoals. The appendix prompt instructs the LLM to break the instruction into minimal goal-directed steps, make implicit relationships explicit, preserve order, and avoid leaving temporal markers in the output. The stated purpose is to make temporal logic explicit so that later routing can operate over executable subgoals rather than over a temporally entangled sentence [2508.07642].

Given reordered subgoals \(I_{\text{reorder}}=[p_1,p_2,\dots,p_m]\), observation history \(H_{t-1}\), and previously executed subgoals \(G_{t-1}\), the localizer selects the next subgoal:
\[
p_t^*, r_t = \text{Localize}(I_{\text{reorder}}, H_{t-1}, G_{t-1}),
\]
followed by
\[
G_t = G_{t-1} \parallel p_t^* .
\]
The localizer is instructed to infer which sub-instructions are already completed, use previous viewpoints to detect implicit completion, skip steps already passed, and return the next exact sub-instruction to execute. In effect, it is a grounded step selector over a decomposed plan [2508.07642].

After subgoal localization, the skill router chooses the best expert from
\[
\mathcal{S}=\{\pi_{\text{da}},\pi_{\text{vm}},\pi_{\text{sp}},\pi_{\text{ld}},\pi_{\text{ar}}\}.
\]
The selection is written as
\[
\pi_t^* = \arg\max_{\pi\in \mathcal{S}} \text{Router}(I,p_t^*,r_t).
\]
The router is conditioned on the original instruction \(I\), the selected subgoal \(p_t^*\), and the reasoning trace \(r_t\). The paper stresses that routing is therefore not based on the subgoal alone. Full instruction context is retained to preserve linguistic cues such as verbs and spatial references, while the reasoning trace provides local context about what has already happened. This is meant to resolve ambiguities in cases where similar subgoals may demand different dominant skills under different contexts [2508.07642].

Finally, the selected specialist executes the action using the DUET-style policy:
\[
a_t^* = \pi_t^*(I,O_t,M_t).
\]
The paper describes the router as zero-shot and in-context, instantiated with large VLMs such as GPT-4o or Qwen2.5-VL. It also reports ablation comparisons with GLM-4.1V-9B-Thinking. This architecture makes SkillNav a mixture-of-skill-agents system rather than a single policy with latent specialization.

## 4. Specialized agents, data synthesis, and training procedure

Each atomic navigation ability is handled by a specialized DUET-based agent. To train these experts, the framework constructs skill-specific synthetic datasets by sampling trajectories in Matterport3D and generating instructions that emphasize one skill at a time with GPT-4o. The datasets are designed so that the target skill is the dominant factor for success [2508.07642].

The reported dataset statistics are specific. The temporal synthetic dataset contains \(2{,}000\) instructions, and each atomic skill dataset contains \(450\) instructions. Natural VLN data used for comparison include R2R and GSA-R2R. Trajectory generation is filtered so that the target skill is the main factor: vertical movement paths require substantial elevation changes; direction adjustment requires significant heading changes; stop and pause requires semantically meaningful pauses or stops; landmark detection requires visible, distinctive objects; and area/region identification requires region transitions. Trajectory length is constrained to **4–7 steps** to keep the synthetic data comparable to natural VLN data [2508.07642].

Training proceeds in two stages. In **Stage 1**, the framework fine-tunes pre-trained DUET on the R2R training split, ScaleVLN augmentation data, and the Temporal Synthetic dataset. In **Stage 2**, it further fine-tunes on the corresponding skill-specific synthetic dataset. The resulting specialists are denoted
\[
\pi_{\text{da}},\ \pi_{\text{vm}},\ \pi_{\text{sp}},\ \pi_{\text{ld}},\ \pi_{\text{ar}}.
\]
The paper reports that, when evaluated individually on NavNuances, each expert performs best on its corresponding category, which is presented as evidence that the decomposition creates genuine specialization rather than merely redundant replicas of the same policy [2508.07642].

The implementation stack is explicit. SkillNav uses **CLIP-B/16** as visual backbone, **BERT-base-uncased** as language backbone, **DUET** as the base navigator, **GPT-4o** for temporal reordering and synthetic instruction generation, **Qwen2.5-VL-7B-Instruct** as the main action router, and **GLM-4.1V-9B-Thinking** in ablation comparisons. Stage 1 fine-tuning uses **50,000 iterations**, **batch size 32**, and **learning rate \(5\times10^{-5}\)**; Stage 2 specialization uses **30,000 iterations** and **batch size 16**. The best temporal backbone is selected by SPL on R2R Val-Unseen [2508.07642].

## 5. Benchmarks, quantitative results, and ablations

SkillNav is evaluated on two benchmarks. **R2R** is the standard Room-to-Room benchmark with unseen validation and test splits. **GSA-R2R** is the framework’s main generalization benchmark for General Scene Adaptation, including residential and non-residential scenes, more diverse instruction styles, role-specific dialogue styles, and splits such as Test-R-Basic, Test-N-Basic, and Test-N-Scene. The reported metrics are **NE**, **OSR**, **SR**, and **SPL** [2508.07642].

On **R2R**, the main table reports **Val-Unseen: SR \(83\%\), SPL \(77\%\)** and **Test-Unseen: SR \(78\%\), SPL \(70\%\)**. The paper states that SkillNav achieves strong state-of-the-art-level performance and notably beats ScaleVLN on both SR and SPL in the reported test setting. On **GSA-R2R**, the paper reports that SkillNav achieves **the best SPL on all GSA-R2R splits**, with particularly strong SR/SPL on more challenging non-residential and scene-specific splits. It contrasts this with R2R-specialized systems such as SRDF, which perform very well on R2R but degrade on GSA-R2R, interpreting the difference as evidence that modular skill structure generalizes better than memorized trajectory patterns [2508.07642].

The paper also evaluates the experts on **NavNuances**, which isolates the categories Direction Change, Vertical Movement, Landmark Recognition, and Room Recognition. The corresponding experts outperform generic baselines in their matched categories: the Vertical Movement agent achieves the best VM SR and SPL, the Direction Adjustment agent leads DC, the Landmark Detection agent performs best in LR, and the Area/Region Identification agent leads RR. This is presented as skill-level evidence that the specialists have acquired category-specific competence rather than only contributing through ensemble diversity [2508.07642].

Ablation results support both major architectural additions. With the same router, adding temporal reordering improves results across all GSA-R2R splits, with the strongest benefit on Test-N-Basic. Comparing random skill selection with the VLM router shows that routing improves both SR and SPL, indicating that expert choice is non-trivial and benefits from grounded multimodal reasoning. Router model comparisons further show that Qwen2.5-VL outperforms GLM-4.1V-Thinking, especially on SPL in complex splits such as Test-N-Scene, which the paper interprets as evidence that stronger visual grounding improves skill choice [2508.07642].

The qualitative examples are aligned with the modular interpretation. The router is shown choosing **stop** once the agent reaches the target pillars, and choosing **vertical movement** when the next instruction requires moving toward stairs. These cases are used to illustrate the intended execution pattern: instruction restructuring into subgoals, subgoal localization from visual history, and subsequent routing to the appropriate expert [2508.07642].

## 6. Position within skill-based agent research and open issues

SkillNav belongs to a broader research program that treats skills as explicit computational objects for decomposition, routing, retrieval, organization, and evaluation rather than as unstructured prompt text. Within this landscape, SkillNav addresses embodied VLN, but adjacent work studies related problems in LLM-agent systems. Query-conditional skill retrieval has been formalized as a set-selection problem with explicit **skill compatibility** in R3-Skill, which argues that routing must account for whether retrieved skills can collaborate under a query, not only for independent relevance [2606.03565]. SkillRouter studies routing over a pool of approximately **80K skills** and reports that removing the skill body causes **29–44 percentage point** degradation across retrieval methods, with **91.7%** of cross-encoder attention concentrated on the body field, thereby challenging the assumption that names and descriptions suffice for large-scale routing [2603.22455]. SelSkill shifts the focus from relevance to **selective invocation**, formulating use as a skill-or-skip decision and reporting gains of **10.9 percentage points** on ALFWorld and **5.7 percentage points** on BFCL [2606.00510].

Other nearby systems emphasize execution and organization. WebXSkill closes a “grounding gap” in autonomous web agents by pairing parameterized action programs with step-level natural-language guidance and indexing them in a URL-based skill graph for context-aware retrieval [2604.13318]. Formal Skill introduces a runtime-native abstraction with JSON metadata, Python executors, lifecycle hooks, runtime state, and routing metadata, moving reusable procedure into executable state machines and hook policies [2605.19604]. SkillJuror isolates how skill organization changes runtime behavior, showing that Progressive Disclosure increases distinct resources touched per trajectory from **1.18** to **3.85** and effective uptake events from **1.33** to **3.92**, with **17** additional verifier-passing trials out of **410** matched trials over a normalized flat baseline [2606.11543].

Against this broader backdrop, SkillNav’s distinctive contribution is to locate skill routing inside an embodied navigation loop. Its experts are not retrieved from a repository at query time; they are learned motion-and-perception specialists that are activated per step from subgoal and observation context. This suggests a complementary relation to the LLM-agent literature: SkillNav studies **intra-policy routing** among navigation experts, whereas many other skill systems study **inter-skill retrieval**, **progressive disclosure**, **skill execution control**, or **repository-scale selection**.

The paper does not present a long explicit limitations section, but several failure modes are stated or implied. **Stop errors** remain a major challenge in VLN generally, which is why a dedicated Stop and Pause skill is added. **Temporal reasoning mistakes** can still produce subgoal-ordering errors if temporal reordering or localization is wrong. Routing quality depends on the VLM’s grounding ability, and weaker routers reduce SPL. The system is also more complex than a single end-to-end navigator because it requires synthetic skill data, multiple specialized models, and multi-stage routing. Finally, the decomposition depends on a fixed skill taxonomy, so unseen skills or more subtle mixed-skill instructions may remain difficult [2508.07642].

Taken together, these characteristics define SkillNav as a skill-aware modular VLN system that decomposes an instruction into ordered subgoals, localizes the current subgoal using instruction and history, routes to the appropriate specialist, and executes with that expert. Its significance lies not only in reported benchmark gains, but in presenting VLN as an explicitly navigable composition of reusable skills rather than as a monolithic policy [2508.07642].

Source: https://www.emergentmind.com/topics/skillnav