ActiveVLN: Dynamic Vision-Language Navigation

• ActiveVLN is a family of vision-and-language navigation methods where embodied agents dynamically combine perception, reasoning, exploration, and control.
• It employs frameworks like multi-turn reinforcement learning and hierarchical skill scheduling to enhance route discovery, self-correction, and active decision-making.
• Applications range from ground robots to indoor drones and UAVs, showcasing improvements in dynamic viewpoint selection, obstacle avoidance, and memory-driven backtracking.

ActiveVLN denotes a family of Vision-and-Language Navigation formulations in which an embodied agent actively coordinates perception, reasoning, exploration, and control while following natural-language instructions under partial observability. In the recent literature, the term spans continuous indoor VLN for ground robots, long-horizon embodied navigation with large vision-LLMs, and UAV navigation in open 3D environments; the common emphasis is that the agent does not merely map observations to actions, but dynamically selects viewpoints, recovers from ambiguity, seeks additional information, backtracks when needed, and decides when and how to terminate motion or land (Xin et al., 18 Mar 2026, Zhang et al., 16 Sep 2025, Xu et al., 15 Mar 2026, Dominguez-Dager et al., 5 Feb 2026, Li et al., 9 Jul 2025).

1. Conceptual scope and formal problem statements

Vision-and-Language Navigation requires an embodied agent to follow natural-language instructions and navigate in unseen environments. In ActiveVLN, “active” emphasizes autonomy and continuous decision-making: the agent must reason under uncertainty and imperfect localization, adjust heading and altitude continuously in aerial settings, avoid obstacles, choose informative viewpoints, and determine when to stop or land (Xu et al., 15 Mar 2026, Li et al., 9 Jul 2025).

Several formalizations appear in the literature. AgentVLN models VLN as a Partially Observable Semi-Markov Decision Process jointly conditioned on language and temporal memory,

$M = \langle S, O, F, T, I, H\rangle,$

where $S$ is the state space, $O$ the observation space, $F$ the skill library, $T$ the semi-Markov transition model, $I$ the instruction, and $H$ the multimodal temporal history. The high-level controller schedules skills from the library through

$$c_k \sim \pi_\theta(f \mid H_{t_k}, o_{t_k}, I), \qquad f \in F,$$

with perception skills assigned holding time $\tau = 0$ and planning skills assigned $\tau > 0$ (Xin et al., 18 Mar 2026).

The multi-turn RL formulation called ActiveVLN defines an episode by $S$0, where $S$1 is the initial agent state, $S$2 the instruction, and $S$3 the target. At time $S$4 the agent receives a forward-facing RGB observation $S$5 and produces a low-level action $S$6 in a continuous environment. The policy is written as

$S$7

with $S$8 containing the history of past observations and actions. The paper distinguishes the common single-turn form

$S$9

from the multi-turn form

$O$0

arguing that the latter better supports trajectory-level credit assignment (Zhang et al., 16 Sep 2025).

For UAV settings, the problem expands from planar navigation to 3D control. AerialVLA states that UAV VLN requires the agent to manage 6-DoF state dynamics while acting in 3D open-world environments, with the additional difficulty of continuous control under gravitational and inertial constraints (Xu et al., 15 Mar 2026). VLN-Pilot similarly casts the indoor drone setting as a POMDP $O$1 with state variables including pose, room identity, door states, and latent object states, and observations including egocentric images, a topological map, controller state, and previous command (Dominguez-Dager et al., 5 Feb 2026). SkyVLN formalizes the UAV’s initial pose as

$O$2

and couples language-conditioned planning to continuous 3D motion with dynamic obstacle avoidance (Li et al., 9 Jul 2025).

These formulations collectively define ActiveVLN less as a single benchmark and more as a design stance: the agent must use action to reduce uncertainty, not only execute a precomputed route.

2. Architectural regimes in ActiveVLN

Recent ActiveVLN systems fall into several architectural regimes. AgentVLN adopts a “VLM-as-Brain” paradigm in which a VLM, instantiated as Qwen2.5-VL-3B, serves as the central controller for high-level semantic reasoning, instruction grounding, and skill scheduling, while perception and planning are explicitly decoupled into a plug-and-play skill library. Perception skills perform SLAM, occupancy mapping, waypoint extraction, and 2D–3D projections; planning skills generate and track 3D waypoints and produce low-level controls over multiple timesteps (Xin et al., 18 Mar 2026).

AerialVLA represents the opposite pole: a minimalist end-to-end Vision-Language-Action model that maps raw dual-view images and fuzzy onboard hints directly to continuous 3-DoF actions and an intrinsic landing signal. Its backbone is OpenVLA-7B with a Llama 2 LLM, hybrid SigLIP and DINOv2 visual encoders, LoRA adaptation of the language backbone, and a fully fine-tuned visual projector for aerial embodiment (Xu et al., 15 Mar 2026).

VLN-Pilot is hierarchical but not learned end-to-end. It uses a VLLM supervisor, instantiated with GPT-4.1 or Gemini-2.5-Flash, plus an FSM executor and a Python controller connected to a Unity-based indoor simulator. The VLLM receives the front image, user query, topological map, current FSM state, and previous action, then outputs a JSON record containing room estimate, movement command, next FSM state, brief visual description, and door position (Dominguez-Dager et al., 5 Feb 2026).

SkyVLN combines an LLM-based navigation agent with GroundingDINO for landmark detection, a fine-grained High-resolution Spatial Descriptor for spatial verbalization, a TrackBack Memory Array for graph-based memory and backtracking, and an NMPC controller for dynamic obstacle avoidance and reference tracking (Li et al., 9 Jul 2025). The result is neither purely end-to-end nor purely symbolic: language grounding, memory-driven route recovery, and safety-constrained continuous control are treated as separate but coupled layers.

A compact summary of these regimes is given below.

System	Embodiment and control	Defining active mechanism
AgentVLN	Ground agent; skill-based semi-Markov execution	VLM-as-Brain, cross-space mapping, self-correction, QD-PCoT
ActiveVLN	Ground agent; low-level actions in continuous environments	Multi-turn RL with open-ended active exploration
AerialVLA	UAV; continuous 3-DoF actions plus LAND	End-to-end control from dual-view images and fuzzy hints
VLN-Pilot	Indoor UAV; safety-wrapped discrete primitives	VLLM supervisor with FSM-based dynamic re-planning
SkyVLN	Urban UAV; continuous 3D motion with NMPC	HSD, TBMA, panorama-based viewpoint selection, obstacle-aware control

This architectural diversity suggests that ActiveVLN is not defined by a single network topology. The recurring pattern is that perception, memory, control, and language are organized so that the agent can actively interrogate and reshape its own navigation context.

3. Mechanisms of activity: exploration, correction, and information seeking

A central mechanism in AgentVLN is cross-space mapping, introduced to resolve the disconnect between 3D trajectories and the 2D visual reasoning space of pre-trained VLMs. Given a feasible world path point $O$3, its image coordinates are computed through

$O$4

and projected waypoints are rendered as pixel-aligned visual prompts on the current RGB image. The VLM selects among these prompts in pixel space, after which the chosen 2D selection is lifted back to a 3D waypoint for planning control. AgentVLN complements this bridge with context-aware self-correction: when valid global path projections are absent or semantically mismatched, the agent switches to fine-grained local actions

$O$5

to expand the field of view, recover from occlusions, and re-establish waypoint visibility. Its Query-Driven Perceptual Chain-of-Thought adds a further active step: the VLM may generate a natural-language query such as “How far is the chair?”, receive geometric feedback, and then produce target pixel coordinates for calibrated 2D-to-3D lifting (Xin et al., 18 Mar 2026).

The multi-turn RL framework named ActiveVLN operationalizes activity differently. Rather than querying geometry or invoking modular skills, it samples multiple rollouts per instruction, compares them within a group, and reinforces stronger trajectories without expert reward shaping. The paper defines active exploration as self-directed interaction with the environment to discover diverse, plausible navigation routes that may deviate from expert demonstrations (Zhang et al., 16 Sep 2025). This is an explicitly route-discovery view of activity.

In UAV ActiveVLN, AerialVLA treats autonomy as the removal of what it calls the “double crutches” of dense oracle guidance and external open-set detectors. Directional prompts are derived solely from onboard sensors and discretized into coarse buckets: $O$6 corresponds to “straight ahead,” $O$7 to “forward-right” or “forward-left,” $O$8 to “to your right” or “to your left,” and $O$9 to “to your right rear” or “to your left rear.” The policy then grounds the instruction reactively from current observation and fuzzy hint, rather than from dense oracle step annotations (Xu et al., 15 Mar 2026).

SkyVLN makes viewpoint selection explicit. The UAV rotates to collect panoramas, GroundingDINO detects landmarks across views, and HSD partitions each view into a $F$0 grid with sector labels $F$1 so that landmarks can be verbalized with fine spatial anchors. When ambiguity persists, TBMA recovers feasible routes through shortest-path search over a graph of previously encountered landmarks and inter-landmark instructions (Li et al., 9 Jul 2025). VLN-Pilot uses a more heuristic form of activity: state-specific FSM rules encourage alternating rotations and short forward motions to search for doors or objects, while lateral motions and fine yaw corrections center a doorway before traversal (Dominguez-Dager et al., 5 Feb 2026).

A persistent misconception is that ActiveVLN is synonymous with “interactive prompting” alone. The literature indicates a broader meaning. Activity may take the form of active prompt construction, active self-correction, active information seeking, active panorama acquisition, active backtracking, or active rollout generation. What unifies these mechanisms is not a particular module, but the use of action to improve the agent’s informational state.

4. Learning paradigms and optimization strategies

ActiveVLN research does not converge on a single training recipe. AgentVLN does not run reinforcement learning; it trains the VLM via instruction tuning to realize the POSMDP decision structure. Its AgentVLN-Instruct dataset, built in Habitat, aligns high-level instructions with low-level skill invocations and includes dynamic stage routing conditioned on target visibility, generalizable skill invocation patterns, localization reasoning with multi-round CoT, and active QA interactions with trajectory noise injection. The reported training setup uses Qwen2.5-VL-3B as the brain, freezes the visual encoder and multimodal projector, and uses AdamW with batch size 128, cosine learning rate with peak $F$2 and warmup 0.03, 640×480 observations, $F$3 FOV, and a historical context window of 8 frames (Xin et al., 18 Mar 2026).

The RL-oriented ActiveVLN framework is explicitly two-stage. Stage 1 uses a small fraction of expert trajectories for IL bootstrapping. Stage 2 performs multi-turn RL in the simulator with multiple sampled rollouts per instruction and Group Relative Policy Optimization. The IL loss over chunked actions is

$F$4

while the RL stage uses a PPO-style clipped GRPO objective with group-wise normalized advantages and clip parameter $F$5. The preferred reward is a soft success reward,

$F$6

which outperforms both the hard success reward and path-alignment variants in the reported ablations. The framework also introduces dynamic early-stopping,

$F$7

to prune long-tail or likely failed trajectories (Zhang et al., 16 Sep 2025).

AerialVLA is trained purely by behavior cloning. Its loss is reported as

$F$8

with no RL and no auxiliary grounding losses. The action space is

$F$9

and terminal frames are aligned with a zero-displacement label vector $T$0 and the standard text token LAND. Optimization uses AdamW with weight decay 0.03, gradient clip 1.0, cosine scheduler with peak learning rate $T$1 and 5% warmup, BF16 precision, 4× RTX 4090, global batch size 64, and 5 epochs over 420k training frames from 7,922 trajectories (Xu et al., 15 Mar 2026).

SkyVLN and VLN-Pilot occupy a prompt-based regime rather than a learned-policy regime. SkyVLN uses off-the-shelf GroundingDINO and GPT-4-family models with no task-specific fine-tuning, while VLN-Pilot also uses GPT-4.1 and Gemini-2.5-Flash out of the box and attributes behavior to prompt engineering plus hierarchical control logic rather than supervised or RL optimization (Li et al., 9 Jul 2025, Dominguez-Dager et al., 5 Feb 2026).

This heterogeneity is significant. It suggests that ActiveVLN is compatible with instruction tuning, behavior cloning, RL post-training, and zero-shot prompt-based control, provided that the resulting system preserves active perception–action coupling.

5. Benchmarks, reported performance, and comparative evidence

The empirical record for ActiveVLN is benchmark-specific. AgentVLN evaluates on R2R-CE and RxR-CE Val-Unseen. On R2R-CE Val-Unseen, AgentVLN-3B reports NE 3.88, OS 73.5, SR 67.2, and SPL 64.7, compared with InternVLA-N1-8.3B at SR 58.2 and SPL 54.0, and EfficientVLN-4B at NE 4.18, SR 64.2, and SPL 55.9. On RxR-CE Val-Unseen, AgentVLN-3B reports NE 3.92, SR 69.5, SPL 61.3, and nDTW 74.6, compared with EfficientVLN-4B at NE 3.88, SR 67.0, SPL 54.3, and nDTW 68.4. Its R2R ablation progresses from a baseline with NE 6.53, OS 48.50, SR 38.60, SPL 35.10 to the full system with NE 3.88, OS 73.50, SR 67.20, SPL 64.70, indicating gains from VLM-as-Brain, context-driven fine-grained control, and QD-PCoT (Xin et al., 18 Mar 2026).

The RL framework called ActiveVLN evaluates on R2R and RxR val-unseen in Habitat continuous environments. On R2R val-unseen, its 3B model improves from IL performance of NE 6.83, OS 42.7, SR 38.5, SPL 33.7 to IL + RL performance of NE 5.31, OS 58.3, SR 50.1, SPL 43.7, a +11.6 SR improvement. On the same benchmark, IL + DAgger yields SR 45.4 and SPL 41.5. On RxR val-unseen, it improves from NE 7.25, SR 41.0, SPL 34.1 without RL to NE 5.84, SR 50.7, SPL 41.2, nDTW 58.1 with RL (Zhang et al., 16 Sep 2025).

For UAV ActiveVLN, AerialVLA reports results on TravelUAV using NE, SR, OSR, and SPL across Seen, Unseen Object, and Unseen Map. On Seen Full, it achieves SR 47.96%, SPL 38.54%, NE 65.88, and OSR 57.69. On Unseen Object Full, it reports SR 56.60%, SPL 46.61, and OSR 64.86. On Unseen Map Full, it reports SR 37.58%, SPL 28.22, and OSR 52.92, approximately three times LongFly’s SR 11.27% and SPL 9.32. The paper also reports 17GB VRAM and 0.38s latency on RTX 4090, versus TravelUAV’s 20GB and 0.63s, and states that no statistical tests or confidence intervals are reported (Xu et al., 15 Mar 2026).

SkyVLN evaluates on the AVDN urban setup. On unseen testing, NavGPT reports SPL 18.9% and SR 16.6%, the NMPC variant reports SPL 22.4% and SR 26.8%, and the full system reports SPL 28.11% and SR 42.37%. Ablations without HSD or without TMA substantially reduce SPL and SR, indicating that spatial verbalization and memory-driven backtracking are functional rather than cosmetic additions (Li et al., 9 Jul 2025).

VLN-Pilot evaluates in a custom photorealistic indoor simulator using 50 total episodes over room- and object-level queries. GPT-4.1 achieves 43/50 success (86%), with 3 collisions total and 3 episodes exceeding the step limit; Gemini-2.5-Flash achieves 33/50 success (66%), with 5 collisions total and 12 episodes exceeding the step limit (Dominguez-Dager et al., 5 Feb 2026).

A concise cross-paper summary is useful.

System	Benchmark or setting	Reported headline result
AgentVLN	R2R-CE Val-Unseen	NE 3.88, SR 67.2, SPL 64.7
ActiveVLN	R2R val-unseen	IL + RL: NE 5.31, SR 50.1, SPL 43.7
AerialVLA	TravelUAV Unseen Map Full	SR 37.58%, SPL 28.22
SkyVLN	AVDN unseen testing	SPL 28.11%, SR 42.37%
VLN-Pilot	Indoor UAV simulation	GPT-4.1: 43/50 success (86%)

These results are not directly interchangeable because the embodiments, control spaces, and benchmarks differ. Even so, the aggregate evidence is consistent on one point: mechanisms that increase autonomy at inference time—self-correction, rollout exploration, fuzzy onboard prompting, memory-guided backtracking, or intrinsic landing—tend to improve true task completion rather than only intermediate perception metrics.

6. Limitations, failure modes, and research directions

The current ActiveVLN literature identifies several recurring limitations. AgentVLN depends on accurate camera intrinsics and extrinsics and reliable depth $T$2; severe sensor noise or calibration errors can degrade 2D–3D lifting and prompt quality. Its SLAM layer can also incur transient mapping errors in highly dynamic scenes, and very long contexts may dilute attention, motivating adaptive memory or hierarchical temporal abstraction. The paper further notes that formal RL within the POSMDP remains conceptual and suggests future reinforcement fine-tuning with option-level objectives (Xin et al., 18 Mar 2026).

The RL-based ActiveVLN framework exposes a different set of constraints. Reward sparsity remains significant, dynamic sampling is used to mitigate this issue, and the early-stopping rule depends on oracle trajectory length $T$3, which introduces a weak dependency on expert metadata even though policy optimization itself does not use expert corrections. The paper also notes compute cost of roughly 40 hours total on 4× L40S GPUs and highlights safety concerns for real-world deployment under ambiguous instructions or unseen obstacles (Zhang et al., 16 Sep 2025).

AerialVLA identifies a memory-versus-reactivity trade-off: without long-horizon memory, the agent may struggle with global backtracking in repetitive urban layouts, while in extreme out-of-distribution settings with severe occlusion it may hover safely rather than explore aggressively. Proposed directions include lightweight memory and reinforcement learning fine-tuning for more active exploration (Xu et al., 15 Mar 2026).

VLN-Pilot shows that VLLM-based control can be highly flexible while remaining sensitive to prompt design and model choice. Reported failure modes include doorway alignment oscillations, collisions at door frames due to lack of volumetric awareness of drone dimensions, misidentification of room or object under challenging viewpoints, and the practical latency of cloud-based inference. Potential improvements include probabilistic mapping, occupancy grids, explicit uncertainty modeling, learned low-level avoidance, local MPC, and exploitation of altitude adjustments for safer crossings (Dominguez-Dager et al., 5 Feb 2026).

SkyVLN highlights sim-to-real and model-fidelity issues. The AirSim rotor aerodynamic model is described as coarse, the NMPC state excludes yaw and angular rates, and the framework assumes reliable front-view RGB, depth, semantic sensing, sectorization via HSD, and predictable obstacle trajectories. The paper treats these as limitations for deployment in GNSS-denied and highly dynamic real environments (Li et al., 9 Jul 2025).

A broader implication emerges from these limitations. ActiveVLN systems gain robustness by becoming more agentic, but greater agency also raises the burden on memory design, sensing reliability, safety constraints, and real-time control. The field is therefore moving toward tighter coupling between language-grounded reasoning and physically grounded uncertainty handling, rather than toward language-only navigation policies or purely geometric controllers.