Aerial Vision-Language Navigation (AVLN)

• Aerial Vision-Language Navigation is a language-conditioned decision-making task where UAVs use natural instructions and visual cues to navigate in partially observable 3D spaces.
• Research in AVLN emphasizes the trade-offs between discrete and continuous control methods while leveraging spatial representations like semantic-topo-metric maps for improved performance.
• Benchmark studies reveal challenges such as long-horizon navigation, multimodal reasoning, onboard constraints, and collaborative UAV strategies, driving ongoing innovations in the field.

Aerial Vision-Language Navigation (AVLN), also called Aerial VLN, is a language-conditioned sequential decision-making problem in partially observable 3D space in which a UAV interprets natural-language instructions and uses visual perception to navigate to a goal (Xia et al., 9 Apr 2026). In contrast to ground VLN, AVLN is coupled to motion in $SE(3)=\mathbb{R}^3\times SO(3)$, severe viewpoint variation, city-scale or otherwise extended environments, long trajectories, and tight onboard compute constraints; current formulations span both single-instruction navigation and dialog-based interaction, and both discrete primitive control and continuous waypoint or velocity prediction (Xia et al., 9 Apr 2026).

1. Task structure and formal problem variants

A common formalization models AVLN as a language-conditioned POMDP with state, observation, and action evolving over time. The survey literature defines the state as

$$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$

the observation as

$$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$

and the policy as

$a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$

with success measured by

$\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$

The same survey distinguishes Aerial Vision-and-Instruction Navigation (AVIN), where one complete instruction is given at episode start, from Aerial Vision-and-Dialog Navigation (AVDN), where a dialog history is part of the observation (Xia et al., 9 Apr 2026).

Benchmark instantiations differ materially. The original AerialVLN benchmark initializes the UAV at pose $P=[x,y,z,p,r,y']$, provides front-view RGB-D observations $V_t=\{v_t^R,v_t^D\}$, and uses an eight-action set consisting of Move Forward, Turn Left, Turn Right, Ascend, Descend, Move Left, Move Right, and Stop; success requires stopping within 20 meters of the goal (Liu et al., 2023). CityNav instead formulates the agent as a UAV navigating a 2D urban environment that receives an egocentric top-down RGB-D observation, its current pose, a goal-oriented instruction, and polygonal landmark priors, and must output continuous control actions; an episode is successful if the agent emits $[stop]$ within 20 steps and ends within 20 m of the target (Ding et al., 16 Dec 2025). More recent work pushes further toward fully aerial motion: ImagineUAV treats AVLN as grounding free-form instructions into full 6-DoF flight under partial observability, with relative 6-DoF poses or motions inferred over a planning horizon and then refined into dynamically feasible trajectories (Liu et al., 31 May 2026).

A recurrent field-level distinction is between discrete and continuous control. The survey notes that many AVLN systems still use a finite primitive set

$$\mathcal{A}_{\text{disc}} = \{forward, backward, left, right, up, down, rotation, stop\},$$

whereas a smaller but growing set of methods predicts continuous controls such as

$$a_t = (\mathbf{v}_t, \boldsymbol{\omega}_t) \in \mathbb{R}^3 \times \mathbb{R}^3$$

or target waypoints in $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$0 (Xia et al., 9 Apr 2026). This divide is central to AVLN because many benchmarked methods inherit discrete action abstractions from earlier VLN, while deployment-oriented systems increasingly argue for continuous control, waypoint interfaces, or kinodynamic planning.

2. Benchmarks, simulators, and evaluation infrastructure

AVLN research is organized around several benchmark families that differ in sensing assumptions, map scale, and interaction style.

Benchmark	Scale and data	Distinguishing property
AerialVLN / AerialVLN-S	25 city-level scenarios, 8,446 routes, 25,338 instructions; AerialVLN-S has 3,916 routes and 11,748 instructions	UE4 + AirSim, continuous navigation, front-view RGB-D, hidden-goal test server (Liu et al., 2023)
CityNav	32,326 natural-language descriptions for about 5.8K objects	Goal-oriented instructions with static geographic landmark priors; built from SensatUrban and CityRefer (Ding et al., 16 Dec 2025)
TravelUAV / UAV-Need-Help	12,149 trajectories total; 9,152 train, 1,410 Test-Seen, 958 Test-Unseen-Map, 629 Test-Unseen-Object	Long-horizon outdoor UAV navigation with assistance levels and hard trajectories at or above 250 m (Lin et al., 9 Nov 2025)
HaL-13k	13,838 collaborative trajectory pairs across 14 scenarios	Dual-altitude collaborative high-low UAV demonstrations with unseen-map and unseen-object validation splits (Wu et al., 21 Aug 2025)

The original AerialVLN paper established the difficulty of the problem by showing an enormous human-model gap: on test unseen, the best standard baseline reached only $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$1 SR, whereas humans reached $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$2 SR (Liu et al., 2023). CityNav later shifted attention toward landmark-centric, goal-oriented navigation in large urban regions and further exposed the importance of annotation quality, because its landmark priors are consumed explicitly by policy architectures (Ding et al., 16 Dec 2025). TravelUAV emphasized long-range flights, scarce data, and seen versus unseen-object versus unseen-map generalization (Lin et al., 9 Nov 2025). DuAl-VLN and HaL-13k introduced paired high-altitude and low-altitude UAV trajectories, making collaboration itself the benchmarked capability (Wu et al., 21 Aug 2025).

Metric practice is only partially standardized. AerialVLN popularized NE, SR, OSR, and SDTW (Liu et al., 2023). CityNav-based work commonly reports NE, SR, OSR, and SPL (Ding et al., 16 Dec 2025). The survey formalizes

$$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$3

and

$$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$4

while also noting that action spaces, success thresholds, and evaluation protocols still vary substantially across benchmarks (Xia et al., 9 Apr 2026). A plausible implication is that cross-paper comparison remains informative but not fully commensurate.

3. Spatial representation, memory, and the geometry of instruction grounding

A persistent AVLN design question is how to represent large aerial space so that language can be grounded without collapsing into purely local reactive control. One influential answer is Semantic-Topo-Metric Representation (STMR), which extracts instruction-relevant landmarks, projects them into a top-down map using RGB and depth, and serializes a local $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$5 matrix with 5 m cell spacing for LLM reasoning; on AerialVLN-S, this representation yielded absolute OSR gains of $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$6 on validation seen and $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$7 on validation unseen over LAG (Gao et al., 2024). The key claim was that aerial reasoning needs explicit semantics, topology, and approximate metric layout, rather than raw RGB prompting alone.

A second line of work argues that AVLN should be formulated as structured candidate selection in 3D. The grid-based view-selection framework captures a six-view skybox, maps each view to a candidate position, predicts per-view vertical adjustment, and fuses navigation history into a grid-based bird’s-eye-view map via

$$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$8

On AerialVLN-S val-seen it reported $$s_t = (\mathbf{x}_t, \dot{\mathbf{x}}_t, \boldsymbol{\omega}_t, \mathbf{q}_t, \mathcal{E}),$$9 SR, $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$0 OSR, and $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$1 SDTW, substantially above earlier baselines (Zhao et al., 14 Mar 2025). This work is important because it treated altitude change as coupled to horizontal motion rather than as an isolated action token.

CityNav methods pushed spatial memory in a different direction. HETT introduces a Historical Grid Map over an $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$2 grid, storing visual descriptors in cells according to UAV position and reweighting them by instruction relevance before both coarse target prediction and fine local action refinement. Its two-stage policy first predicts a normalized target coordinate

$$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$3

then refines action with instruction-conditioned local view analysis. On refined CityNav annotations, HETT$$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$4 reached $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$5 SR on test unseen, with test-unseen NE dropping to $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$6 and SPL rising to $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$7 (Ding et al., 16 Dec 2025). The same paper showed that manually refined landmark annotations improved not only HETT but also AerialVLN and MGP, indicating that benchmark-level landmark noise is a major variable for AVLN.

Another representation shift is direction-aware rather than landmark-only reasoning. LookasideVLN argues that prior systems underuse directional cues and introduces an Egocentric Lookaside Graph (ELG) plus a Spatial Landmark Knowledge Base (SLKB). ELG edges encode egocentric horizontal angle, elevation change, and horizontal distance between candidate landmarks, then verbalize those relations into path descriptions for an MLLM. On AerialVLN-S validation unseen, LookasideVLN reported $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$8 SR and $$o_t = (\mathcal{V}_t, \hat{\mathbf{p}}_t, \mathcal{L}),$$9 OSR, outperforming CityNavAgent in SR and OSR while using a shallow lookahead horizon (Ning et al., 19 Apr 2026). ViSA-Enhanced Aerial VLN makes a related critique from another angle, arguing that detection-and-planning pipelines built from discrete textual scene graphs suffer from inadequate spatial reasoning and linguistic ambiguity; its abstract reports a $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$0 improvement in success rate over the fully trained SOTA method on CityNav (Tong et al., 9 Mar 2026).

Across these systems, a plausible synthesis is that AVLN has progressively moved from sequence modeling over egocentric frames toward explicit spatial abstractions: serialized maps, BEV memories, directional graphs, and structured visual prompting.

4. Large-model reasoning, zero-shot navigation, and RGB-only AVLN

The survey organizes recent AVLN systems into end-to-end LLM/VLM methods, hierarchical methods, multi-agent methods, and dialog-based navigation, reflecting a broad shift from task-specific encoders toward pretrained multimodal backbones (Xia et al., 9 Apr 2026). One consequence is the rise of zero-shot or training-free AVLN under strong prompting and structured perception.

STMR is an early example of this shift: an end-to-end zero-shot framework where GPT-4V or GPT-4o receives a textualized top-down map prompt and emits the next UAV action without task-specific training (Gao et al., 2024). A more radical learning-free formulation is See, Point, Fly (SPF), which treats AVLN as 2D spatial grounding rather than text action generation: a frozen VLM predicts a waypoint in image space, an adaptive travel-distance heuristic maps that to 3D displacement, and low-level control executes the motion in closed loop. SPF reported $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$1 success in simulation and $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$2 in real-world evaluations, far above TypeFly and PIVOT, while remaining training-free and model-agnostic across multiple VLM backbones (Hu et al., 26 Sep 2025).

Other systems emphasize deployment constraints rather than training-free operation. OnFly is a fully onboard, real-time zero-shot framework that decouples high-frequency target generation from low-frequency progress monitoring via a shared-perception dual-agent architecture, adds hybrid keyframe-recent-frame memory, semantic-geometric target verification, and a receding-horizon planner, and reported simulation SR $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$3 versus $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$4 for the strongest baseline, with collision rate dropping to $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$5 (Zheng et al., 11 Mar 2026). VLFly instead decomposes RGB-only AVLN into instruction reformulation by LLaMA, CLIP-based goal-image retrieval, and a ViNT-derived waypoint planner producing continuous velocity commands; in real-world experiments it achieved $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$6 success for direct instructions and $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$7 for indirect instructions without localization or active ranging sensors (Zhang et al., 12 Jun 2025).

A separate line attempts to recover strong AVLN under the strictest sensing regime. The unified monocular RGB-only framework based on NVILA-lite casts aerial navigation, spatial perception, and trajectory reasoning as prompt-guided next-token prediction under a shared decoder, and on AerialVLN-S reported $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$8 SR on validation seen and $a_t = \pi(\mathcal{L}, o_{1:t}, a_{1:t-1}),$9 SR on validation unseen while substantially outperforming other RGB-only baselines (Xu et al., 9 Dec 2025). This suggests that stronger multimodal priors and auxiliary reasoning objectives can partially replace privileged panoramic, depth, or odometry inputs, although the same paper explicitly notes that absolute SR remains modest.

5. Continuous control, world models, and collaborative aerial planning

A distinct trajectory in AVLN research rejects direct action token prediction and instead models future world evolution, kinodynamics, or cooperation explicitly. ImagineUAV formulates AVLN as an imagination-driven pipeline: an instruction-conditioned latent video diffusion model generates short-horizon future egocentric observations, a geometry-aware action extractor recovers relative 6-DoF poses, and a kinodynamic planner converts them into collision-free trajectories. With step-distilled inference and only 1.3B parameters, it reports a $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$0 success rate on UAV-Flow together with real-world flight demonstrations (Liu et al., 31 May 2026).

WorldVLN makes the world-model argument more explicit by introducing the first autoregressive world action model for aerial VLN. It predicts short-horizon future latent transitions

$\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$1

decodes them into waypoint segments

$\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$2

then replaces imagined futures with real executed observations in the autoregressive context. Its second training stage, Action-aware GRPO, optimizes segment decisions through rollout consequences rather than one-step imitation alone. On UAV-Flow-Sim it reports $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$3 SR for fixed-template instructions and $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$4 for open-vocabulary instructions; on IndoorUAV-VLA it reports $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$5 average SR and zero-shot real-drone transfer (Zhao et al., 15 May 2026).

At the opposite architectural extreme, AerialVLA advocates minimalist end-to-end control. Built on OpenVLA-7B, it uses a dual-view perception strategy, fuzzy directional prompting derived from onboard sensors, and a unified 3-DoF control space with an intrinsic landing signal. On TravelUAV it reaches $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$6 SR on seen environments and $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$7 SR on unseen maps, nearly three times the success rate of leading baselines on the unseen-map split (Xu et al., 15 Mar 2026). The paper’s core claim is that dense oracle guidance and external target detectors are “double crutches,” and that forcing a policy to couple perception, language, control, and termination improves generalization.

Multi-UAV collaboration pushes AVLN beyond single-agent embodiment. AeroDuo introduces Dual-Altitude UAV Collaborative VLN (DuAl-VLN) and the HaL-13k dataset, pairing a high-altitude UAV for broad environmental reasoning with a low-altitude UAV for precise navigation. The high UAV runs Pilot-LLM over an orthographic map and outputs a target probability map, while the low UAV runs Multi-Stage Pathfinder for waypoint selection, collision-free navigation, and target localization. On unseen maps AeroDuo reports $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$8 SR and $\text{Success} = \mathds{1}\left[\|\mathbf{x}_T - \mathbf{x}_g\|_2 \le d_{\text{th}}\right].$9 SST; on unseen objects it reports $P=[x,y,z,p,r,y']$0 SR and $P=[x,y,z,p,r,y']$1 SST, substantially above retrained single-UAV baselines (Wu et al., 21 Aug 2025). This suggests that altitude itself can be used as a role-specialization axis in AVLN, not merely as another control dimension.

6. Conceptual issues, common misconceptions, and open problems

A recurrent misconception is that AVLN is merely “VLN with a drone camera.” Multiple papers explicitly reject this. AerialVLN frames UAV navigation as fundamentally harder because of 3D motion, flying height, city-scale environments, and more complex spatial reasoning (Liu et al., 2023). HETT makes the same point from the CityNav perspective, arguing that AVLN is not the usual indoor waypoint-following problem but a large-scale, landmark-centric target-localization problem that requires balancing global reasoning and local grounding (Ding et al., 16 Dec 2025). The survey generalizes this critique by emphasizing continuous 6-DoF motion, viewpoint robustness, scalable spatial representation, and onboard deployment as field-defining issues rather than implementation details (Xia et al., 9 Apr 2026).

A second misconception is that stronger pretrained VLMs alone solve aerial navigation. Several papers argue otherwise, but in different ways. ImagineUAV identifies geometric inconsistency, dynamics mismatch, and partial observability as failure sources for direct VLA regression (Liu et al., 31 May 2026). OnFly shows that planner-based collision avoidance alone does not solve reliable stopping or progress monitoring (Zheng et al., 11 Mar 2026). VLFly demonstrates that open-vocabulary grounding remains tied to a predefined candidate image pool, and currently supports effectively planar rather than full 3D maneuvering (Zhang et al., 12 Jun 2025). AerialVLA, despite strong unseen-map results, explicitly notes the cost of reactive policies without explicit memory or backtracking (Xu et al., 15 Mar 2026).

Benchmark construction and data quality are themselves open research problems. CityNav annotation refinement improved multiple baselines, showing that landmark extraction noise can materially distort both training and evaluation (Ding et al., 16 Dec 2025). The survey argues that AVLN evaluation is fragmented across action spaces, success thresholds, splits, and metrics, and identifies seven concrete open problems: long-horizon instruction grounding, viewpoint robustness, scalable spatial representation, continuous 6-DoF action execution, onboard deployment, benchmark standardization, and multi-UAV swarm navigation (Xia et al., 9 Apr 2026). A plausible implication is that progress in AVLN will continue to depend as much on representation, interfaces, and benchmark fidelity as on raw model scale.

Taken together, the literature suggests a field moving away from monolithic recurrent policies toward more structured aerial reasoning: explicit spatial abstractions, coarse-to-fine or dual-timescale decision stacks, continuous-control interfaces, world models, and, increasingly, deployment-aware systems. What remains unsettled is not whether language can guide UAVs in simulation, but which combination of spatial memory, multimodal reasoning, control abstraction, and evaluation protocol best captures the genuinely aerial character of navigation.