SkyVLN: Urban UAV Navigation Framework

• SkyVLN is a framework that combines natural language navigation with nonlinear model predictive control for UAVs in cluttered urban 3D spaces.
• It leverages multimodal inputs—RGB, depth, and semantic segmentation—using models like GroundingDINO and LLMs for landmark detection and route planning.
• By decoupling semantic decision-making from dynamic execution, SkyVLN enhances trajectory tracking, collision avoidance, and interpretability over end-to-end policies.

Searching arXiv for papers on SkyVLN and closely related aerial VLN systems. SkyVLN is a framework for unmanned aerial vehicle navigation in urban environments that integrates vision-and-language navigation with nonlinear model predictive control. In its specific formulation, a multimodal navigation agent driven by a LLM interprets natural-language instructions and front-view observations, generates high-level navigation commands, and hands them to an NMPC back end that enforces continuous trajectory tracking and collision avoidance in dynamic 3D scenes (Li et al., 9 Jul 2025). Within the broader aerial VLN literature, SkyVLN belongs to the line of work that extends instruction-following from ground robots to first-person UAVs operating in continuous 3D space, a problem formalized earlier by AerialVLN and later re-examined by both larger-scale benchmarking platforms and end-to-end Vision-Language-Action systems (Liu et al., 2023, Gao et al., 25 Feb 2025, Xu et al., 15 Mar 2026).

1. Problem formulation and aerial VLN context

SkyVLN addresses a continuous 3D navigation problem in which a UAV must follow human natural-language instructions while flying through complex urban geometry, including tall buildings, narrow corridors, and dynamic obstacles. The UAV state is specified as

$P = [x,y,z,u,v,w,\phi,\theta,\psi,p,q,r]^\top,$

with position, linear velocity, Euler angles, and angular rates, and the observation at time $t$ is

$V_t = \{v_t^R, v_t^D, v_t^S\},$

where $v_t^R$ is RGB, $v_t^D$ is depth, and $v_t^S$ is semantic segmentation (Li et al., 9 Jul 2025). The framework is motivated by the claim that direct low-level control is insufficient when the environment is GNSS-challenged and instructions are vague or underspecified, while purely end-to-end imitation or reinforcement learning policies often overfit specific maps and do not provide strong physical control guarantees.

This formulation inherits the defining properties of aerial VLN established by AerialVLN: continuous 3D state space, vertical motion, multirotor-specific maneuvering, and long-horizon language grounding in city-scale outdoor scenes. AerialVLN formalized UAV-based instruction following with first-person RGB and depth observations, a 20 m success criterion, and trajectories that are substantially longer and more spatially complex than standard indoor VLN, thereby establishing the technical background against which SkyVLN was proposed (Liu et al., 2023). SkyVLN differs from early aerial VLN baselines by explicitly coupling semantic reasoning to predictive control rather than treating the problem as a purely recurrent action-classification task.

2. System architecture and processing pipeline

SkyVLN is organized into a multimodal navigation front end and an NMPC control back end. At each step, the UAV acquires front-view RGB, depth, and semantic images; an object grounding model, specifically GroundingDINO, detects candidate landmarks; an LLM processes the instruction to extract landmark-centric subgoals,

$L = \operatorname{LLM}(T, \text{prompt}),$

and a prompt-construction layer converts current perception, subgoals, and accumulated history into a structured text input for the LLM (Li et al., 9 Jul 2025). The LLM then outputs the next high-level action, such as “move forward 10 m,” “turn right and ascend,” or “backtrack to the last intersection.” These outputs are mapped to reference trajectories or reference states $x_{\text{ref}}$ and reference controls $u_{\text{ref}}$, which the NMPC uses to compute feasible continuous control.

A defining architectural feature is the separation of semantic decision-making and dynamic execution. The LLM determines what the UAV should do in terms of landmark-centric or route-centric macro actions, whereas NMPC determines how to do it under dynamic and obstacle constraints. This decomposition is central to SkyVLN’s identity in later comparative work, where it is described as exemplifying the hierarchical planning plus NMPC control paradigm for urban UAV navigation (Xu et al., 15 Mar 2026).

The system is not trained end-to-end as a neural control policy. Instead, the VLN component uses off-the-shelf VLMs and LLMs with prompt engineering and external modules, and the optimization problem appears in the controller rather than in a learned action loss (Li et al., 9 Jul 2025). This distinguishes SkyVLN from later Vision-Language-Action formulations that replace the planner-controller split with a single transformer policy.

3. Spatial grounding, prompt optimization, and memory

SkyVLN’s navigation agent is built around three coupled mechanisms: landmark detection, a fine-grained spatial verbalizer, and an explicit path memory. GroundingDINO is used to detect landmarks mentioned in the instruction, and for each landmark instance visible in multiple views, the view with the highest detection score is designated as the canonical observation viewpoint (Li et al., 9 Jul 2025). The system does not introduce a new cross-modal transformer for navigation; instead, it uses off-the-shelf models and converts geometric and perceptual structure into textual prompt content.

The fine-grained spatial verbalizer is the High-resolution Spatial Descriptor. HSD partitions the front image into a $3 \times 3$ grid, with sector IDs such as “#0” for upper-left and “#8” for lower-right, assigns each detected landmark to a sector using the bounding-box center, and injects that sector label into the textual description of the landmark (Li et al., 9 Jul 2025). This enables the LLM to reason over distinctions such as “KFC on the left” versus “McDonald’s on the right” using text rather than explicit geometric tensors. The paper also describes a feature-fusion and candidate-ranking procedure in which visual features $t$0 and landmark text features $t$1 are used to retrieve top-$t$2 candidates, after which the LLM compares the query description with each candidate and ranks them.

The memory mechanism is the TrackBack Memory Array, a graph-structured store whose nodes are previously encountered landmarks and whose edges are navigation instructions or paths between landmarks. As new landmarks are recognized, TBMA updates this graph and can compute a shortest path over visited nodes to support backtracking or route recovery (Li et al., 9 Jul 2025). That path summary is then inserted into the prompt. This design targets a specific failure mode of language-guided flight: ambiguous instructions with insufficient landmark anchoring, such as “turn left, then move right, then go straight,” where a memoryless agent is prone to loops. In the full system, HSD and TBMA are combined through Wayfinding Prompt Optimization, which packages current landmarks, spatial sector labels, temporal context, and memory-graph summaries into the LLM input.

SkyVLN also reports an LLM ablation on GPT-4V, GPT-4o, and GPT-4 Turbo. Under unseen testing, GPT-4V yields SPL $t$3, SR $t$4, and NE $t$5 m; GPT-4o yields SPL $t$6, SR $t$7, and NE $t$8 m; GPT-4 Turbo yields SPL $t$9, SR $V_t = \{v_t^R, v_t^D, v_t^S\},$0, and NE $V_t = \{v_t^R, v_t^D, v_t^S\},$1 m (Li et al., 9 Jul 2025). The paper interprets this as evidence that different foundation models induce different navigation styles, with GPT-4V favoring success and GPT-4o favoring path efficiency.

4. Dynamic model, NMPC formulation, and obstacle avoidance

SkyVLN’s controller uses a simplified but physically grounded quadrotor model. The NMPC state is

$V_t = \{v_t^R, v_t^D, v_t^S\},$2

where $V_t = \{v_t^R, v_t^D, v_t^S\},$3 is position, $V_t = \{v_t^R, v_t^D, v_t^S\},$4 is linear velocity, and $V_t = \{v_t^R, v_t^D, v_t^S\},$5 are roll and pitch; the control input is

$V_t = \{v_t^R, v_t^D, v_t^S\},$6

with total thrust and attitude references for the inner loop (Li et al., 9 Jul 2025). The dynamics are written as

$V_t = \{v_t^R, v_t^D, v_t^S\},$7

$V_t = \{v_t^R, v_t^D, v_t^S\},$8

with diagonal damping matrix $V_t = \{v_t^R, v_t^D, v_t^S\},$9, and first-order roll and pitch tracking: $v_t^R$0

Over a prediction horizon $v_t^R$1, SkyVLN minimizes a standard tracking-and-smoothness objective,

$v_t^R$2

subject to dynamics, collision-avoidance constraints, and actuator or attitude limits (Li et al., 9 Jul 2025). Obstacles are modeled as spheres with radius $v_t^R$3 and safety margin $v_t^R$4, and avoidance is expressed through

$v_t^R$5

Input-rate constraints are also imposed on $v_t^R$6 and $v_t^R$7 to prevent aggressive changes.

The optimization is solved with PANOC through the OpEn library, using a quadratic penalty method for equality constraints (Li et al., 9 Jul 2025). In effect, SkyVLN delegates semantic ambiguity to the LLM and dynamic feasibility to NMPC. This modular division is also the basis on which later work contrasts SkyVLN with end-to-end VLA systems that attempt to remove the planner-controller interface entirely (Xu et al., 15 Mar 2026).

5. Evaluation, ablations, and reported performance

SkyVLN is evaluated in a UE4 plus AirSim urban environment with dynamic vehicles and pedestrians, using the Aerial Vision-and-Dialog Navigation dataset. AVDN contains more than $v_t^R$8k UAV navigation trajectories in a continuous photorealistic environment, with average trajectory length about $v_t^R$9 m and human-human dialogs between a commander and a follower (Li et al., 9 Jul 2025). The reported metrics are Success Rate, Success weighted by Path Length, and Navigation Error.

On the unseen test set, the paper reports the following SPL and SR values: NavGPT, $v_t^D$0 SPL and $v_t^D$1 SR; a variant with NMPC, $v_t^D$2 SPL and $v_t^D$3 SR; SkyVLN without HSD, $v_t^D$4 SPL and $v_t^D$5 SR; SkyVLN without TMA, $v_t^D$6 SPL and $v_t^D$7 SR; and the full SkyVLN system, $v_t^D$8 SPL and $v_t^D$9 SR (Li et al., 9 Jul 2025). The ablations attribute substantial gains to both HSD and TBMA: HSD improves fine-grained landmark grounding, while TBMA improves success by enabling memory-based recovery and backtracking.

The controller ablations identify NMPC as an important contributor. Compared with SimpleFlight PID and a linear MPC variant, NMPC yields better trajectory tracking and obstacle avoidance, especially in unseen environments, and figure-level analyses show smaller position and attitude errors under dynamic obstacle scenarios (Li et al., 9 Jul 2025). The system therefore claims gains from both halves of the architecture: semantic prompt engineering improves route selection, while predictive control improves execution fidelity.

6. Position within the broader SkyVLN literature

SkyVLN sits at the intersection of several strands of aerial navigation research. AerialVLN established the underlying task family by defining first-person UAV instruction following in continuous 3D city environments, with $v_t^S$0 paths, $v_t^S$1 instructions, and a large human-agent gap: on full AerialVLN test_unseen, CMA achieved approximately $v_t^S$2 SR while human performance was approximately $v_t^S$3 SR (Liu et al., 2023). SkyVLN can be understood as one response to that difficulty: it replaces flat recurrent action prediction with a hierarchical LLM-plus-NMPC stack.

OpenFly generalizes the benchmarking side of the same research program. It provides a platform spanning Unreal Engine, GTA V, Google Earth, and 3D Gaussian Splatting, with $v_t^S$4k trajectories across $v_t^S$5 scenes, and introduces OpenFly-Agent, a keyframe-aware model with explicit visual-history compression (Gao et al., 25 Feb 2025). On OpenFly Test-Seen, OpenFly-Agent reports NE $v_t^S$6 m, SR $v_t^S$7, OSR $v_t^S$8, and SPL $v_t^S$9, outperforming prior baselines; on Test-Unseen it reports NE $L = \operatorname{LLM}(T, \text{prompt}),$0 m, SR $L = \operatorname{LLM}(T, \text{prompt}),$1, OSR $L = \operatorname{LLM}(T, \text{prompt}),$2, and SPL $L = \operatorname{LLM}(T, \text{prompt}),$3 (Gao et al., 25 Feb 2025). This suggests that the chief bottleneck is not only controller design but also data scale, environmental diversity, and long-horizon visual memory.

AerialVLA later takes the opposite design direction. It explicitly presents SkyVLN as the hierarchical planning plus NMPC paradigm and argues that such modular systems suffer from semantic gaps between planner, controller, and landing logic. AerialVLA removes dense oracle guidance, external object detectors, and separate planner-controller modules, and instead predicts discretized $L = \operatorname{LLM}(T, \text{prompt}),$4, $L = \operatorname{LLM}(T, \text{prompt}),$5, and $L = \operatorname{LLM}(T, \text{prompt}),$6 directly from visual and linguistic input, with intrinsic landing as part of the same policy (Xu et al., 15 Mar 2026). On TravelUAV Test Seen, AerialVLA reports SR $L = \operatorname{LLM}(T, \text{prompt}),$7 and SPL $L = \operatorname{LLM}(T, \text{prompt}),$8, compared with LongFly’s SR $L = \operatorname{LLM}(T, \text{prompt}),$9 and SPL $x_{\text{ref}}$0; on Unseen Map it reports SR $x_{\text{ref}}$1 and SPL $x_{\text{ref}}$2, compared with LongFly’s SR $x_{\text{ref}}$3 and SPL $x_{\text{ref}}$4 (Xu et al., 15 Mar 2026). The contrast is substantive: SkyVLN represents modular, interpretable, controller-centric aerial VLN, whereas AerialVLA argues for minimalist end-to-end control.

A further adjacent line concerns whether the perception backbone itself is adequate for UAV reasoning. SpatialSky-Bench evaluates VLM spatial intelligence across $x_{\text{ref}}$5 UAV tasks and reports that mainstream models achieve overall averages mostly around $x_{\text{ref}}$6–$x_{\text{ref}}$7, whereas Sky-VLM reaches $x_{\text{ref}}$8 (Zhang et al., 17 Nov 2025). A plausible implication is that some limitations of SkyVLN-style systems may originate in the underlying VLM’s poor aerial spatial competence rather than only in planning or control. Finally, a distinct use of sky-based navigation appears in Sky-GVIO, where sky-view fisheye segmentation is used for LOS/NLOS-aware GNSS/INS/Vision fusion rather than instruction following, demonstrating that “sky” navigation research also includes systems in which sky imagery acts as a visibility sensor rather than a language-grounded semantic input (Wang et al., 2024).

7. Limitations and future directions

SkyVLN’s stated limitations are structural. Evaluation is simulation-only in AirSim and UE4; the system depends on external proprietary LLMs; it does not perform end-to-end task-specific learning; the NMPC obstacle model uses spheres with known or predicted trajectories; and the language regime is natural but still bounded by AVDN-style instructions rather than adversarially noisy or highly metaphorical commands (Li et al., 9 Jul 2025). These constraints delimit the framework’s present scope: it is a modular urban UAV navigation stack, not a general-purpose aerial autonomy system.

The future directions listed for SkyVLN are balancing safety and mission efficiency more systematically, extending to multi-UAV coordination, improving robustness to severe environmental conditions, moving toward real-world deployment and sim-to-real transfer, and integrating more advanced multimodal LLMs with tighter coupling to perception and control (Li et al., 9 Jul 2025). Related work sharpens those directions. OpenFly indicates that scaling data, rendering diversity, and real-to-sim 3D reconstruction are likely prerequisites for broader generalization (Gao et al., 25 Feb 2025). AerialVLA suggests that some planner-controller interfaces may be replaceable by unified VLA policies when continuous control can be learned directly (Xu et al., 15 Mar 2026). SpatialSky-Bench suggests that UAV-specific spatial reasoning benchmarks are needed to verify whether a vision-language backbone is suitable for aerial deployment at all (Zhang et al., 17 Nov 2025).

Taken together, SkyVLN occupies a central historical and methodological position in aerial vision-and-language navigation. It is neither the earliest benchmark nor the most minimalist control architecture, but it crystallizes a specific design thesis: LLM-based semantic planning, explicit spatial verbalization, graph-based history memory, and NMPC-based execution can be composed into a UAV navigation system that is more interpretable and physically grounded than flat VLN policies, while still addressing the long-horizon, ambiguous, and safety-critical characteristics of urban aerial navigation (Li et al., 9 Jul 2025).