- The paper introduces a causally structured model that predicts current maps, future trajectories, and future maps, linking them to waypoint generation through cross-space planning consistency.
- The method achieves 56.99 m navigation error, 35.83% success rate, and 30.22% SPL on OpenUAV Test Seen (Full), outperforming TravelUAV while adding only 12.31 ms of inference latency.
- The results show that map-space imagination improves over egocentric future prediction, but pseudo-label errors, curved trajectories, and performance drops on unseen environments remain important challenges.
Motivation and problem setting
UAV Vision-Language Navigation (UAV-VLN) requires an aerial agent to ground natural-language instructions in sparse multi-view observations and execute continuous 3D motion over long horizons. The authors identify three structural deficiencies in existing LLM-based approaches such as TravelUAV (Wang et al., 2024) and OpenFly (Gao et al., 25 Feb 2025). First, these systems predict waypoints directly from discrete multi-view observations without forming a structured spatial state, yielding weak current-scene grounding; map-based alternatives (e.g., VLMaps, MapNav) incur online construction latency. Second, prior visual-imagination methods forecast egocentric RGB, depth, or latent future states (Huang et al., 9 May 2025, Zhang et al., 6 Jul 2025, Liu et al., 1 Dec 2025), which are confined to local first-person views and do not model map-level environmental evolution. Third, intermediate spatial predictions are typically optimized as auxiliary objectives with only a weak connection to action generation. AirForesight (2608.12835) addresses all three by learning a causally ordered chain of latent spatial states—current map, future trajectory in map space, future map—and tying them to waypoint prediction through a cross-space planning consistency loss (CPCL). Crucially, spatial maps serve only as training supervision; no dense map is constructed at inference time.
Architecture
The framework builds on a Vicuna-7B backbone (Tian et al., 2023) with frozen weights except LoRA parameters, the visual projector (EVA-CLIP + Q-Former with LLaMA-VID-style 17-token compression per image), a lightweight MAE decoder (Zhao, 2022), task heads, and the waypoint MLP. Three groups of learnable query tokens—current-map (9 tokens), future-map (9 tokens), and a single waypoint token—are appended to the multimodal sequence under structured causal attention: current-map tokens attend only to language and visual inputs; future-map tokens additionally attend to current-map tokens; the waypoint token attends to everything. Tokens within the same group are mutually masked, forcing each token to extract complementary information independently. The predicted waypoint is refined into a local path by TravelUAV's path decoder.
Current-map modeling. The current-map representation is decoded into a 224×224 semantic map over C superclasses and supervised with weighted cross-entropy. Supervision maps are produced by an offline pipeline: an LLM (Gemini 3 Pro) extracts instruction-relevant categories, GroundingDINO (2303.19488) grounds them, MobileSAM segments them, and depth-based unprojection plus orthogonal projection yields pseudo-label maps in the UAV-local frame. This pipeline costs roughly 7,922 API calls, 7.4 hours, and about $8.42 total, with manually verified category accuracy of ~94.0% and spatial-projection accuracy of ~82.5%. The annotation models are not needed at training or inference time, which is what keeps online latency low.
Future trajectory and future map imagination. A binary head on the shared current-map features predicts an n-step future trajectory mask (n=10), supervised by projected future waypoints. Because trajectory supervision shares features with map reconstruction, the current-map representation is encouraged to encode both scene structure and motion intent. Notably, the decoded trajectory mask is not re-encoded as input to downstream branches; planning information flows only through the shared representation. Future-map tokens, conditioned on current-map tokens via the causal mask, predict the map at step t+n in the same UAV-local coordinate frame, enabling direct comparison of current and anticipated states. The waypoint token aggregates all preceding representations to predict the next waypoint as a normalized 3D direction and distance, trained with an L1-plus-cosine loss.
Cross-space planning consistency. CPCL extracts a local planning direction from the predicted trajectory mask via PCA (dominant principal axis, oriented away from the UAV), and aligns it with the expert action direction—the projection of the ground-truth waypoint displacement onto the map plane—using cosine distance. The authors deliberately use the expert displacement rather than the predicted waypoint as the directional reference, avoiding error propagation from immature waypoint predictions early in training. Measured cosine similarity between map-space and expert directions rises from 0.903 without CPCL to 0.960 with it.
Training is two-stage: stage one optimizes only the navigation loss; stage two jointly optimizes navigation plus the three spatial objectives (λspatial​=0.5) and CPCL (λcons​=10−3).
Main results
On the OpenUAV benchmark (Wang et al., 2024) under the L1-assistance setting, AirForesight outperforms Random, Fixed Action, CMA, NavFoM, and TravelUAV across all metrics and difficulty levels. The headline numbers on Test Seen (Full):
| Method |
NE (m) ↓ |
SR (%) ↑ |
OSR (%) ↑ |
SPL (%) ↑ |
| TravelUAV |
68.44 |
30.47 |
61.64 |
25.37 |
| NavFoM |
93.05 |
29.17 |
49.24 |
25.03 |
| AirForesight |
56.99 |
35.83 |
69.25 |
30.22 |
Under distribution shift, gains persist but shrink: on Test Unseen Map, SR improves from 9.92% to 13.67% and NE drops from 106.19 m to 90.30 m; on Test Unseen Object, SR rises from 40.10% to 44.36%, though NE is essentially unchanged (66.98 m vs. 66.92 m). On AerialVLN-S (Bortz et al., 2023), AirForesight attains the highest SR on both validation splits (14.4% seen, 12.4% unseen) and the lowest seen NE (77.8 m), while CityNavAgent retains a substantially lower unseen NE (60.2 m vs. 86.5 m)—a point the comparison table makes visible without discussion. Multi-seed evaluation over three runs shows standard deviations below one percentage point for SR, OSR, and SPL.
Ablations and analysis
The component ablation isolates each contribution on Test Seen (Full). The navigation-only baseline achieves NE 74.02 m / SR 31.94%. Adding Current Spatial Map Modeling (CSMM) reduces NE to 62.72 m and lifts SR to 35.33%; adding Future Spatial Imagination (FSI) alone yields 63.20 m / 33.40%. Combining CSMM and FSI improves NE (62.28 m) and OSR but leaves SR and SPL slightly below the CSMM-only setting—an honest indication of directional mismatch between independently supervised trajectory and action branches. CPCL resolves this: the full model reaches 56.99 m / 35.83% / 69.25% / 30.22%. FSI's benefit is larger on the Hard split, where adding it reduces NE from 93.45 m to 84.74 m.
A representation-form comparison is informative: an online spatial-prior baseline (no prediction) performs worst (NE 99.51 m), local egocentric future prediction reaches 58.63 m, and map-space imagination reaches 56.99 m, suggesting map-level future representations are more useful than egocentric ones in this setting. Perturbation experiments support causal use of the map tokens: replacing them with zeros or Gaussian noise at inference drops Test Unseen Map SR from 13.67% to 11.44% and 11.16%, respectively, though the model retains substantial navigation capability from the raw vision-language inputs. t-SNE and cross-correlation analyses show scene clustering with trajectory-dependent variation.
Sensitivity studies identify n=10, 224×224 maps, 9 query tokens, and a 2-layer/768-dim decoder as optima, with performance degrading at both smaller and larger settings—16 tokens, for instance, degrade NE to 69.06 m, attributed to redundancy or optimization difficulty. Inference latency is 211.51 ms per step on an L40S: only 12.31 ms (~6.2%) over TravelUAV, versus 1076.39 ms for online spatial-prior construction, confirming the practical benefit of training-only map supervision.
Limitations and open questions
The authors concede several limitations. The spatial supervision rests on automatically generated pseudo-labels; errors in category extraction, grounding, segmentation, depth, and 3D-to-2D projection can propagate into training, and the 82.5% spatial-projection accuracy bounds the fidelity of the supervision. CPCL's PCA-based direction estimate assumes locally smooth trajectories and may be unreliable for sharply curved paths or sparse trajectory masks. Trajectory information reaches the future-map branch only through the shared current-map representation; explicit trajectory-conditioned future-map imagination remains unexplored. Performance degrades substantially under unseen maps and objects, and the policy continues to rely partly on raw vision-language inputs rather than exclusively on the map tokens. A further open question raised implicitly by the AerialVLN-S comparison is whether map imagination helps unseen-scene robustness as much as it helps seen-environment accuracy.
Conclusion
AirForesight formulates UAV-VLN as current-to-future spatial map imagination within a causally structured token sequence, supervised by joint map reconstruction, trajectory prediction, and a cross-space planning consistency loss that ties map-space direction to expert action direction. The approach delivers consistent improvements over strong LLM-based baselines on OpenUAV and AerialVLN-S at negligible inference overhead, and ablations attribute the gains specifically to grounding, future imagination, and their directional alignment. The main unresolved issues are pseudo-label fidelity, the smoothness assumption underlying CPCL, and generalization under large environmental shift.