HALO: High-Altitude Language-Conditioned Monocular Aerial Exploration and Navigation
Abstract: We demonstrate real-time high-altitude aerial metric-semantic mapping and exploration using a monocular camera paired with a global positioning system (GPS) and an inertial measurement unit (IMU). Our system, named HALO, addresses two key challenges: (i) real-time dense 3D reconstruction using vision at large distances, and (ii) mapping and exploration of large-scale outdoor environments with accurate scene geometry and semantics. We demonstrate that HALO can plan informative paths that exploit this information to complete missions with multiple tasks specified in natural language. In simulation-based evaluation across large-scale environments of size up to 78,000 sq. m., HALO consistently completes tasks with less exploration time and achieves up to 68% higher competitive ratio in terms of the distance traveled compared to the state-of-the-art semantic exploration baseline. We use real-world experiments on a custom quadrotor platform to demonstrate that (i) all modules can run onboard the robot, and that (ii) in diverse environments HALO can support effective autonomous execution of missions covering up to 24,600 sq. m. area at an altitude of 40 m. Experiment videos and more details can be found on our project page: https://tyuezhan.github.io/halo/.
Summary
- The paper introduces HALO, a novel system that integrates monocular metric-semantic mapping with language-guided exploration for high-altitude UAV missions.
- It employs a VGGT-based 3D reconstruction pipeline, robustly fusing GPS, ICP, and SALAD descriptors to overcome scale ambiguities.
- Real-world tests and simulations show HALO achieves up to 68.7% higher competitive ratios and reduced exploration times by 12–85%.
High-Altitude Language-Guided Monocular Aerial Exploration and Navigation: An Expert Analysis of HALO (2511.17497)
Introduction and Motivation
This work presents HALO, an autonomous system addressing semantic mapping and exploration for high-altitude unmanned aerial vehicles (UAVs) equipped only with monocular RGB cameras. It is specifically designed to enable real-time metric-semantic reconstruction, language-driven navigation, and task-size scalability onboard aerial robots. The approach enables the UAV to generate dense geometric and open-set semantic representations from monocular aerial imagery, crucially allowing for planning and execution of complex missions specified in natural language. The technical significance lies in overcoming depth estimation challenges at altitude and fusing them with dense language grounding for open-set, flexible mission execution.
Figure 1: Conceptual overview of HALO capturing monocular aerial imagery, generating metric-semantic maps, and executing hierarchical semantic planning for language-specified tasks.
System Architecture and Methods
HALO comprises two tightly integrated modules: a real-time monocular metric-semantic mapper and a hierarchical semantic planner.
Monocular Metric-Semantic Mapping
At the mapping core, HALO leverages VGGT [wang2025vggt]—a feed-forward dense 3D reconstruction transformer—to estimate depth and camera poses from sequential monocular aerial images. A pose graph is constructed by fusing (i) scale-invariant relative pose estimates, (ii) colored ICP constraints [park2017colored] for submap alignment, and (iii) GPS position priors to resolve scale ambiguities and mitigate drift. Loop closure detection is implemented via SALAD descriptors [izquierdo2024optimal], which provide robust map consistency at scale.
The geometric outputs are fused with per-pixel semantic features extracted via a RADIO/SigLIP-based encoder [heinrich2025radiov25improvedbaselinesagglomerative, zhai2023sigmoid], resulting in a 2D grid storing aggregated dense language-aligned semantic embeddings. These embeddings enable flexible, open-set semantic querying and relevance scoring during planning.
Figure 2: High-level HALO system diagram showing the monocular mapping pipeline, semantic grid formation, frontier extraction, and hierarchical semantic planning.
Hierarchical Semantic Planning
HALO introduces a two-tier planner:
- Global Planner: Segments the operating area into regions, associating each with geometric frontier clusters and semantic utility based on cosine similarity between task embeddings and grid semantic features. It selects the next-best region for exploration or exploitation using maximum cost-benefit ratio and tracks regions containing unexplored frontiers or highly relevant semantic information.
- Local Planner: Implements an ATSP-style optimization to generate shortest, information-rich traversal paths through frontier clusters, ensuring the selected clusters provide sufficient semantic utility. For exploitation, direct paths are computed to high-relevance cells for efficient information retrieval.
Simulation and Quantitative Evaluation
HALO's modules were extensively validated in custom Unity-based urban and unstructured simulators, PolyCity and Forest, with environments spanning up to 78,000 and flights over 1.6 km.
3D Mapping Analysis
Comparative ablations demonstrate HALO's VGGT-based reconstruction substantially outperforms MASt3R-SLAM [murai2024_mast3rslam], VGGT-SLAM [maggio2025vggt], and others in both geometric accuracy and semantic consistency—preserving critical scene structures at metric scale. Notably, HALO (w/o GPS) yields Chamfer distances as low as 7.01 m in the Forest scene, indicating precise 3D mapping.
Figure 3: Visual comparison of 3D reconstructions in PolyCity and Forest using multiple incremental methods.
Semantic Exploration and Planning
Simulations covered three types of multistage missions per environment, with semantic objectives ranging from “find a car accident” to “find somewhere to have a picnic." HALO achieves up to 68.7% higher competitive ratio compared to state-of-the-art semantic planners like VLFM [yokoyama2024vlfm] and geometric baselines such as FUEL [zhou2021fuel] and classical Frontier [yamauchi1997frontier]. Exploration time reductions range from 12–85%, tightly correlating with the relatedness of the semantic tasks. Notably, semantic-driven planning remains robust even for under-specified or sequentially revealed missions.
Real-World Autonomous Flight Results
HALO was deployed on custom quadrotors with Jetson AGX Orin, ZED 2i camera, and GNSS/IMU integration onboard. Missions covered up to 24,600 at 40 m altitude, validating full onboard autonomy and dense semantic navigation in complex real-world settings.
Figure 4: Real-world reconstruction and trajectory execution; onboard 3D maps visualize both explored routes and detected safe waypoints.
Dense 3D reconstructions from HALO demonstrably surpassed ZED stereo SDK-based mapping, especially in object and structure fidelity at altitude.
Figure 5: Qualitative comparison of HALO’s dense monocular mapping against ZED camera stereo mapping, highlighting superior structure preservation.
HALO consistently maintained competitive ratios exceeding 0.35, and up to 0.86 for sequential missions leveraging previously explored semantic context. This confirms both geometric accuracy and semantic flexibility for on-demand mission specification and adaptation.
Implications, Limitations, and Future Directions
HALO uniquely advances high-altitude, monocular, language-conditioned autonomy with dense open-set semantic mapping. Its approach to embedding per-pixel language features ensures adaptability: planners can flexibly respond to new, arbitrary tasks as they arise, without object-level or closed-set semantic constraints.
Strong claims in the paper include the preservation of metric-scale geometry at high altitude and the ability to execute complex, multistage, open-vocabulary tasks efficiently. Contradictory to prior art—which either compresses semantics, relies on sparse labels, or fails at altitude due to hardware or computational constraints—HALO achieves real-time performance onboard with modest hardware.
Practical impacts extend to rapid aerial surveying, search and rescue, agricultural analysis, and outdoor scene understanding under diverse task demands. Theoretically, HALO’s fusion of feed-forward geometric priors and dense language embeddings provides a template for future open-set semantic robotic autonomy.
Current memory requirements (∼12GB) and computational load pose scaling challenges for longer-term onboard deployments. Future work may integrate submapping, disk offloading, and optimized embeddings for massive environments. Additionally, uncertainty-aware mapping, next-best-view selection with radiance fields [mildenhall2021nerf, chen2024catnips], and semantic relocalization remain ripe areas for extension.
Conclusion
HALO introduces a technically rigorous, highly integrated platform for high-altitude monocular UAV exploration, combining state-of-the-art 3D reconstruction and language-embedded mapping with hierarchical semantic planning. Extensive simulation and real-world results evidence efficient, flexible, and accurate execution of open-set language-driven missions, exceeding previous semantic and geometric planners both in exploration efficiency and competitive ratio. The system sets a new benchmark for open-vocabulary semantic autonomy and paves the way for further advances in scalable real-time aerial intelligence.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
HALO: High-Altitude Language-Conditioned Monocular Aerial Exploration and Navigation — Explained Simply
What is this paper about?
This paper introduces HALO, a smart drone system that flies high above the ground and builds a detailed map using just one regular camera, GPS, and a motion sensor. The special thing about HALO is that you can give it tasks in plain English (like “find a bridge” or “look for a parking lot”), and it plans where to go and what to look at to complete those tasks quickly and efficiently.
What questions are the researchers trying to answer?
The paper focuses on two big questions:
- How can a drone flying high in the air build an accurate, detailed 3D map using only a single camera, in real time?
- How can it use that map—plus the meaning of words in your instructions—to plan smart paths and finish different tasks you describe in everyday language?
How does HALO work? (Methods in simple terms)
Think of HALO as doing two jobs at the same time: making a map and planning where to go next.
1) Building the map: “What does the world look like?”
- Camera + GPS + motion sensor: The drone uses a single camera (monocular), along with GPS for location and an IMU (a motion sensor) to understand how it’s moving.
- Turning pictures into 3D: HALO uses a modern vision model (called VGGT) to estimate how far away things are (depth) and where the camera is pointing. Imagine turning a flat photo into a pop-up book that shows distance.
- Fixing scale with GPS: A single camera can’t tell the real-world size of things by itself (is that a tiny toy car nearby or a real car far away?). GPS helps set the correct “scale” so distances make sense.
- Making an easy-to-handle map: Instead of storing a full 3D world (which is heavy on memory), HALO stores a 2D grid (like a city map from above) that marks where it has “looked” and what it “knows” there.
- Adding language meaning: For every spot on the map, HALO also stores “language features”—numbers that capture meaning from vision-LLMs (kind of like a supercharged highlighter that knows what “bridge,” “car,” or “lake” might look like). This lets the drone understand and react to new instructions on the fly.
Helpful idea: “Frontiers” are the edges between known and unknown parts of the map. HALO finds these edges to know where to explore next.
2) Planning where to go: “How do I finish the task?”
HALO uses a two-level planning system:
- Global planner (big-picture decisions):
- It divides the area into regions (like squares on a grid).
- It scores each region in two ways:
- Exploration: Is this region next to unknown space (a frontier), and might it contain useful info for the task?
- Exploitation: Is this region already known to be relevant to the task (for example, we saw something that looks like a bridge there earlier)?
- It then picks the next best region by balancing usefulness vs. travel distance.
- Local planner (exact path):
- If exploring: It plans a route that visits the most promising frontier points. This is like doing a short “tour” that hits multiple good spots efficiently.
- If exploiting: It takes the shortest path straight to the most relevant spot already found.
What did they find? (Main results and why they matter)
- In large simulations (cities and forests up to about 78,000 m²), HALO completed language-based tasks faster and with less flying distance than other strong methods.
- It achieved up to 68% higher “competitive ratio.” This score compares the path the drone actually flew to the best possible path—higher is better.
- It also reduced exploration time compared to baselines.
- In real-world flights:
- HALO ran fully onboard a custom quadrotor with an NVIDIA Jetson (so no offloading to a big computer).
- It flew at about 40 m altitude and covered areas up to about 24,600 m².
- It built good, metric-accurate 3D maps from high up with a single camera (better than off-the-shelf stereo at that height).
- It handled multiple tasks given one after another, like “find a car in the parking lot” and then “inspect the roadblock,” without needing to rebuild its map.
Why this is important:
- Many drones use heavy, expensive sensors like LiDAR for depth. HALO shows you can get high-quality mapping and task completion using just a camera with smart software.
- The system understands language-based goals, so people can direct robots using normal instructions.
What does this research mean for the future? (Implications)
- Easier human-robot teamwork: You can tell a drone what to do in plain English, and it will figure out where to go and what to look for.
- Cheaper, lighter drones: Since it uses a single camera and clever algorithms, this approach can be more affordable and practical.
- Useful for many outdoor missions: From search-and-rescue to farming to infrastructure inspection, high-altitude language-aware mapping can save time and effort.
- Next steps:
- Make it even more memory-efficient for larger maps.
- Speed up the 3D model for higher-resolution images.
- Plug in different “finish-the-task” checkers (like object detectors) depending on the mission.
In short: HALO is like giving a drone a smart, language-aware brain. It builds a meaningful map from above using just a camera, understands your instructions, and plans efficient routes to get the job done—fast and on its own.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of unresolved issues, open questions, and limitations that future researchers could address to strengthen, generalize, or stress-test HALO.
- Dependence on GPS for metric scale and drift correction: How does HALO perform in GPS-degraded or GPS-denied environments, and what alternative scale/orientation sources (barometer/altimeter, UWB, visual scale recovery) can replace or augment GNSS?
- Orientation estimation robustness: VGGT provides orientation but the paper does not quantify orientation error or drift over long flights; how does such drift impact mapping/planning and how can it be mitigated without magnetometer or multi-sensor heading?
- Domain shift for language features: The RADIO+SigLIP embeddings are not evaluated for aerial imagery domain shift; what is their precision/recall for aerial scenes, small objects, and unusual viewpoints, and does fine-tuning or domain adaptation improve relevancy maps?
- 2D semantic grid only: Semantics are stored in a 2D grid without volumetric occupancy probabilities or height; how to model occlusions, multi-level structures (e.g., bridges), and altitude changes with a memory-efficient 3D or multi-layer representation?
- No uncertainty modeling: Geometry and semantic features lack uncertainty estimates; can probabilistic semantic scores and geometric covariance be incorporated to enable risk-aware, information-theoretic planning (e.g., CSQMI) and threshold calibration?
- Frontier detection in 2D with limited occlusion reasoning: How to extend frontier detection to truly 3D occlusion-aware planning, including selecting altitudes and viewpoints that reveal occluded task-relevant areas?
- Loop closure scalability and robustness: SALAD-based loop closures are restricted to nearby submaps and do not use ICP for closure; what are recall/precision trade-offs in large loops and how to robustify closures (e.g., TEASER++, multimodal features)?
- Real-time performance characterization: The paper lacks end-to-end latency/throughput measurements (Hz) for mapping and planning; how do frame rate and cycle times scale with image resolution, flight speed (>2 m/s), and environment size?
- Memory scalability of dense semantic features: With ~12 GB usage, how do submap strategies, feature compression (e.g., PQ, PCA, learned codes), and on-disk streaming affect retrieval fidelity and planning performance over very large areas?
- Exploitation vantage selection: Re-visiting exploitation regions via their centroid may not yield the best viewpoint; how to select next-best view/altitude for semantic verification or relational queries?
- ATSP-based exploration visiting “all frontiers”: Is visiting all candidate frontiers budget-optimal? Investigate budgeted/submodular exploration with adaptive stopping to avoid diminishing returns.
- Safety and airspace constraints: The planner does not model no-fly zones, geo-fences, wind, or dynamic aerial obstacles; how to incorporate airspace regulations, weather, and safety margins into the cost-benefit trade-off?
- Ground-truth “optimal path” definition: How is the shortest possible path defined for under-specified tasks (e.g., “find somewhere to have a picnic”)? Provide transparent ground-truth construction and fairness checks across baselines for subjective semantics.
- Task termination via VLM: There is no quantitative evaluation of Qwen2.5-3B termination accuracy (false positives/negatives) under aerial imagery; can reliability be improved via multi-sensor confirmation, detector ensembles, or human-in-the-loop protocols?
- Generalization breadth: Real-world tests cover two areas at 40 m altitude; evaluate across diverse environments (agricultural, coastal, industrial), seasons, lighting, and weather to assess robustness and failure modes.
- Altitude-aware mapping/planning: HALO is demonstrated at a fixed 40 m; how do altitude changes affect reconstruction fidelity, semantic reliability, and region sizing (
s_reg), and can altitude be planned adaptively? - High-speed flight robustness: Mapping at 2 m/s avoids significant motion blur; what are the limits at higher speeds and how do shutter settings, temporal batching, and motion compensation affect VGGT and planning?
- Sensor fusion variants: Explore fusing barometric altitude, magnetometer, RTK vs standard GNSS, and camera intrinsics variability (lens, FOV, distortion) to quantify sensitivity and improve resilience.
- Persistent/multi-session mapping: How does HALO handle dynamic scene changes (construction, vegetation, traffic), multi-session re-localization, map updating/forgetting, and temporal consistency of semantic features?
- Multi-robot cooperation: Extend to multi-UAV collaborative exploration and mapping, including distributed semantic feature consistency and communication constraints.
- Compositional and relational queries: Tasks like “find a house near the lake” require relational reasoning; HALO uses cosine similarity to single embeddings—can map-level relational representations and compositional language planning improve success and path optimality?
- Semantic relevancy map evaluation: Provide quantitative metrics (precision/recall, localization error) for relevancy scoring relative to labeled aerial datasets to validate planning decisions driven by semantics.
- Real-world geometric accuracy: Beyond qualitative comparisons to ZED, measure reconstruction accuracy against LiDAR/RTK ground truth (e.g., per-structure error, alignment drift) to validate metric scale and structural fidelity.
- Parameter sensitivity and auto-tuning: Thresholds (
ε_e,ε_r,ε_ftr), frontier min/max radii, overlap frames, and region sizes are heuristic; conduct sensitivity analyses and investigate adaptive/learned parameter tuning. - Loop closure descriptors for aerial: Benchmark SALAD vs alternative aerial-suitable descriptors (e.g., NetVLAD variants, cross-view attention) under large baselines and viewpoint changes typical of high altitude.
- Waypoint graph constraints in real-world tests: Remapping continuous plans to pre-defined GPS waypoints may limit optimality; quantify the performance gap and develop continuous geofenced planners with onboard collision avoidance.
- Energy and SWaP awareness: Characterize compute power draw (VGGT, RADIO, VLM), flight-time impact, and propose resource-aware planning (e.g., semantic rate control, variable resolution) respecting SWaP and battery budgets.
- Open-set semantics at high altitude: Determine minimum detectable object size and class prevalence limits for aerial imagery; characterize failure cases for rare or small targets and propose hierarchical zoom-in behaviors.
- Georegistration and map alignment: Investigate errors aligning reconstructed maps to world coordinates and satellite base maps, and their effect on planning and task success.
- Confidence and fallback for feed-forward reconstruction: Develop confidence estimation and fallback strategies when VGGT fails (textureless water, repetitive patterns, low parallax), including switching to alternative sensors or behaviors.
Practical Applications
Immediate Applications
Below are actionable use cases that can be deployed today, based on HALO’s demonstrated onboard, real-time monocular metric-semantic mapping, language-conditioned exploration, and hierarchical planning.
- Healthcare and Public Safety: Rapid disaster scene triage and search for safe crossing points
- Description: Fly at ~40 m altitude to quickly identify semantically described features (e.g., “find a bridge,” “find a house near the river,” “find roadblocks”) to support emergency routing, resource placement, and victim search coordination.
- Tools/products/workflow: HALO stack on a Jetson AGX Orin–equipped UAV; language prompts via ground station; real-time map with frontiers and relevancy grid; exploitation mode to revisit high-value regions.
- Assumptions/dependencies: Sufficient GPS accuracy; clear weather/visibility; permissible flight operations (BVLOS/altitude restrictions); reliable semantic matching of aerial imagery; operator-in-the-loop for task termination.
- Municipal Infrastructure and Urban Planning: Compliance and inventory checks
- Description: Use natural language tasks (“find pedestrian crossings,” “find barriers,” “find benches”) to rapidly inventory and check urban features for compliance (e.g., ADA crosswalk visibility), construction progress, or temporary closures.
- Tools/products/workflow: City drone unit with HALO; automated report with geotagged findings; exploitation mode to revisit previously detected relevant regions for verification.
- Assumptions/dependencies: Open-set semantics provide robust cues for feature-level identification from altitude; adequate resolution for small features; legal flight corridors.
- Construction and Facilities Management: Site monitoring and progress verification
- Description: Periodic missions to “find the construction site,” “locate materials staging areas,” “identify blocked access routes,” and maintain a semantic map for quick re-visits and visual logs.
- Tools/products/workflow: HALO-enabled drone with waypoint graph over the site; language-conditioned queries; exploitation paths to previously identified regions; ROS 2 integration with project BIM/GIS tools.
- Assumptions/dependencies: Feature scale visible from 40 m; GPS integrity on dynamic sites; map updates handle occlusions and evolving structures.
- Precision Agriculture Scouting
- Description: Natural-language scouting for “find standing water,” “find brown patches,” “find missing rows near the north field,” leveraging open-vocabulary features for real-time area-of-interest discovery.
- Tools/products/workflow: HALO on farm UAV; task queue of semantic prompts; region-scoring to focus exploration; logging AOIs for agronomist follow-up.
- Assumptions/dependencies: Visual signs of stress or anomalies distinguishable at altitude; variability in crop types and seasons; VLM features sufficiently sensitive to agronomic semantics without custom training.
- Environmental Monitoring: Water bodies and trail accessibility
- Description: Detect lakes, rivers, bridges, trail houses/cabins, and evaluate access points (“find somewhere to cross the river”) for recreational planning and conservation operations.
- Tools/products/workflow: HALO-enabled drone with periodic patrols; semantic relevancy maps; exploit mode for revisits and time-series monitoring.
- Assumptions/dependencies: Seasonal changes and vegetation density; cloud shadows and glare; local airspace rules for parks and protected areas.
- Security and Event Operations: Dynamic feature reconnaissance
- Description: Quickly find “barriers,” “crowd congregation spots,” “vehicle parking areas,” enabling event teams to optimize routing and resource deployment.
- Tools/products/workflow: Rapid pre-event survey; natural-language task queue; exploitation mode to revisit high-risk areas; integration with command dashboards.
- Assumptions/dependencies: Adequate altitude for risk-managed flight; data privacy policies; real-time semantic performance in dense scenes.
- Real Estate and Campus Mapping: Property feature discovery
- Description: Natural-language queries like “find the pool,” “find parking lots,” “find picnic areas” for fast property or campus feature inventories.
- Tools/products/workflow: HALO-equipped prosumer/professional drone; report generation with geotagged findings; exploitation paths for photo/video capture.
- Assumptions/dependencies: Property owner consent; mapping resolution suitable for small features (e.g., benches, signs); weather and lighting.
- Software/Robotics Engineering: Drop-in onboard autonomy module
- Description: Integrate HALO’s VGGT-based monocular reconstruction, dense language embeddings (SigLIP/RADIO encoder), and hierarchical planner in existing UAV stacks for semantic mission planning.
- Tools/products/workflow: HALO ROS 2 package; PX4/ArkV6X flight controller via Air-router; SALAD loop closure, colored ICP; Qwen2.5-3B (or alternative) for task termination; 2D semantic grid outputs for downstream apps.
- Assumptions/dependencies: ~12 GB unified memory on edge GPU; image resolution tuned for latency; GNSS quality for scale recovery; operator-defined safety waypoints.
- Academia and Education: Aerial robotics coursework and benchmarking
- Description: Use HALO as a teaching/research platform for metric-semantic mapping, open-vocabulary exploration, and language-conditioned planning in aerial settings.
- Tools/products/workflow: Unity-based simulator with ROS 2 interface; baselines (Frontier, FUEL, VLFM); experiments in urban and forest scenes; analysis via competitive ratio and exploration time metrics.
- Assumptions/dependencies: Availability of GPU edge compute; access to test fields; safety and IRB approvals for campus flights.
- Daily Life (Prosumer UAVs): Natural-language “find X” missions
- Description: Hobbyists use prompts (e.g., “find my car in the parking lot,” “find picnic spots,” “find a bridge nearby”) to quickly explore and capture relevant footage.
- Tools/products/workflow: HALO packaged in consumer drone apps; waypoint graph seeded from satellite imagery; semantic AOI tagging.
- Assumptions/dependencies: Local regulations; battery/flight time; image quality and semantic feature reliability for small objects.
Long-Term Applications
These applications require further research, scaling, regulatory alignment, or productization beyond current HALO capabilities.
- BVLOS Fleet Operations for Public Safety and Logistics
- Description: Multi-drone teams performing large-area, language-conditioned search and routing (“find all viable crossing points within 10 km,” “locate roadblocks on arterial routes”).
- Dependencies/assumptions: BVLOS/regulatory approvals; robust inter-UAV coordination; cloud-edge pipelines; resilient comms; automated deconfliction.
- City-scale Semantic Geospatial Layers (“Real-time semantic orthomaps”)
- Description: Persistent aerial layers embedding open-vocabulary features for urban analytics (e.g., inventory of crossings, benches, barriers, construction progress).
- Dependencies/assumptions: Scalable memory and storage (submap streaming, disk offloading); periodic refresh; integration with municipal GIS; privacy-preserving pipelines.
- Autonomous Infrastructure Inspection Scheduling
- Description: Language-based tasking (“inspect all pedestrian crossings near schools,” “revisit bridges flagged last month”), dynamically planned via exploitation of prior semantic maps.
- Dependencies/assumptions: Condition assessment models; reliable object-scale recognition from altitude; integration with asset management systems; change detection.
- Precision Agriculture at Scale with Domain-tuned Semantics
- Description: Region-wide scouting with domain-specific features (“nutrient stress signs,” “irrigation leaks,” “pest damage”), adapting exploration plans as conditions evolve.
- Dependencies/assumptions: Custom aerial semantic encoders trained on crop types; seasonal models; multi-spectral/thermal sensors; agronomic validation.
- Post-Disaster Damage Mapping and Recovery Planning
- Description: Open-vocabulary search for “collapsed structures,” “blocked roads,” “temporary encampments,” with efficient exploit-based revisits for confirmation and aid routing.
- Dependencies/assumptions: Robust semantics for damaged/altered scenes; priority-aware planners; human-in-the-loop verification; rugged hardware for adverse conditions.
- Wildlife and Conservation Monitoring with Natural-language Queries
- Description: “Find beaver dams,” “identify new trails,” “locate illegal dump sites,” combining semantic exploration and scheduled revisits for longitudinal studies.
- Dependencies/assumptions: Ecological domain features; ethical considerations and permits; low-disturbance flight behavior; cross-season generalization.
- 3D Obstacle-aware High-altitude Autonomy
- Description: Extend HALO’s 2D grid semantics to full 3D occupancy and obstacle reasoning for altitude changes and complex terrain.
- Dependencies/assumptions: Memory-efficient 3D representations (e.g., submaps, compression, sparse voxel fields); improved occlusion handling; robust pose scaling without GPS.
- Edge-to-Cloud Semantic Query Services
- Description: Onboard acquisition with streamable semantic feature maps; cloud services for complex queries (“find all likely picnic areas near water and shade”).
- Dependencies/assumptions: Reliable uplink; standardized feature formats; privacy/security; scalable indexing and retrieval.
- Multi-robot Cooperative Semantic Exploration
- Description: Teams decompose tasks and share language-embedded maps, balancing exploration/exploitation across agents to minimize global mission time.
- Dependencies/assumptions: Map merging of dense language features; communication constraints; consensus on task relevancy; fault tolerance.
- Insurance, Claims, and Risk Assessment
- Description: Language-based aerial surveys (“identify roof damage,” “find flood standing water zones”) with repeatable exploitation paths for re-inspection.
- Dependencies/assumptions: Higher-resolution imaging or lower altitude; domain-tuned detectors; compliance with data privacy and property access rights.
Notes on Feasibility and Key Assumptions
- Sensor and compute: Demonstrated on Jetson AGX Orin with ~12 GB unified memory; performance depends on image resolution, VGGT latency, and feature map size. Higher-fidelity mapping requires more compute/memory or submap streaming to disk.
- Navigation: GPS used for scale recovery and drift reduction; orientation primarily from feed-forward reconstruction. GPS degradation, multipath, or magnetometer issues can impact mapping accuracy.
- Semantics: Open-vocabulary language features (SigLIP/RADIO) enable flexible prompts but may need domain-specific tuning for niche tasks (agriculture, damage assessment).
- Mapping representation: 2D semantic grid is memory-efficient but limits vertical obstacle reasoning; frontier-based exploration assumes planar mapping suffices at high altitude.
- Environmental conditions: Weather, lighting, and scene texture influence reconstruction and semantic reliability. Performance in dense urban canyons or heavy foliage may vary.
- Safety and regulation: Altitude, BVLOS, and operations over people require regulatory compliance. Real deployments should employ geofencing, waypoint safety graphs, and operator oversight.
- Mission workflows: HALO supports sequential task execution with exploitation of prior findings; task termination is currently via VLM or operator—mission-specific policies may be needed.
Glossary
- Air-router: A software interface that bridges the autonomy stack and the flight controller for waypoint tracking in GPS coordinates. "We use Air-router to interface with the flight controller for waypoint tracking in GPS coordinates, and remap the planned path to the nearest reachable waypoint."
- Asymmetric Traveling Salesman Problem (ATSP): A variant of TSP where travel costs between nodes are direction-dependent; used to compute efficient visitation paths over frontier targets. "We set up an Asymmetric Traveling Salesman Problem (ATSP) to explore a region as follows."
- boustrophedon coverage path: A systematic back-and-forth coverage trajectory used to survey an area efficiently. "we ran a boustrophedon coverage path and collected images every 2 m along the trajectory."
- camera intrinsics: The internal parameters of a camera (e.g., focal length, principal point) required for geometric reconstruction. "Fig.~\ref{fig:vggt_factor_graph} provides an overview of the VGGT-based mapping module to estimate camera intrinsics, dense depth and camera poses."
- Chamfer distance: A two-sided average nearest-neighbor distance metric for comparing point clouds. "and (iii) Chamfer distance, which is the averaged two-sided distance."
- competitive ratio: A measure of efficiency comparing the executed path length to the shortest possible path to complete the task. "Efficiency is quantified by the competitive ratio between the executed path length and the shortest possible path required to complete the task."
- cosine similarity: A metric for comparing the alignment of two vectors, used here to match text and visual embeddings. "We then compute the cosine similarity between the task text embeddings and the semantic feature embeddings to obtain a relevancy score for each grid cell in the semantic feature grid."
- DUST3R: A feed-forward model for pairwise image geometry that predicts dense pointmaps via joint encoders and decoders. "DUST3R and its variants operate on image pairs with joint encoders and separate decoders to predict pointmaps."
- exponential moving average: A smoothing technique that updates aggregated features by weighting recent observations more. "When we have multiple observations of the same grid cell, we use exponential moving average to update its features."
- feed-forward 3D reconstruction: Learned models that predict geometry directly from images without iterative optimization, enabling real-time dense mapping. "Feed-forward 3D reconstruction models have brought about a significant step forward in unified, end-to-end deep learning frameworks for 3D reconstruction."
- frontiers: Boundaries between known and unknown areas in a map used to guide exploration. "We compute geometric frontiers on the occupancy grid map."
- Gaussian Splatting: A rendering and mapping technique that represents scenes with Gaussian primitives for fast, differentiable updates. "ATLAS and Rayfronts further incorporate language features into Gaussian Splatting and voxel maps, enabling semantic navigation and exploration."
- GNSS: Global Navigation Satellite System providing absolute positioning (e.g., GPS), used for onboard localization. "It is equipped with an NVIDIA Jetson AGX Orin onboard computer, a ZED 2i camera, a VectorNav VN-100T IMU, a u-blox ZED-F9P GNSS, and an ARKV6X flight controller running PX4 1.16."
- Iterative Closest Point (ICP): An algorithm to align point clouds by iteratively matching nearest points and refining the transformation. "calculate a relative transformation between two adjacent submaps with colored Iterative Closest Point (ICP)"
- loop closure detection: The process of recognizing previously visited places to reduce drift and improve global consistency. "Similar to VGGT-SLAM, we implement loop closure detection using SALAD descriptors."
- occlusion-aware 3D reasoning: Planning and mapping that account for unseen space hidden by obstacles using frontier analysis. "We detect geometric frontiers from the 3D reconstruction, enabling occlusion-aware 3D reasoning."
- octomap: A probabilistic 3D occupancy mapping framework using octrees, here augmented with semantics for exploration. "the work in~\cite{asgharivaskasi2023semantic} presented an information-driven exploration framework based on a semantic-augmented octomap."
- open-set metric-semantic map: A map that combines geometry and language features without restricting to a fixed set of classes. "build open-set metric-semantic maps from aerial imagery."
- open-vocabulary semantics: Semantic representations that are not limited to predefined categories, enabling flexible task specifications. "most existing methods that leverage open-vocabulary semantics for exploration are designed for ground robots"
- orthomapping: Generating georeferenced maps by reconstructing scene structure (e.g., via SfM) and projecting imagery onto a consistent ground plane. "Structure-from-Motion (SfM) can be used for orthomapping to obtain dense point clouds"
- orthomosaics: Large stitched and georeferenced 2D maps created from aerial images. "The most common use of aerial images for real-time applications is to build orthomosaics by stitching 2D aerial images."
- pointmaps: Dense per-pixel 3D point predictions aligned with image pixels produced by feed-forward models. "DUST3R and its variants operate on image pairs with joint encoders and separate decoders to predict pointmaps."
- pose graph: A graph of poses connected by constraints (e.g., relative transforms, priors) used for global consistency in mapping. "process batches of images to build submaps and align these submaps using a pose graph."
- Qwen2.5-3B: A compact vision-LLM used to determine task completion from onboard imagery. "We used a Qwen2.5-3B vision-LLM (VLM) to determine task completion by querying the input images with the task description."
- RADIO (encoder): A vision foundation model encoder adapted to produce dense language-aligned features for semantic mapping. "We use a modified RADIO encoder with a SigLIP adapter to obtain dense language-aligned features"
- region-growing algorithm: A clustering method that expands regions based on local connectivity to detect and group frontier cells. "detect new frontier clusters using a region-growing algorithm."
- SALAD descriptors: Learned global image descriptors for aerial place recognition used to detect loop closures. "Similar to VGGT-SLAM, we implement loop closure detection using SALAD descriptors."
- scale ambiguity: The inability of monocular vision to determine absolute scale without external cues. "monocular systems have scale ambiguities."
- semantic feature grid: A 2D grid storing dense language features per cell for open-set semantic reasoning. "We maintain a grid of task relevance scores that are computed from the semantic feature grid."
- SigLIP: A language-image pretraining method whose encoders provide aligned text-image embeddings for open-vocabulary tasks. "We first obtain text embeddings of the task description using the SigLIP text encoder."
- SIM(3): The 3D similarity transformation group (scale, rotation, translation) used for optimization in SLAM variants. "We ran VGGT-SLAM with both the SL(4) and SIM(3) optimization approaches presented in their work."
- SL(4): A Lie group used as an alternative optimization domain for projective transformations in SLAM. "We ran VGGT-SLAM with both the SL(4) and SIM(3) optimization approaches presented in their work."
- Simultaneous Localization and Mapping (SLAM): Jointly estimating the robot trajectory and the environment map from sensor data. "or from monocular visual SLAM to create elevation maps"
- Structure-from-Motion (SfM): Reconstructing 3D structure and camera motion from multiple overlapping images. "Structure-from-Motion (SfM) can be used for orthomapping to obtain dense point clouds"
- submaps: Locally consistent chunks of the map built from batches of images and later aligned globally. "Similar to VGGT-SLAM, we process batches of images to build submaps and align these submaps using a pose graph."
- Visual Geometry Grounded Transformer (VGGT): A transformer model that predicts dense depth and camera parameters for multi-view geometry. "We design an incremental mapping module based on VGGT for estimating dense depth and poses from high-altitude aerial imagery."
- vision-LLMs (VLMs): Models that jointly process images and text to enable language-conditioned perception and planning. "Advances in vision-LLMs (VLMs) have shown promising results in robotics for providing semantically rich representations"
- voxel maps: 3D grid-based representations discretizing space into volumetric cells used for geometric and semantic mapping. "ATLAS and Rayfronts further incorporate language features into Gaussian Splatting and voxel maps, enabling semantic navigation and exploration."
Collections
Sign up for free to add this paper to one or more collections.
