ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

Abstract: We introduce ActiveFly-Bench, the first benchmark to bridge cyberspace reasoning and physical-world interaction for UAV embodied perception. The benchmark decomposes active perception into three hierarchical tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC), explicitly connecting high-level task understanding, behavior planning, and low-level control. The datasets are collected from both real-world and simulated outdoor environments for training and evaluation. We further develop ActiveFly, a closed-loop UAV agent that integrates visual-language reasoning with fine-grained control, and deploy it on a physical UAV platform. Experiments with representative VLMs and VLA models show that current UAV agents still struggle with behavior planning, viewpoint adjustment, and robust task completion in active perception. These results establish ActiveFly-Bench as a new testbed for embodied aerial intelligence.

• The paper introduces a hierarchical benchmark that segments UAV embodied perception into Air-EQA, OBP, and FLUC to bridge language reasoning with physical control.
• It employs cascaded vision-language and action modules, validated through real-world and simulated tests with metrics like SR, OSR, and nDTW.
• Findings indicate that high language accuracy does not ensure efficient UAV control, underlining a need for robust behavior planning and multimodal integration.

ActiveFly-Bench: A Hierarchical Benchmark Connecting Vision-Language-Action for UAV Embodied Perception

Motivation and Overview

The paper "ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception" (2607.10180) rigorously addresses the semantic and algorithmic gap between high-level language-guided perception tasks and fine-grained physical-world control in Embodied AI, specifically for UAVs. Recognizing that previous Embodied Question Answering (EQA) and Vision-Language-Action (VLA) benchmarks suffered from either coarse action spaces, disconnected reasoning and execution, or simulation-only experiments, this work formulates a multi-level benchmark for evaluating the full pipeline from language-driven reasoning through planning to low-level action, all instantiated on real and simulated UAV platforms.

The principal contribution is the introduction of the hierarchical ActiveFly-Bench, explicitly partitioned into three aligned subtasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-Grained Language-Guided UAV Control (FLUC). This decomposition enables the evaluation of how well UAV agents can ground language in physical-world behavior and achieve robust task completion through closed-loop interaction.

Figure 1: Overview of the ActiveFly-Bench task structure, illustrating the sequential flow through reasoning (EQA), planning (OBP), and fine-grained UAV control (FLUC).

Benchmark Structure and Data Generation

Hierarchical Task Decomposition

• Air-EQA (Cyberspace reasoning): Given an open-vocabulary question and initial observation, the UAV cannot directly answer and must actively explore to collect semantically relevant information. This forces explicit embodied reasoning and demonstrates that vision-LLMs alone are insufficient for robust scene understanding in a mobile platform.
• OBP (Bridging intention to execution): The agent must translate high-level questions into explicit, context-aware observation behaviors. OBP grounds linguistic intent into motor-affordance plans, aligning high-level semantics with physical constraints.
• FLUC (Fine-grained physical-world execution): Unlike previous VLA/VLN tasks, FLUC requires precise pose and gimbal alignment (7DoF action space) to ensure that not just the UAV's position, but its orientation and camera framing, are optimized for observation.

This decomposition is visualized in explicit task types and data examples:

Figure 2: Task examples illustrating the span of question types in Air-EQA, the bridging role of OBP, and the diversity and complexity of FLUC trajectories in various scenarios.

Dataset Generation and Annotation Pipeline

The dataset construction hinges on expert-driven and systematic protocols. High-quality demonstration trajectories are curated across both simulation and real-world settings, ensuring a coverage of occlusion, non-frontal, peripheral, and distant scenarios, explicitly designed to prevent shortcut solution paths from initial observations.

The annotation process leverages template-based methods for elementary behaviors and expert freeform descriptions for complex scenarios, then applies targeted augmentation by perturbing key waypoints, yielding diverse but semantically equivalent demonstration bundles for each instruction/question. Rigorous multi-model and human-in-the-loop filtering guarantees that questions require actual embodiment to answer, and that distractor options do not leak information.

Figure 3: Visualization of the dataset generation pipeline and statistics, including trajectory length distributions, language annotation statistics, and scenario balancing.

System and Evaluation Architecture

Closed-Loop ActiveFly Agent

ActiveFly comprises cascaded VLM and VLA modules. The VLM infers the observation behavior plan given the current context and language, which then conditions the VLA for fine-grained action prediction. After executing the control trajectory, the VLM consumes a window of onboard images and the question to produce a final answer. This system is deployed on real UAV hardware using a ground–drone collaborative framework with onboard SLAM and offboard inference, achieving sub-second control latency.

Figure 4: Sample of real-world deployment results, with both first- and third-person perspectives demonstrating end-to-end closed-loop execution.

Metrics and Decomposition Analysis

Evaluation on Air-EQA uses both strict QA accuracy and APL (accuracy weighted by path length) to probe reasoning and efficiency. FLUC leverages Success Rate (SR), Oracle Success Rate (OSR), Navigation Error (NE), and normalized Dynamic Time Warping (nDTW) for route similarity, penalizing both routing inefficiency and suboptimal viewpoint selection. The composite embodied perception (EP) success metric quantifies holistic performance across planning, execution, and reasoning.

Experimental Results and Error Diagnoses

Numerical Results and Contradictory Findings

On the three-tiered benchmark, all non-human agent baselines exhibit a marked performance drop between isolated Air-EQA reasoning (acc. >63%) and end-to-end embodied perception (SR 13–49% depending on VLA selection). Notably, the continuous-control VLA (Pi-0.5) achieves 31% SR and 71% OSR in FLUC, outperforming discretized baselines in oracle success but suffering from longer, less efficient paths as shown by lower APL and higher NE. Subsequent error analysis reveals that a majority of failures emanate from OBP errors and Air-EQA "escape" cases, where strong VLMs can answer questions via partial observations without following the intended optimal exploration plan.

Figure 5: Category-level analysis and error breakdown, highlighting the "escaped" case where non-optimal trajectories still yield correct answers due to accumulated observations.

These results stress that current systems, despite attaining nominally high QA accuracy, frequently lack robust behavior planning and do not reliably execute the physically optimal plans needed for efficient information acquisition. Human upper-bound is not approached except on sub-modules with human input, clarifying the substantial gap remaining for fully autonomous agents.

Implications and Future Directions

Practically, ActiveFly-Bench sets a new empirical standard for the development and evaluation of language-driven embodied agents operating in unconstrained, high-DOF environments. The rigorous filtration of dataset leakage and the introduction of OBP as an explicit intermediate aligns physical-world deployment with simulated evaluation and constrains the previously ambiguous notion of "active perception" to measurable quantities. Most notably, it exposes the current limitations in generalizing semantic plans to context-grounded motor execution and demonstrates that mere scaling of VLMs is insufficient for robust, sample-efficient embodied intelligence.

Theoretically, this work makes clear that embodied perception for aerial agents requires joint optimization not only of vision-language grounding but also of physically validated behavioral affordances. Future directions should target integrated architectures for hierarchical planning, more tightly coupled multimodal representations, and real-world closed-loop learning frameworks robust to noise, delay, and environmental uncertainty. As aerial robotics become more widespread in inspection, logistics, and search domains, benchmarks of this scale and rigor will be critical for driving progress beyond current simulation-constrained capabilities.

Conclusion

ActiveFly-Bench provides a strong foundation for the systematic evaluation of UAV embodied perception, bridging language-guided reasoning, behavior planning, and physical-world action with aligned, semantically rigorous benchmarks. Extensive multi-model evaluation and real-world deployment highlight critical limitations and enable reproducible comparison for future research. This benchmark is poised to anchor the next wave of research at the intersection of language, perception, and physical actuation in aerial robotics.