Can Aerial VLA Models Cooperate? Evaluating Closed-Loop Air-Ground Coordination with CARLA-Air
Abstract: Recent aerial vision-language-action (VLA) models show promising single-UAV capabilities, such as tracking moving objects and navigating to language-specified landmarks. However, it remains unclear whether these capabilities can transfer to air-ground cooperation, where a UAV and a UGV must act jointly in a shared, closed-loop physical world. We study this question with CARLA-Air, a single-process air-ground evaluation environment that unifies CARLA and AirSim inside one Unreal Engine runtime. By sharing the same world state, physics tick, and sensing pipeline, CARLA-Air enables physically consistent UAV--UGV interaction and precise measurement of simulation-timestamp alignment and effective coordination latency. Using CARLA-Air, we evaluate representative aerial VLA and planning baselines on two complementary diagnostic tasks: moving-platform landing and occlusion-recovery escort. The results show that current aerial VLA models can often track or follow a ground partner, but struggle to convert this single-agent competence into stable cooperative behavior. State prompting provides limited benefit, and naive bidirectional interaction fails to consistently improve performance and can amplify errors for most baselines. These findings suggest that, under the tested text-based cue interfaces, zero-shot cooperative air-ground VLA requires three components beyond the current paradigm: explicit partner-state grounding, low-latency action coordination, and team-level objective alignment. Our code is available at https://github.com/louiszengCN/CarlaAir.
Summary
- The paper demonstrates that current aerial VLA models can track effectively but fail to translate this competence into reliable cooperative landing and recovery actions.
- CARLA-Air employs a unified simulation platform that synchronizes UAV and UGV physics and sensor data, enabling rigorous assessment of closed-loop coordination.
- Empirical results show that traditional cue-based methods do not remedy the fundamental lack of partner-aware action grounding, highlighting a need for explicit cooperative modules.
Evaluating Air-Ground Cooperative Competence in Aerial VLA Models Using CARLA-Air
Motivation and Context
The integration of air-ground cooperative intelligence is pivotal for applications spanning urban search-and-rescue, logistics, inspection, and transportation. Unmanned Aerial Vehicles (UAVs) equipped with vision-language-action (VLA) models exhibit advanced single-agent tracking, navigation, and reasoning abilities. However, previous research has predominantly focused on single-agent metrics or offline perception, leaving a gap regarding the efficacy of VLA models in realizing joint UAV–UGV (Unmanned Ground Vehicle) closed-loop cooperation in shared physical worlds. Existing simulation platforms often lack unified world state or real-time latency instrumentation, thereby undermining rigorous evaluation of true interaction effects.
CARLA-Air: Unified Single-Process Air-Ground Simulation
CARLA-Air resolves core architectural barriers by embedding CARLA and AirSim within a shared Unreal Engine runtime. This architecture ensures synchronous physics tick, unified world state, and aligned sensor pipelines, eliminating cross-process clock drift and timestamp mismatches endemic to ROS-bridge solutions. By preserving native APIs for both CARLA and AirSim, the platform enables seamless integration and evaluation for a wide range of UAV and UGV policies, including proprietary or legacy codebases.
Figure 1: CARLA-Air runtime architecture: the simulation embeds both CARLA and AirSim under a unified world state, advancing ground and aerial actors synchronously and sampling paired sensors within the same tick.
CARLA-Air’s design maintains CARLA as the authoritative world manager (via CARLAAirGameMode), with AirSim UAVs instantiated as actor-level entities. This prevents competing GameMode controllers and supports multi-modal sensing, coordinate-frame unification (centimetre Z-up vs. metre NED Z-down), and efficient rendering across UAV and UGV platforms.
Figure 2: Resolving the single-GameMode constraint: CARLA is retained as the global manager, while AirSim’s flight actor operates as a regular world entity, enabling direct air-ground integration.
Figure 3: Coordinate-frame alignment ensures deterministic conversion and Z-axis sign flip for consistent state and metric evaluation across UAV and UGV.
CARLA-Air achieves by-construction zero simulation-timestamp offset, demonstrated across diverse sensor payloads, control frequencies, agent counts, and traffic conditions. Noise reduction ranges from – compared to bridge-based setups, enabling reliable measurement of latency-sensitive multi-agent phenomena.
Figure 4: Example of synchronized aerial-ground sensing: sensor streams from both platforms are sampled in the same simulation tick, providing aligned multi-modal observations without post-processing or interpolation.
Diagnostic Tasks: Cooperative Moving-Platform Landing and Occlusion-Recovery Escort
To rigorously assess air-ground cooperative capabilities, two tasks are formalized:
- Moving-Platform Landing: UAV tracks a moving UGV (truck), aligns with its cargo bed, and lands—diagnosing conversion from tracking to coordinated action.
- Occlusion-Recovery Escort: UAV must re-acquire visual contact with a UGV after temporary occlusion, utilizing partner-state cues—probing perception-side recovery under real-time constraints.
Three interaction modes are studied:
- C0: Independent execution (no communication);
- C1: Semantic state cue from UGV to UAV;
- C2: Naïve bidirectional action coupling (UAV forward velocity modulates UGV speed).
Baselines and Metrics
The evaluation covers representative end-to-end VLA models (AerialVLA, OpenFly, OpenUAV), VLM/planning hybrids (SPF), and traditional VLN (AerialVLN). The state-based Rule-Coop-State serves as a solvability reference with explicit metric access and deterministic rules. Metrics include Tracking Success Rate (TSR), Landing Success Rate (LSR), Cooperative Conversion Rate (CCR), Cooperation Gain (CG), Recovery Success Rate (RSR), Re-acquisition Time (RAT), Decision Frequency (DF), and Effective Coordination Latency (ECL).
Empirical Findings
Moving-Platform Landing: VLA baselines consistently show high tracking success (TSR ) but fail to convert this to cooperative landing (LSR ; CCR ). Addition of semantic cues (C1) or action feedback (C2) yields negative or insignificant Cooperation Gain; performance can even degrade (CG for AerialVLA, C2).
Rule-Coop-State achieves markedly higher LSR ($0.42$) and CCR ($0.50$), confirming task feasibility when explicit state and low-latency coordination are available.
Figure 5: Landing process trajectory diagnosis: representative aerial and ground trajectories illustrate instability in VLA baselines—cargo-bed alignment rarely translates to touchdown, especially under naive interaction coupling.
Occlusion-Recovery Escort: C1 partner cues bring limited success; for most baselines, RSR differences between C0 and C1 are within seed-level variability, and RAT remains above 4 seconds. Only SPF, whose output space aligns with cue structure, shows a modest gain.
Rule-Coop-State attains RSR $0.86$ and RAT $1.8$ sec, again exposing the gap as architectural rather than intrinsic.
Figure 6: Cooperative Occlusion-Recovery Escort: UAV recovers from temporary occlusion by searching with partner-state cues—VLA baselines rarely leverage the cues to stabilize reacquisition.
Prompt-format Ablation: Across semantic, numeric, noisy, and oracle geometric cues, VLA models exhibit consistent bottlenecks; even oracle cues only marginally improve LSR, with CG within statistical noise. Action grounding into cooperative behavior remains the dominant failure mode.
Timing: Lower-frequency baselines (e.g., SPF, OpenUAV) have higher ECL and less stable bidirectional coupling, but even high-frequency methods (AerialVLN, 9 Hz) display poor cooperative conversion, implicating fundamental architectural, not temporal, deficiencies.
Implications and Future Directions
The study decisively demonstrates that zero-shot aerial VLA models do not naturally transfer single-agent competence to air-ground team-level cooperation. Merely adding cross-agent cues, semantic prompts, or naive interaction does not close the gap—tracking does not reliably become coordination, and partner-state signaling rarely catalyzes recovery or stable joint action.
Theoretical implications: Cooperation requires explicit partner-state modeling, shared phase representation, team-level objectives, and mechanisms for grounding partner cues in action—none of which are present in current autoregressive text-based VLA architectures.
Practical ramifications: Deployment of zero-shot VLA for cooperative air-ground missions (e.g., UAV–UGV delivery, search-and-rescue) should be precluded absent cooperative-aware modules, interface redesign, and closed-loop validation. Platforms lacking strict tick synchronization will confound real-time evaluation and may mask coordination failures.
Future work: Architectural advances are necessary—possibly integrating learned partner-state embeddings, latency-mitigated inter-agent communication, and explicit cooperative reward shaping. Cooperative fine-tuning, multi-agent reinforcement learning, and analysis of team-level scaling could address current limits. Further, sim-to-real transfer and ethical deployment in privacy- or safety-critical scenarios warrant independent investigation.
Conclusion
CARLA-Air establishes a robust, causally consistent infrastructure for air-ground embodied intelligence evaluation. Empirical results across tracked baselines and diagnostic tasks reveal that current aerial VLA models, despite strong single-agent proficiency, fail to realize closed-loop cooperative competence in multi-platform settings. The cooperation bottleneck stems not from cue design or information formats but from the absence of partner-aware action grounding. Future VLA and embodied AI research must prioritize explicit coordination policies, latency mitigation, and cooperative interface design to advance deployed multi-agent autonomy.
(2605.31066)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper asks a simple question with big implications: if a flying robot (a drone) already knows how to see, understand language, and act on its own, can it team up well with a driving robot (a ground vehicle) in real time?
To study this, the authors built a new simulator called CARLA-Air that lets a drone and a robot car act together in the exact same virtual world at the exact same time. Then they tested several smart drone models to see if “being good alone” turns into “being good teammates.”
What are the main questions?
- Do drone models that can follow instructions and track moving things on their own naturally work well with a ground vehicle partner?
- Does giving the drone extra hints about its partner help?
- Does letting the drone and car influence each other’s actions help or hurt?
- How much do delays and timing glitches matter when two robots try to cooperate?
How did they study it?
A new test world: CARLA-Air
Think of two friends trying to dance together to music. If they’re playing songs on different phones that are slightly out of sync, they’ll keep stepping on each other’s toes. Many older robot simulators are like those out‑of‑sync phones: the drone and the car run in separate programs with different clocks, causing small time mismatches.
CARLA-Air puts both the drone and the car in one “room” with one “clock”:
- One shared world and physics “tick” (the heartbeat of the simulation).
- Cameras and sensors for both robots are captured at the same moment.
- This makes cooperation fair to evaluate and lets the researchers measure timing delays accurately.
They showed that common “bridged” setups create about a 12 millisecond average mismatch between the drone and car sensors, while CARLA-Air removes that mismatch by design. That lowers measurement noise by about 5 times, so results are clearer and more trustworthy.
Two teamwork challenges
The team kept the drone’s normal interface: it sees through its camera, reads a text instruction, and outputs drone actions. They designed two simple but revealing tasks:
- Moving-Platform Landing: The car drives with a flat cargo bed. The drone must follow the car and land on the moving bed. This checks if “tracking” turns into precise, coordinated action.
- Occlusion-Recovery Escort: The car sometimes goes out of view behind buildings or bridges. The drone must find it again using clues. This checks if partner hints turn into better perception and recovery.
Ways the robots “talked” (cooperation modes)
They tried three modes to see if communication helps:
- C0: No communication. The drone and car act independently.
- C1: The car sends short text cues to the drone (for example, which direction it’s turning).
- C2: For landing only, the drone’s forward-speed signal influences the car’s speed, creating simple two-way coupling.
They tested several popular “vision-language-action” (VLA) drone models without extra training for teamwork (this is called “zero-shot”). For comparison, they also tried a simple rule-based teammate that is allowed to use exact numbers about positions and speeds (like GPS-level data). This rule-based method isn’t “fair” to compare directly, but it shows what the tasks look like when the drone really knows where everything is.
What did they find?
- Good at following, bad at cooperating: Most drone models could track or follow the car, but often failed to turn that into a smooth, successful landing. In other words, “I can see you” didn’t become “I can land on you.”
- Extra hints didn’t fix it: Sending the drone text cues about the car (C1) helped inconsistently, and sometimes not at all.
- Two-way influence often made things worse: Letting the drone’s speed signal affect the car’s speed (C2) created unstable behavior and reduced success more often than not.
- Clear state + fast coordination works: The rule-based teammate, which had exact partner-state info and very low delay, did far better. That shows the tasks are solvable, but today’s text-driven VLA approach lacks key pieces for real cooperation.
- Timing matters—but isn’t the only problem: Slower decision-making and delays make cooperation harder, but even faster models struggled. So the issue is not just speed; it’s also how actions are grounded in the partner’s state.
Why this matters: The results reveal a gap between “single-robot skill” and “team skill.” Just giving a smart drone more messages doesn’t automatically produce teamwork. It needs the right kind of shared understanding and quick, reliable coordination.
What does this mean going forward?
To get real, stable air–ground teamwork from general-purpose, language-driven drone models, the paper suggests three missing ingredients:
- Explicit partner-state grounding: The drone needs a clear, reliable way to understand where its teammate is and what it’s doing, not just vague text hints.
- Low-latency action coordination: Decisions and adjustments must happen quickly, with minimal delay, so the two robots don’t trip each other up.
- Team-level objective alignment: Both robots must act toward the same goal in a coordinated way, not just do their own thing while “talking.”
CARLA-Air gives researchers a clean, fair place to test these ideas because it removes timing glitches and measures coordination delays precisely. The authors also note limitations: they only tested in simulation, mostly put the cooperation burden on the drone side, and did not fine-tune the VLA models specifically for teamwork. Real-world testing and better training for cooperation are important next steps.
In short: Today’s drone models can see and follow, but real teamwork with a ground partner needs clearer shared state, faster coordination, and shared goals. CARLA-Air helps us figure out how to build that.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains missing, uncertain, or unexplored.
- Sim-to-real transfer: No hardware-in-the-loop or field trials quantify how the findings hold under real sensor noise, wind, GPS drift, actuation delays, or heterogeneous clocks. Action: evaluate on a real UAV–UGV pair with synchronized logging to measure effective coordination latency (ECL), success rates, and safety margins.
- Communication realism: The study assumes idealized in-engine timing without network jitter, packet loss, bandwidth constraints, or clock drift. Action: introduce configurable networking (latency distributions, dropouts, rate limits) and test robustness of cooperative policies to these conditions.
- UGV-side cooperation is unstudied: C2 uses naive longitudinal coupling, with no partner-aware planning or negotiation on the ground vehicle. Action: implement and compare UGV planners that reason over UAV intent/state (e.g., co-optimization, model predictive control, negotiation protocols).
- Training for cooperation: All VLA evaluations are zero-shot; it is unknown whether fine-tuning or RL with partner-state cues can learn stable cooperative mappings. Action: create train/val/test splits with curriculum-based cooperative tasks and report CCR/LSR/RSR gains from cooperation-aware training.
- Interface bottleneck (text prompts): Only text-based cues were tested; even numeric cues were delivered as text. Action: test structured, low-latency vector inputs (e.g., relative pose/velocity, covariance) and differentiable communication channels (learned message encoders/decoders) integrated into the policy.
- Partner-state grounding in architecture: Models lack explicit mechanisms to map partner state into action. Action: add partner-state encoders, phase estimators (e.g., approach/align/descend), and joint action heads; ablate their impact on CCR and stability.
- Team-level objective alignment: How to encode shared rewards/goals into VLAs is open. Action: design loss functions and multi-agent training signals reflecting team success (e.g., touchdown quality, escort continuity) and measure CG improvements.
- Latency–stability characterization: ECL is measured but not linked to closed-loop stability. Action: derive/control-theoretic analyses and empirical phase portraits relating ECL, decision frequency, and gain to oscillations and failure modes.
- Safety and reliability metrics: Success rates omit safety-critical aspects (minimum separation, collision/near-miss counts, touchdown force, energy/time efficiency). Action: add safety/efficiency metrics and evaluate risk–performance trade-offs.
- Task diversity and coverage: Only moving-platform landing and occlusion-recovery escort were studied. Action: add cooperative mapping, convoy with traffic, payload handoff, coordinated crossing, and search-and-rescue with dynamic constraints.
- Multi-agent scaling: Single UAV–UGV pair only. Action: evaluate many-to-many (multi-UAV/multi-UGV) settings, interference management, role assignment, and scalability under urban traffic.
- Perception modalities: VLAs appear to use primarily RGB; benefits of IMU, depth, LiDAR, GPS/RTK, and HD maps are unknown. Action: provide multi-modal sensor suites and quantify their effect on occlusion recovery and landing precision.
- Environment and physics realism: Effects of wind, turbulence, ground effect, downwash, and moving-surface compliance during landing are unvalidated. Action: calibrate/augment physics and assess sensitivity to aerodynamic and contact-model fidelity.
- Asynchronous sensing: Real systems have non-identical sensor clocks; the platform enforces perfect alignment. Action: inject controlled asynchrony/skews between modalities and agents to test policy robustness.
- Communication semantics: Only coarse “forward-right” and bearing cues were tested; richer georeferenced messages (e.g., ENU pose, velocity, uncertainty) might help. Action: systematically vary semantic richness/uncertainty of partner messages and measure CCR/RSR.
- Reference baseline asymmetry: Rule-Coop-State has privileged metric state and low latency, confounding architectural vs information advantages. Action: evaluate a learning baseline with the same metric state and latency to isolate algorithmic benefits.
- Prompt ablations stop at bearings: It remains unknown whether partial metric state (e.g., relative pose with noise) helps if ingested as structured features rather than text. Action: compare text, vector, and hybrid interfaces with matched information.
- Failure-mode taxonomy: The paper identifies degradation qualitatively but lacks a fine-grained error taxonomy by task phase, scene factors, or policy internals. Action: produce phase-wise diagnostics (approach/alignment/descent/search) and correlate with attention/activation analyses.
- Foundation model variety: Limited set of aerial VLAs tested; effects of larger VLA backbones, control heads, tokenization, and decision rates are unknown. Action: benchmark diverse architectures and control frequencies under identical protocols.
- Urban complexity and traffic interaction: Although CARLA supports traffic, experiments do not stress-test dense traffic/pedestrians obstructing cooperation. Action: design scenarios with variable traffic density and measure robustness.
- ROS/bridge compatibility: Many real systems rely on ROS-like stacks; the single-process design avoids typical ROS jitter but may limit integration. Action: provide adapters that preserve low jitter and benchmark their timing impact.
- Reproducibility details: Build versions, GPU/CPU requirements, and cross-OS behavior can affect timing and results. Action: release deterministic builds, scenario seeds, and CI-tested configs across common platforms.
- Human-in-the-loop cooperation: Mixed-initiative teaming (human + VLA + UGV) and natural-language negotiation are not explored. Action: evaluate shared autonomy and handover strategies in the same tasks.
- Standardized training resources: No curated datasets for learning cooperative VLA policies. Action: release logged trajectories, partner-state annotations, and supervision for cooperation-aware training and benchmarking.
- Ethical and safety governance: While risks are noted, there is no protocol for evaluating compliance with geo-fencing, no-fly zones, or privacy constraints in cooperative tasks. Action: add constraint-violation metrics and safety monitors into the evaluation suite.
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that build directly on CARLA-Air’s single-process runtime, diagnostic tasks, and metrics. Each item includes likely sectors, potential tools/products/workflows, and feasibility notes.
Industry
- Closed-loop QA and regression testing for UAV–UGV products
- Sectors: robotics, autonomy software, logistics, inspection
- What: Use CARLA-Air to run nightly CI test suites that evaluate cooperative behaviors (e.g., moving-platform landing, occlusion-recovery escort) with built-in metrics (TSR, LSR, CCR, RSR, RAT, ECL, DF).
- Tools/workflows: “CoopTest” CI harness; scenario packs that vary occlusion types and traffic; latency profilers that track Effective Coordination Latency and sensor timestamp alignment.
- Assumptions/dependencies: Teams can interface via CARLA/AirSim Python APIs; compute resources (GPU) available; simulation fidelity sufficient for targeted failure modes.
- Latency budgeting and coordination tuning
- Sectors: robotics, edge/cloud networking, telecom
- What: Characterize how end-to-end latency affects cooperative stability (e.g., C2 oscillations), set per-module latency budgets, and tune coordination loops before field trials.
- Tools/workflows: ECL time-series dashboards; network-in-the-loop delay injectors; parameter sweeps for control gains and decision frequencies.
- Assumptions/dependencies: Accurate measurement in CARLA-Air; optional integration of realistic network delay models.
- Communication protocol prototyping for partner-state sharing
- Sectors: robotics, V2X, automotive
- What: Design and A/B test compact partner-state cues (semantic, numeric, bearing) to evaluate their effect on recovery and coordination performance using the provided tasks and prompt templates.
- Tools/workflows: “CueLab” prompt/cue designer; automated ablation runner (C1-Sem/C1-Num/C1-Oracle); confusion analysis between cue formats and action outputs.
- Assumptions/dependencies: Initial tests show text-based cues give limited gains; best used to diagnose requirements for richer interfaces.
- Model debugging for aerial VLA in multi-agent settings
- Sectors: autonomy software, AI tooling
- What: Use the platform to expose gaps where single-UAV tracking does not convert to cooperative success; analyze failure modes under C0/C1/C2 to guide redesign (e.g., adding partner-aware action grounding).
- Tools/workflows: Process-level trajectory analysis; CCR/CG-based dashboards; per-episode video + state overlays to detect oscillations, mis-grounded actions.
- Assumptions/dependencies: Access to model internals optional but beneficial; data export for offline analysis.
- Safety pre-certification sandbox
- Sectors: mobility, insurance, safety certification
- What: Run standardized cooperative scenarios to build safety evidence (e.g., stable moving-platform landing), quantify failure rates under perturbations, and prepare for regulatory reviews.
- Tools/workflows: Scenario catalogs; risk scoring tied to LSR/RSR and RAT; variance tracking to bound uncertainty (noise-floor validated by CARLA-Air’s zero timestamp mismatch).
- Assumptions/dependencies: Simulation evidence is a precursor, not a substitute, for field tests.
Academia
- Focused benchmark for cooperative robotics research
- Sectors: robotics research, ML, multi-agent systems
- What: Evaluate new algorithms for partner-state modeling, cooperative observation design, and team-level objective alignment with clean separation of perception-side vs action-side competencies.
- Tools/workflows: Plug-in policies via native CARLA/AirSim APIs; publish results with CCR/CG and ECL to fairly compare methods.
- Assumptions/dependencies: Community adoption; clear reporting of information asymmetries (e.g., Rule-Coop-State vs VLA).
- Education and lab courses on embodied cooperation
- Sectors: education
- What: Hands-on labs teaching closed-loop control, synchronization, and timing effects, highlighting the pitfalls of naïve bidirectional coupling.
- Tools/workflows: Starter notebooks; assignments to modify cues, change decision frequency, and measure coordination stability.
- Assumptions/dependencies: Student access to compatible hardware; instructor-provided scenarios.
Policy and Standards
- Drafting measurable, simulation-backed requirements for cooperative autonomy
- Sectors: regulators, standard bodies, municipalities
- What: Define metrics/thresholds for cooperative performance (e.g., maximum ECL, minimum LSR/RSR) and standardized test procedures using unified-clock simulation.
- Tools/workflows: Public test suites; conformance reports summarizing CG and CCR under set network/jitter conditions.
- Assumptions/dependencies: Simulation standards recognized as pre-compliance evidence; eventual field validation required.
Daily Life and Maker Communities
- Safe, low-cost prototyping for hobbyist robot teams
- Sectors: education/maker, competitions
- What: Use CARLA-Air to stage UAV–UGV challenges and learn about closed-loop cooperation before hardware builds.
- Tools/workflows: Preconfigured scenes; team performance boards based on CCR and RSR.
- Assumptions/dependencies: Enthusiasts comfortable with Unreal/CARLA/AirSim tooling; simulated results are for learning, not deployment guidance.
Long-Term Applications
These applications require further research and development, scaling, or new interfaces beyond text prompts, as the paper’s findings show partner-state grounding and low-latency coordination are essential.
Industry and Services
- Mobile handoff and landing for logistics
- Sectors: logistics, e-commerce, postal, campus operations
- What: UAVs landing on moving UGVs (trucks/carts) for parcel handoff, battery swap, or mobile charging.
- Products/workflows: “CoopDock” system combining partner-state grounding modules, cooperative NMPC, and robust localization.
- Assumptions/dependencies: High-precision relative pose sensing (visual markers, RTK, UWB); safety envelopes; non-text interfaces with structured metric state; regulatory approval.
- Cooperative inspection convoys
- Sectors: energy, utilities, construction, mining
- What: UAVs escort UGVs in large facilities, recovering from occlusions and guiding routes through complex layouts.
- Products/workflows: Multi-agent planners with team-level objectives; cooperative observation design to fuse aerial-ground view; alerting for occluded hazards.
- Assumptions/dependencies: Site maps and communications; robust partner-aware control; validated latency management.
- Emergency response and public safety teaming
- Sectors: public safety, defense, smart cities
- What: UAV–UGV teams for search-and-rescue, convoy overwatch, incident response with dynamic task-phase sharing (search, localize, escort).
- Products/workflows: “TeamOps” command software integrating mission phases, shared objectives, and role-aware actions; training simulators built atop CARLA-Air.
- Assumptions/dependencies: Reliable comms; rules of engagement and ethical safeguards; ruggedized systems; beyond-visual-line-of-sight approvals.
- Precision agriculture pairing
- Sectors: agriculture
- What: UAV scouting and UGV actuation (e.g., weeding, spraying) with on-the-fly occlusion recovery and cooperative routing.
- Products/workflows: Farm management stack with aerial-ground data fusion; cooperative autonomy tuned for low-latency rural networks.
- Assumptions/dependencies: Terrain-aware mapping; communications coverage; integration with farm machinery standards (ISOBUS).
Technology and Tools
- Partner-aware VLA and multi-agent policy stacks
- Sectors: AI software, robotics
- What: New VLA architectures that integrate explicit partner-state models, shared task-phase representations, and differentiable controllers for action grounding beyond text prompts.
- Products/workflows: “CoopPilot SDK” with APIs for structured partner state graphs, cooperative observation schemas, and low-latency control interfaces.
- Assumptions/dependencies: Training data for cooperation-aware fine-tuning; scalable multi-agent RL/self-play; interface standards for state exchange.
- Hardware-/network-in-the-loop (HIL/NIL) evaluation
- Sectors: telecom, edge compute, autonomy
- What: Test 5G/edge offload strategies and network slicing under realistic cooperative control loops to meet ECL budgets.
- Products/workflows: CARLA-Air + network emulator; automatic failover strategies when ECL thresholds are breached.
- Assumptions/dependencies: Accurate network models; integration with real radios and autopilots (PX4/ROS2).
- Standardization and certification frameworks
- Sectors: regulators, industry consortia
- What: Formalize closed-loop cooperative metrics (CCR, CG, ECL) and scenario protocols as certification criteria for UAV–UGV teams.
- Products/workflows: Open reference tests; certification toolkits; auditor dashboards linking metrics to risk profiles.
- Assumptions/dependencies: Stakeholder consensus; correlation studies from sim to field; governance for updates.
Research and Education
- Cooperative dataset generation and curriculum learning
- Sectors: academia, applied research
- What: Use CARLA-Air to create datasets emphasizing state grounding and team objectives; develop curricula that scaffold single-agent skills toward cooperation.
- Products/workflows: Public datasets for occlusion/landing with synchronized multi-modal logs; benchmarks that factor timing and partner state.
- Assumptions/dependencies: Community curation; standardized annotation for partner-specific variables.
- Smart city planning and infrastructure testing
- Sectors: urban planning, mobility
- What: Simulate uses of shared landing pads, rooftop hubs, and UGV corridors while evaluating latency-sensitive coordination at scale.
- Products/workflows: Scenario forecasting tools; placement optimization for comms and infrastructure using CARLA-Air as the engine.
- Assumptions/dependencies: City data; realistic traffic models; stakeholder engagement.
Notes on Feasibility and Dependencies
- Simulation-to-real gap: Current results are in simulation; real-world deployment will require field validation and safety cases.
- Interface limitations: Text-prompt cues showed limited benefits; practical systems likely need structured metric state interfaces, shared maps/graphs, and tight control integration.
- Latency sensitivity: Low-latency coordination is critical; network reliability and computation pipelines must meet ECL budgets.
- Safety and regulation: Moving-platform landings and urban escort tasks are safety-critical and subject to evolving regulations.
- Hardware/software stack: Requires Unreal Engine, CARLA, AirSim compatibility; GPU resources for high-fidelity rendering and multiple agents.
- Information asymmetry: Rule-based references used explicit state; deploying comparable performance demands improved sensing, fusion, and action grounding in learned models.
In summary, CARLA-Air immediately supports robust, closed-loop evaluation and engineering of cooperative autonomy, while the paper’s findings chart a clear R&D path toward deployable UAV–UGV teaming: explicit partner-state grounding, low-latency action coordination, and team-objective alignment.
Glossary
- Actor-level aerial subsystem: An Unreal Engine design where aerial dynamics are implemented as a world actor rather than the global controller. "instantiates AirSim as an actor-level aerial subsystem in the same simulation world."
- Autoregressive text-prompt interface: A control interface where actions are generated sequentially conditioned on textual prompts and prior outputs. "within the autoregressive text-prompt interface itself."
- Bidirectional interaction: Two-way coupling where UAV and UGV exchange information or influence each other’s actions. "naive bidirectional interaction fails to consistently improve performance and can amplify errors for most baselines."
- CARLA: An open-source urban driving simulator providing traffic, vehicles, and sensors. "CARLA-Air preserves CARLA's urban traffic simulation and AirSim's multirotor dynamics"
- CARLA-Air: A single-process environment unifying CARLA and AirSim for closed-loop air-ground evaluation. "we introduce CARLA-Air~\cite{zeng2026carla}, a single-process air-ground evaluation environment"
- Closed-loop: An interactive setting where actions immediately affect the environment and subsequent observations. "respond to each other in a closed-loop physical world"
- Cooperative Conversion Rate (CCR): A metric measuring how well tracking success converts into final cooperative success. "We define Cooperative Conversion Rate as with "
- Cooperation Gain (CG): The improvement (or drop) in cooperative success relative to a no-communication baseline. "and Cooperation Gain as ."
- Decision Frequency (DF): The rate at which a policy outputs control decisions. "We also report Decision Frequency (DF) and Effective Coordination Latency (ECL)."
- Effective Coordination Latency (ECL): End-to-end delay between receiving partner information and producing coordinated action. "We also report Decision Frequency (DF) and Effective Coordination Latency (ECL)."
- GameMode: The Unreal Engine world-lifecycle controller responsible for authoritative simulation control. "CARLAAirGameMode inherits CARLA's native GameMode"
- Landing Success Rate (LSR): The fraction of episodes where the UAV achieves a successful landing in the cooperative task. "For landing, we report Tracking Success Rate (TSR) as the single-UAV primitive score and Landing Success Rate (LSR) as the final cooperative task score."
- Longitudinal controller: The vehicle controller regulating forward velocity (speed along the direction of travel). "passes the magnitude of the UAV's forward-velocity action signal to the UGV's longitudinal controller,"
- Metric state: Explicit numerical state (e.g., poses, velocities) used for precise control or coordination. "uses explicit metric state---UAV--cargo-bed relative pose, relative velocity, UGV speed, and landing phase---"
- Moving-Platform Landing: A diagnostic task requiring a UAV to land on a moving UGV using coordination. "Cooperative Moving-Platform Landing tests whether visible tracking of a ground partner can become coordinated action."
- Multirotor dynamics: The physics governing drones with multiple rotors, including forces and torques. "AirSim's multirotor dynamics"
- Occlusion-Recovery Escort: A diagnostic task where the UAV must re-acquire a temporarily occluded UGV using partner cues. "Cooperative Occlusion-Recovery Escort tests whether partner-state cues can help recover visual contact after temporary occlusion."
- Physics tick: The discrete simulation timestep at which physics and sensors are advanced. "world state, physics tick, and sensing pipeline"
- ROS bridge: A middleware linkage connecting separate simulators via ROS processes. "via a ROS bridge, introducing timing jitter that corrupts latency measurement."
- RPC (Remote Procedure Call): A client-server mechanism for sending commands and retrieving data from simulators. "retain their original RPC servers and client protocols,"
- Rule-Coop-State: A state-based, rule-driven cooperative reference policy used as a solvability baseline. "Rule-Coop-State reaches LSR $0.42$"
- Sensing pipeline: The sequence that schedules and generates sensor data from the world state. "world state, physics tick, and sensing pipeline"
- Sign test: A nonparametric statistical test for paired comparisons. "( vs.\ $0.031$, sign test)."
- Simulation-timestamp alignment: Ensuring that sensor data from different agents are temporally matched at generation. "precise measurement of simulation-timestamp alignment and effective coordination latency."
- Single-authority constraint: The requirement that only one controller manages the simulation world’s lifecycle. "Resolving the single-authority constraint."
- Single-process: Running multiple simulation components within one OS process and shared runtime. "a single-process air-ground evaluation environment"
- Synchronous rendering: Rendering tied to the simulation tick to keep visuals aligned with physics. "synchronous rendering"
- Timing jitter: Variability in timing that introduces noise into latency and synchronization. "introducing timing jitter that corrupts latency measurement."
- Traffic manager: The simulator module controlling dynamic vehicle traffic behaviors. "its traffic manager, weather system, episode management, sensor scheduling, and RPC services"
- Tracking Success Rate (TSR): The fraction of episodes where the UAV successfully tracks the target (single-agent primitive). "For landing, we report Tracking Success Rate (TSR) as the single-UAV primitive score and Landing Success Rate (LSR) as the final cooperative task score."
- Unreal Engine runtime: The execution environment of the Unreal Engine that hosts the simulation world. "inside one Unreal Engine runtime."
- Vision-and-Language Navigation (VLN): Policies that navigate using natural language and visual input. "traditional aerial VLN."
- Vision-Language-Action (VLA): Models that integrate perception, language understanding, and action generation. "Recent aerial vision-language-action (VLA) models provide a tempting starting point."
- Zero-shot: Evaluation without task-specific fine-tuning or additional training for the target tasks. "Under zero-shot evaluation and tested text-based cue protocols, current aerial VLA models do not naturally transfer single-UAV competence to air-ground cooperation:"
Collections
Sign up for free to add this paper to one or more collections.