GoalSwarm: Multi-UAV Semantic Coordination for Open-Vocabulary Object Navigation

Abstract: Cooperative visual semantic navigation is a foundational capability for aerial robot teams operating in unknown environments. However, achieving robust open-vocabulary object-goal navigation remains challenging due to the computational constraints of deploying heavy perception models onboard and the complexity of decentralized multi-agent coordination. We present GoalSwarm, a fully decentralized multi-UAV framework for zero-shot semantic object-goal navigation. Each UAV collaboratively constructs a shared, lightweight 2D top-down semantic occupancy map by projecting depth observations from aerial vantage points, eliminating the computational burden of full 3D representations while preserving essential geometric and semantic structure. The core contributions of GoalSwarm are threefold: (1) integration of zero-shot foundation model -- SAM3 for open vocabulary detection and pixel-level segmentation, enabling open-vocabulary target identification without task-specific training; (2) a Bayesian Value Map that fuses multi-viewpoint detection confidences into a per-pixel goal-relevance distribution, enabling informed frontier scoring via Upper Confidence Bound (UCB) exploration; and (3) a decentralized coordination strategy combining semantic frontier extraction, cost-utility bidding with geodesic path costs, and spatial separation penalties to minimize redundant exploration across the swarm.

• The paper introduces a decentralized multi-UAV ObjectNav system that combines SAM3 zero-shot grounding, Bayesian Value Maps, and UCB-based frontier bidding over lightweight shared 2D semantic maps.
• GoalSwarm achieves 45.0% success and 0.179 SPL on GOAT-Bench val_unseen, outperforming single-agent, random-exploration, and no-map-sharing baselines, while extended tests reach 53.8% success and 0.195 SPL.
• The results show that semantic map sharing primarily improves path efficiency by reducing redundant exploration, but real-world deployment still requires handling odometry drift, communication limits, 2D projection errors, and onboard perception costs.

GoalSwarm addresses open-vocabulary object-goal navigation (ObjectNav) for teams of UAVs operating in unknown indoor environments. The framework is fully decentralized: each UAV runs its own perception, mapping, and planning loop, periodically synchronizing a lightweight 2D semantic occupancy map with teammates rather than maintaining a full 3D reconstruction. The design targets two persistent bottlenecks in aerial ObjectNav — the onboard compute cost of heavy perception models and the coordination complexity of multi-agent exploration — by combining zero-shot foundation-model grounding (SAM3) with a Bayesian Value Map (BVM) and a cost-utility bidding scheme for frontier assignment.

System architecture

Each UAV executes a five-step loop at every timestep: acquire an RGB-D frame and odometry pose; run zero-shot grounding to detect the target and produce a confidence score; project depth into a shared 2D top-down semantic occupancy grid and update the BVM; extract frontiers from the fused map and score them via Upper Confidence Bound (UCB); and navigate reactively using depth-based obstacle avoidance with PID-controlled altitude management.

The semantic map is a $480 \times 480$ grid at 5 cm resolution covering $24\,\text{m} \times 24\,\text{m}$, with $K = 16$ semantic channels plus occupancy and explored-area channels. Depth is unprojected to a camera-frame point cloud, transformed to a geocentric frame via the camera elevation rotation and sensor height translation, voxelized, and collapsed along the height axis within an obstacle band of roughly 25 cm to agent height + 50 cm. Local maps are fused into the global map by element-wise max after affine transformation by cumulative pose; synchronization occurs every 25 timesteps. The uncompressed per-synchronization payload is approximately 6.6 MB, which the authors note can be reduced via delta encoding.

Zero-shot grounding and the Bayesian Value Map

Target identification uses SAM3 as a single model for open-vocabulary detection, segmentation, and tracking, prompted directly with the target category name (e.g., "red chair"). The aggregated detection confidence multiplies detector confidence by relative mask area, down-weighting spurious small detections. A confidence gate ($\tau = 0.3$) suppresses ghost detections, and multi-view confirmation requiring two consecutive detections within a sliding window guards against transient false positives. Notably, SAM3 is queried every three simulation steps on an external GPU server over ZMQ — so the "decentralized" claim applies to decision-making, not perception compute, which is offloaded.

The BVM stores a per-pixel mean $\mu$ and variance $\sigma^2$ of goal relevance, updated by standard precision-weighted Bayesian fusion. Observation variance derives from a depth-dependent confidence cone that attenuates with angular offset from the optical axis, floored at $c_{min} = 0.25$. This construction lets frontier scoring exploit uncertainty directly: frontiers are scored by

$$U(f_i) = \tilde{\mu}(f_i) + \beta \sqrt{\max(0,\ \tilde{\sigma}^2(f_i))}$$

with median statistics over a circular footprint and $\beta = 1.7$, biasing agents toward regions where detection evidence is ambiguous rather than merely toward the nearest or largest frontier.

Decentralized coordination

Frontier assignment is formulated as distributed cost-utility optimization. Each UAV scores candidate frontiers as a weighted combination of UCB value, geodesic path cost computed via Fast Marching Method on the occupancy grid, frontier size, minus a spatial separation penalty that activates when another UAV is closer than $d_{min}$ to the frontier. Conflicts are resolved asynchronously by priority favoring the highest-scoring UAV. Once the BVM mean near a location exceeds a goal threshold, the nearest UAV switches from exploration to direct pursuit. Navigation uses a reactive priority cascade over five depth sectors with a 1.0 m safety threshold, a stuck-detection escape maneuver, and phase-dependent cruise altitudes (3.0 m survey, 2.0 m room search, 1.5 m inspection).

Experimental results

Evaluation uses GOAT-Bench val_unseen object-type subtasks in photorealistic HM3D scenes within Habitat, with VisFly providing quadrotor physics. Agents are compact indoor platforms with a front-facing RGB-D camera (42° HFOV, 0.5–5.0 m depth range), six discrete actions, and a 500-step budget per subtask; success requires STOP within 1.0 m of any annotated target viewpoint. All experiments ran on a single RTX 4090.

Method	SR (%)	SPL	Avg. Steps
Single Agent	10.0	0.078	452.5
Random Exploration	20.0	0.084	413.0
No Shared Map	40.0	0.130	149.5
GoalSwarm	45.0	0.179	160.2

Three findings stand out. First, the single-agent baseline's 10.0% SR versus GoalSwarm's 45.0% indicates that doubling the fleet roughly quadruples success under this protocol, which the authors attribute to faster coverage of hard-to-reach targets. Second, the No Shared Map ablation achieves a competitive 40.0% SR but the lowest SPL among multi-agent variants (0.130); the authors diagnose this as redundant room revisits and premature false stops, since neither agent benefits from the other's observations. This is an instructive asymmetry: map sharing improves path efficiency more than raw success. Third, random exploration (20.0% SR) confirms that semantic frontier scoring, not merely additional agents, drives performance. An extended 100-episode evaluation reports SR = 53.8% and SPL = 0.195, suggesting the 20-episode figures are not artifacts of a small sample, though no variance or significance statistics are reported for either set.

Limitations and open questions

The paper concedes several constraints explicitly. The framework assumes perfect odometry and reliable communication; drift, packet loss, and bandwidth limits are unaddressed, and the external GPU server for SAM3 means the system is not deployable within typical onboard compute envelopes. The 2D top-down projection merges vertically separated structures such as tables and shelves into a single plane, which may corrupt both occupancy reasoning and goal projection. Three failure modes are documented: ghost detections leaving residual semantic pixels that mislead goal projection; unreachable goal projections where the FMM planner cannot route within the 1.0 m threshold (e.g., objects behind glass or elevated); and weak detection signals for categories like mirror and statue that fall below the confidence gate, exhausting the step budget. Open questions include whether the BVM/UCB gains persist under realistic sensing noise, whether delta encoding suffices for bandwidth-constrained synchronization, and how the bidding scheme scales beyond two agents.

Conclusion

GoalSwarm demonstrates that a decentralized multi-UAV pipeline built on lightweight 2D semantic mapping, SAM3-based zero-shot grounding, Bayesian fusion of multi-viewpoint detection confidence, and UCB-scored cost-utility bidding outperforms single-agent and uncoordinated baselines on GOAT-Bench val_unseen, reaching 45.0% SR and 0.179 SPL in the primary evaluation and 53.8% / 0.195 at 100 episodes. Its principal contribution is showing that uncertainty-aware spatial reasoning over shared low-dimensional maps can substitute for computationally expensive 3D representations in aerial ObjectNav, though validation remains confined to simulation with idealized odometry and offloaded perception.