3EED: 3D Visual Grounding Benchmark

• 3EED is a benchmark and toolkit that enables language-driven 3D object localization across vehicle, drone, and quadruped platforms using synchronized RGB and LiDAR data.
• It employs cross-platform alignment, multi-scale sampling, and scale-aware fusion to mitigate domain gaps and enhance spatial grounding accuracy.
• The dataset comprises over 20,000 outdoor scenes with extensive annotations, setting a new standard for embodied perception and multimodal scene understanding.

3EED refers to “Ground Everything Everywhere in 3D,” a large-scale multi-platform multimodal benchmark and toolkit for 3D visual grounding in real-world outdoor environments. Its principal objective is to establish a standard, challenging evaluation setting for language-driven 3D object localization across vehicle, drone, and quadruped robotic platforms using synchronized RGB and LiDAR sensing (Li et al., 3 Nov 2025). The following entry surveys the dataset’s design, benchmark protocols, technical contributions, and early empirical findings, linking them to ongoing challenges in scalable embodied perception.

1. Concept and Scope

3EED is a benchmark for visual grounding: given a 3D scene (point cloud and aligned RGB images) and a natural language referring expression, the task is to localize the correct object(s) in 3D space. Unlike prior benchmarks restricted to indoor settings or single robotic platforms, 3EED spans three distinct robot embodiments—vehicle (street-level), drone (aerial), and quadruped (ground-level). The dataset contains over 20,000 scenes, 128,000 unique annotated objects, and 22,000 human-validated referring expressions, making it an order of magnitude larger than prior outdoor datasets. Scene coverage includes diverse settings such as roads, intersections, parking lots, and crowded pedestrian zones, with spatial extents up to $280\,\mathrm{m}\times 240\,\mathrm{m}$ and vertical variation exceeding 80 meters.

2. Dataset Design and Construction

Multiplatform Data Acquisition

RGB and LiDAR observations are collected from:

• Vehicle platform: Street-level drive-by data.
• Drone platform: Top-down aerial sequences with wide coverage.
• Quadruped platform: Ground-level, low-angle viewpoints capturing close pedestrian and environmental features.

Each platform samples distinct spatial and geometric priors, resulting in wide variability in point cloud density, field of view, and spatial occlusion.

Annotation Pipeline

3D Bounding Box Annotation

• Vehicle scenes: Use high-quality 3D object annotations from the Waymo Open Dataset.
• Drone/quadruped scenes: Pseudo-labeling is performed by ensemble 3D detectors (PV-RCNN, Voxel-RCNN, CenterPoint, etc.), followed by kernel density estimation for consolidation, 3D multi-object tracking for temporal consistency, and semantic verification against image projections. Human annotators then refine ambiguous or inconsistent predictions.

Referring Expression Annotation

• Generation: Qwen2-VL-72B, a vision-LLM, is prompted using templates incorporating object category, spatial relations, egocentric context, and temporal states.
• Normalization: Expressions are re-written using platform-invariant rules to standardize linguistic style across robot types.
• Human Verification: Five trained annotators review and refine each referring expression, ensuring semantic accuracy and referential uniqueness.

Dataset Scale

Platform	Scenes	Objects	Expressions
Vehicle	5,409	25,818	—
Drone	7,098	57,138	—
Quadruped	7,860	45,797	—
Total	20,367	128,735	22,439

Objects and expressions are balanced across platforms and cover a broad spectrum of outdoor geometries and density regimes.

3. Benchmark Protocols and Metrics

Benchmark tasks are stratified into four principal settings:

1. Single-platform, single-object grounding: Training and testing on the same platform for baseline in-domain evaluation.
2. Cross-platform transfer: Training on one platform (typically vehicle) and testing on another (drone/quadruped) to evaluate generalization.
3. Multi-object grounding: Expressions refer jointly to multiple objects; localization requires all referred objects to be correctly identified.
4. Multi-platform pooling: Jointly training on all platforms and evaluating on each, assessing transferability and unification.

Standard 3D detection metrics are used: $\text{Acc@}\delta = \frac{1}{N} \sum_{i=1}^N \mathbbm{1}\big[\mathrm{IoU}(\hat{b}_i, b_i^{\mathrm{gt}}) > \delta\big]$ with thresholds $\delta \in \{0.25, 0.50\}$. For multi-object expressions, all referenced objects must satisfy the IoU threshold.

4. Technical Contributions for Cross-Platform Grounding

To address the platform-induced domain gap, the reference method combines:

• Cross-Platform Alignment (CPA): Rotational normalization of point clouds to gravity alignment (z-axis), with altitude offsets for drones. This standardizes egocentric and allocentric spatial language ("above", "behind") across robots.
• Multi-Scale Sampling (MSS): Extension of PointNet++ with variable neighborhood radii (from ~0.6m to ~4.8m), accommodating sparse/dense object distributions and range variations.
• Scale-Aware Fusion (SAF): Fusion of features at multiple spatial scales, using an MLP to predict dynamic per-point scale weights.
• Contrastive Loss and DETR-like Hungarian Matching: Language-vision objectives align language and spatial features; detection and grounding are trained jointly.

These strategies are empirically shown to reduce the cross-platform performance gap compared to indoor and single-platform baselines.

5. Empirical Findings and Observed Challenges

3EED reveals substantial in-domain and transfer gaps:

• In-domain vehicle performance: Acc@25 of 78.37 (proposed) versus 53.54 (best baseline).
• Cross-platform transfer: Dramatic drop when evaluating vehicle-trained models on drone (Acc@25 = 1.54 for baseline; +13.93 for proposed) and quadruped (Acc@25 = 10.18 baseline; +25.86 proposed).
• Multi-platform: Proposed model achieves balanced, improved transfer (vehicle: 80.86/50.11, drone: 53.45/9.75, quadruped: 53.31/24.08 at Acc@25/50).
• Multi-object grounding: Proposed model achieves a substantial lead (Acc@25 = 32.32, mIoU = 56.40) over strong baselines.

Challenges are pronounced for drone data due to high scene clutter, sparse LiDAR returns, angular variation, and unique top-down perspectives. Overall, generalizable cross-platform 3D grounding remains unsolved; performance degrades as point density falls, viewpoint extremity grows, and description complexity increases.

6. Toolkit Availability and Impact

3EED’s dataset, annotation tools, and benchmark code are released under CC BY-SA 4.0 to encourage further research. The platform establishes a new standard for evaluating spatial language understanding in embodied agents operating outdoors, enabling rigorous benchmarking for applications in robotics, autonomous driving, and complex scene understanding. The benchmark demonstrates that achieving robust language-driven 3D perception across platforms and modalities is an open problem needing advances in multimodal fusion, spatial reasoning, and domain adaptation (Li et al., 3 Nov 2025).