LocateAnything-Data: Multi-Domain Localization

• LocateAnything-Data is a multi-domain collection of data assets that use language-conditioned localization for 3D referential grounding, open-vocabulary detection, and unified vision-language tasks.
• The Locate 3D Dataset (L3DD) employs self-supervised learning to convert posed RGB-D streams into detailed 3D masks, anchors, and bounding boxes with robust annotations.
• LAE-1M and the 138M-sample corpus leverage automated, multi-stage curation to enhance detection accuracy and throughput across remote sensing and generic visual grounding challenges.

Searching arXiv for the cited papers to ground the article in the corresponding preprints. Searching arXiv for ([2504.14151](/papers/2504.14151)) Locate 3D LocateAnything-Data is a label that has been used for three distinct data assets in recent vision-language localization research: the Locate 3D Dataset (L3DD) for 3D referential grounding in indoor RGB-D scenes, the LAE-1M and LAE-80C resources for open-vocabulary object detection in remote sensing, and a large-scale multi-domain corpus for parallel box and point decoding in unified visual grounding and detection. Across these usages, the common organizing idea is a data engine that converts heterogeneous sensory or image sources into language-conditioned geometric supervision, but the target geometries, modalities, evaluation protocols, and intended deployment settings differ substantially (Arnaud et al., 19 Apr 2025, Pan et al., 2024, Wang et al., 26 May 2026).

1. Nomenclature and scope

In the literature, the same label denotes three separate dataset-and-pipeline constructs rather than a single canonical benchmark. One usage is sensor-centric and 3D, one is remote-sensing-centric and open-vocabulary, and one is a multi-domain web-scale corpus designed for generative grounding with Parallel Box Decoding. This naming overlap matters because the associated tasks are not interchangeable: 3D mask grounding in posed RGB-D streams, horizontal-box detection in aerial imagery, and block-structured box/point generation in generic 2D vision-language grounding are technically different problem formulations (Arnaud et al., 19 Apr 2025, Pan et al., 2024, Wang et al., 26 May 2026).

Usage of the name	Primary data asset	Core task
Locate 3D	L3DD	3D referential grounding
Locate Anything on Earth	LAE-1M, LAE-80C	Remote-sensing open-vocabulary detection
LocateAnything	138M-sample corpus	Unified grounding and detection with PBD

This suggests that “LocateAnything-Data” is best understood as a family resemblance across projects rather than a single standardized dataset brand. The family resemblance lies in language-conditioned localization, large-scale data curation, and explicit geometry targets.

2. Locate 3D Dataset (L3DD)

In "Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D" (Arnaud et al., 19 Apr 2025), LocateAnything-Data refers to L3DD and the accompanying pipeline that turns posed RGB-D streams into 3D, language-groundable supervision and features. L3DD is a large-scale, mask-grounded 3D referring expressions dataset spanning three indoor scene sources with different capture setups and sensors: ScanNet, ScanNet++ v1, and ARKitScenes. Together, it covers 1,346 venues and 131,641 language annotations across 23,549 unique objects. Each scene includes posed RGB-D streams and 3D point clouds, plus human-generated referring expressions with target masks and grounded anchors, where grounded anchors are noun phrases linked to other objects that appear in the expression.

The point-cloud representation is constructed by unprojecting RGB-D into a point cloud, voxelizing at 5 cm resolution, and lifting dense 2D foundation features into 3D by aggregating features per voxel via trilinear interpolation weights. The 2D feature stack comprises DINOv2 dense patch features, CLIP features computed per 2D instance mask with masks from SAM, and harmonic embedding of RGB pixel intensities. These are concatenated and lifted to 3D, producing a featurized point set

$PtC_{lift}=\{(x_i,y_i,z_i,f_i)\},$

with $f_i \in \mathbb{R}^d$. This featurized point cloud serves as the input for both pretraining and downstream localization.

The annotation taxonomy is unusually rich for 3D grounding. L3DD provides 3D target masks, grounded anchors, and free-form referring expressions. The average expression length is approximately 7.55 words, vocabulary size is approximately 3,058, the average number of grounded instances per query is approximately 2.28, color references appear in 42.6% of samples, and shape references in 38.22% of samples, computed on 5,000 samples using Llama 3.1-8B. Common spatial relations include “on,” “near,” “between,” “left/right of,” “in front of/behind,” and room-level locality. Validation quality control required that all validation split samples be validated by at least one human; over 80% of ARKitScenes and ScanNet++ validation samples were validated three or more times and retained only if a majority judged the sample unambiguously correct.

Methodologically, L3DD is tied to a specific learning pipeline. The encoder is PTv3, pretrained by 3D-JEPA on lifted feature clouds. The self-supervised objective uses masked prediction in latent space,

$$\min_{\theta,\phi}\left\|P_\phi(E_\theta(\tilde{PtC}),M)-sg(\bar{E}_\theta(PtC))\right\|,$$

with a per-point loss averaged over masked points. Serialized percent masking performs best, using PTv3’s bijective space-filling curve to define contiguous masked blocks. After pretraining, a language-conditioned decoder predicts 3D masks and 3D bounding boxes, with Hungarian matching for query-object assignment and supervision in sensor space via k-NN transfer with $k=8$.

3. LAE-1M and LAE-80C in remote sensing

In "Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community" (Pan et al., 2024), LocateAnything-Data refers to the data assets and tooling introduced for the Locate Anything on Earth (LAE) task. The central resource is LAE-1M, described as the first large-scale remote sensing object detection dataset with broad category coverage, and it is paired with the LAE-80C benchmark for open-vocabulary evaluation. The task reformulates open-vocabulary object detection for remote sensing, motivated by the “huge data domain gap” between natural-world imagery and remote-sensing imagery.

LAE-1M combines two subsets produced by the LAE-Label Engine. LAE-FOD is built from existing labeled datasets through image slicing, format alignment, and sampling. LAE-COD is built from unlabeled data by semi-automatic annotation using SAM and a Large Vision-LLM, specifically InternVL, together with rule-based filtering. The resulting corpus covers one million instances and yields around 1600 vocabularies. Constituent datasets include DOTA, DIOR, FAIR1M, NWPU VHR-10, RSOD, xView, HRSC2016, Condensing-Tower, AID, NWPU-RESISC45, SLM, and EMS (From Google Earth). For most datasets, a 0.4 random sampling rate is adopted if the number of instances of the same class is larger than 100, while xView uses 0.2 sampling to eliminate duplicate instances.

The model that consumes this data is LAE-DINO, which introduces Dynamic Vocabulary Construction (DVC) and Visual-Guided Text Prompt Learning (VisGT). DVC addresses the large unified vocabulary by selecting positive and negative vocabularies per iteration, with $N_{\mathcal{DV}}=60$ and alignment heads set to 1600 categories for open-vocabulary pre-training. VisGT enhances text features using visual scene information through Multi-scale Deformable Self-Attention. The detector uses a DINO-style image-text architecture with Swin-Transformer for images and BERT for text, and the training loss is

$$\mathcal{L}=\mathcal{L}_{cls}+\alpha \mathcal{L}_{loc}+\beta \mathcal{L}_{VisGT},$$

with $\alpha=1$ and $\beta=10$ in the reported experiments.

Evaluation is performed on DIOR, DOTAv2.0, LAE-80C, and HRRSD. Metrics include mAP, AP50, and AP75, with DOTAv2.0 results reported using horizontal detection boxes. The paper explicitly notes that oriented detection is not addressed in the reported results. This is a significant design choice because several aerial benchmarks are frequently associated with oriented annotations, whereas the LAE formulation standardizes the detector and evaluations to axis-aligned boxes.

4. The 138M-sample corpus for Parallel Box Decoding

In "LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding" (Wang et al., 26 May 2026), LocateAnything-Data denotes a large-scale, multi-domain corpus curated for a unified vision-language grounding and detection model. Its reported scale is more than 138 million training samples, 12 million unique images, and 785 million annotated bounding boxes. A sample is defined as an image-text-geometry tuple consisting of an input image, a natural-language query, and one or more geometric targets, with explicitly negative queries tied to images with “no target” labels.

The corpus is heterogeneous by design. General object detection contributes 66.9% of queries and 83.1% of the box supervision; GUI grounding contributes 16.5%; referring expression comprehension 7.3%; text localization/OCR detection 3.6%; document or scene layout grounding 3.5%; and point-based localization 2.2%. Data sources include Open Images v4, Objects365, COCO-derived referring corpora such as Flickr30k Entities, gRefCOCO, RefCOCO, and HumanPart, HumanRef, multiple public GUI corpora, unlabeled Unsplash and SA-1B images, and synthetic query generation.

The curation engine is explicitly multi-stage. For labeled detection data, Qwen3-VL generates multiple object-centric queries per box, and Molmo predicts candidate points, retaining only points inside the corresponding ground-truth box. For unlabeled images, Qwen3-VL generates queries; Molmo points may be converted to boxes via SAM 3, or Rex-Omni may be prompted to directly generate boxes; Qwen3-VL then filters inconsistent predictions. Negative samples are created for each domain and assigned a Negative block to train abstention.

The annotation format is adapted to Parallel Box Decoding. Continuous coordinates are normalized to $[0,1000]$, discretized into tokens, and organized into fixed-length blocks with $L=6$. The Box Block is <box> x1 y1 x2 y2 </box>, alongside Semantic Block, Negative Block, End Block, and <null> padding. Training uses a dual formulation: a standard causal Next-Token Prediction sequence and a block-wise Masked-Token Prediction sequence. Attention is causal for shared context and NTP streams, block-causal across blocks, and bidirectional within blocks. The objective is purely token-level,

$f_i \in \mathbb{R}^d$0

and the paper does not report $f_i \in \mathbb{R}^d$1, GIoU, or DIoU regression losses.

5. Data-engine design and supervision paradigms

The three usages of LocateAnything-Data exemplify distinct supervision paradigms. L3DD is built around human annotation over posed RGB-D scenes. A 2D video labeling UI projects 3D instance masks into 2D for user selection; annotators provide target descriptions and align all nouns in the description to corresponding masks; ARKitScenes uses SAMPro3D to produce 3D instance masks, and annotators may merge multiple mask fragments into one object. Sensor-space supervision transfers mesh annotations to sensor point clouds via k-NN and is preferred because it better matches real-world noise and partial observations (Arnaud et al., 19 Apr 2025).

LAE-1M combines curated labeled corpora and semi-automatic coarse labeling. Its LAE-Label Engine addresses inconsistent formats and scarce well-annotated large-scale image-text pairs by unifying labeled sources and generating coarse labels from unlabeled remote-sensing imagery with SAM, InternVL, and rule-based filtering. Vocabulary harmonization is central: categories across all sources are merged into a single vocabulary set, yielding positive and negative vocabularies per batch for open-vocabulary detection (Pan et al., 2024).

The 2026 multi-domain corpus pushes automation further. Query synthesis, point generation, box derivation, negative construction, balancing, densification, and post-verification are all integrated into a multi-target grounding engine. Stage-4 training explicitly raises the proportion of many-object images such as MOT20Det and SKU110K-like scenes while reducing general data to 20%, which is intended to improve dense recall and precision. This suggests a shift from dataset curation as label collection toward dataset curation as large-scale geometry-and-language synthesis under model-in-the-loop verification (Wang et al., 26 May 2026).

Across the three systems, the supervision targets also differ materially. L3DD emphasizes 3D masks, grounded anchors, and language-conditioned 3D boxes. LAE-1M emphasizes horizontal 2D detection boxes with unified vocabularies. The 138M corpus uses a shared tokenized geometry schema for both boxes and points, plus explicit negative blocks. These differences are not merely notational; they shape the loss functions, inference procedures, and admissible benchmarks.

6. Benchmarks, performance, and limitations

The empirical role of these data assets is strongest when examined through their native metrics. For L3DD, standard metrics are top-1 accuracy at IoU thresholds for predicted masks and boxes, specifically Acc@25 and Acc@50. Under sensor point clouds, Locate 3D trained on SR3D, NR3D, and ScanRefer reports Joint Evaluation of 61.7 and 49.4, while Locate 3D+ trained on benchmarks plus L3DD reports 63.7 and 51.3. In cross-setup generalization when trained only on ScanNet, SN++ Acc@25 increases from 51.5 for CF random init to 56.7 for Locate 3D, and ARKitScenes Acc@25 increases from 41.7 to 46.2; adding L3DD training data raises these to 83.9 on SN++ and 57.6 on ARKitScenes. The ablations further tie data design to performance: mask-only gives 55.4% Acc@25, mask+box gives 61.7%, and box-only fails at approximately 0.3% (Arnaud et al., 19 Apr 2025).

For LAE-1M, the main reported metrics are AP50, AP75, and mAP on remote-sensing benchmarks. In open-set detection, LAE-DINO pretrained on LAE-1M reports DIOR AP50 of 85.5, DOTAv2.0 mAP of 46.8, and LAE-80C mAP of 20.2. LAE-DINO with LAE-FOD only reports 84.1, 44.5, and 19.1 respectively, so adding LAE-COD yields gains of +1.4, +2.3, and +1.1. In closed-set fine-tuning, LAE-DINO-FT pretrained on LAE-1M reaches DIOR AP50 92.2 and DOTAv2.0 mAP 57.9. Few-shot HRRSD results also improve, with mAP_all of 32.1, 35.9, and 42.4 for 1-shot, 5-shot, and 10-shot settings (Pan et al., 2024).

For the 138M-sample LocateAnything corpus, performance is measured by F1-score at multiple IoU thresholds and by throughput in Boxes Per Second. LocateAnything-3B in Hybrid mode reports LVIS mean F1 of 50.7, COCO mean F1 of 54.7, Dense200 mean F1 of 58.7, HumanRef mean F1 of 78.7, DocLayNet mean F1 of 76.8, and ScreenSpot-Pro average Acc of 60.3. Throughput on a single NVIDIA H100, batch size 1, is reported as 12.7 BPS for LocateAnything-3B in Hybrid mode, compared with 5.0 BPS for Rex-Omni-3B and approximately 1.0–1.3 BPS for textual-token VLMs. As the number of target boxes grows from 20 to 300, PBD throughput rises from approximately 12 to approximately 25 BPS (Wang et al., 26 May 2026).

The limitations are correspondingly domain-specific. L3DD does not target dynamic scenes, relies on static or quasi-static caching, and notes limited room-level spatial language coverage and mask noise in ARKitScenes. LAE-1M does not specify final format details or licensing terms in the paper and restricts reported results to horizontal boxes. The 138M corpus does not state public release, license, or download details, and potential biases in open-source and web images and synthetic queries are not explicitly studied. A plausible implication is that the three systems collectively demonstrate the utility of data engines for localization, but they also expose an unresolved dependency on upstream dataset licensing, annotation noise control, and benchmark-specific geometry conventions.