LocateAnything: Query-Driven Spatial Grounding

• LocateAnything is a research framework that connects natural language queries with spatial outputs like 2D/3D boxes and mapped viewpoints.
• It employs varied methods, from retrieval-based matching to generative token prediction, to translate queries into precise spatial hypotheses.
• Applications span assistive retrieval, indoor localization, robotic exploration, and remote-sensing detection, showcasing its broad impact.

Searching arXiv for papers related to "LocateAnything" and adjacent formulations. arxiv_search(query="LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR \"Find Everything\" OR \"Text2Loc\" OR \"The wallpaper is ugly\"", max_results=10, sort_by="submittedDate")

arxiv_search(query="LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR Text2Loc", max_results=10, sort_by="submittedDate")

arxiv_search({"query":"LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR Text2Loc", "max_results":10, "sort_by":"submittedDate"})

LocateAnything is a broad research framing for systems that map an open-ended query to a concrete spatial hypothesis, such as a 2D box, a 3D box, an object location, a user viewpoint, a map entity, or an approximate tagged-object region. Recent work uses the term explicitly for unified vision-language grounding and 3D detection (Wang et al., 26 May 2026, Man et al., 25 Nov 2025), and implicitly for related problems including assistive object retrieval, indoor vision-language localization, geospatial querying over OpenStreetMap, robot exploration in unknown environments, remote-sensing open-vocabulary detection, and privacy-preserving tagged-object finding (Sridhar et al., 23 Sep 2025, Pate et al., 2024, Khellaf et al., 2023, Laina et al., 11 Apr 2025, Pan et al., 2024, Sun et al., 2015). This suggests that LocateAnything is best understood as a family of query-conditioned spatial grounding problems rather than as a single benchmark or algorithm.

1. Conceptual scope

Across the literature, the common structure is not a shared dataset or output format, but a shared question: given a query and an environment representation, what spatial entity should be returned, and how should that entity be reached, ranked, or verified? In some papers the answer is an image-space box or point; in others it is a 3D box in camera coordinates, a discrete panoramic viewpoint in a mapped building, a 2D planar pose in a point-cloud map, an executable geospatial database query, or a small set of candidate mobile detectors that likely cover a Bluetooth tag (Pate et al., 2024, Xia et al., 2023, Khellaf et al., 2023, Sun et al., 2015).

A plausible taxonomy is therefore organized by output space rather than by input modality alone.

Formulation	Query form	Localized output
Unified visual grounding (Wang et al., 26 May 2026)	Natural-language prompt	Boxes or points in an image
Monocular 3D detection (Man et al., 25 Nov 2025)	Free-form text description	3D boxes
Assistive retrieval (Sridhar et al., 23 Sep 2025)	Spoken request	3D target position with guidance
Indoor vision-language localization (Pate et al., 2024)	Free-form description	Discrete mapped viewpoint
Point-cloud localization (Xia et al., 2023)	Textual hints	Planar target position
Geospatial search (Khellaf et al., 2023)	Natural-language query	Structured graph query and map results
Crowdsourced tagged finding (Sun et al., 2015)	Tag identifier	Approximate object location

The phrase can also denote task-specific extensions. "Locate Anything on Earth" reformulates remote-sensing open-vocabulary detection as detecting any novel concepts on Earth (Pan et al., 2024). "FindAnything" denotes open-world mapping and exploration with object-centric volumetric submaps (Laina et al., 11 Apr 2025). "Find Everything" studies multi-object search in unknown environments using multi-channel score maps (Choi et al., 2024). The breadth of these formulations is itself a defining property of the topic.

2. Query modalities and localization targets

The dominant query modality is natural language, but the literature uses it in materially different ways. In assistive retrieval, queries can be spontaneous spoken requests such as “Find my coffee cup,” “I am looking for rotini pasta,” or “Find my keys with the red keychain,” with GPT-4o-mini used as a conversational reasoning layer that can ask clarification questions if needed (Sridhar et al., 23 Sep 2025). In geospatial search, the query may describe multiple objects, attributes, and spatial relations, as in “Please show me all cafes in Ouagadougou on rue pavee that are within 2 miles from a moat, that is nearby an office building that is 35m high,” which Spot translates into an intermediate graph-database format (Khellaf et al., 2023). In mapped indoor localization, the input is a description of the user’s surroundings and the output is a distribution over candidate views, $P(v_m \mid d, i_1,\ldots,i_M)$ (Pate et al., 2024).

Other systems use more structured textual inputs. Text2Loc represents a query as a set of hints $T=\{h_k\}_{k=1}^{N}$, where each hint describes the spatial relationship between the target position and one object instance (Xia et al., 2023). Event geolocation in news treats every location mention as a candidate and predicts whether it is the event-occurring location, making the query effectively the document context around each mention rather than a direct search string (Lee et al., 2016). Remote-sensing LAE uses a provided text prompt $\mathcal{T}$ over a large test vocabulary $\mathcal{V}_{test}$, with no restriction on that test vocabulary (Pan et al., 2024).

Not all LocateAnything systems are text-only. The wearable product-localization system begins with speech input for brand name, product name, and optional quantity, but the downstream target representation is a reference image downloaded from the Open Food Facts database, and the retrieval stage is image-to-image rather than text-to-image (Ruan et al., 18 Jan 2026). SecureFind dispenses with semantic description entirely and instead starts from a unique Bluetooth tag identifier $\widehat{ID}$, embedded in an object-finding request $$\langle H(\widehat{ID}\|r),\, r,\, \mathsf{PK}\rangle$$ (Sun et al., 2015). This diversity is important: LocateAnything does not imply language-only grounding, even when language is the most visible interface.

The localized target likewise varies sharply across systems. Some return image-space detections, some return camera-frame 3D boxes, some return candidate map regions, and some return approximate physical zones to be searched in a second stage. This is why a common misconception—that LocateAnything is simply “open-vocabulary detection”—is too narrow. The literature includes retrieval, navigation, verification, and embodied access layers in addition to recognition.

3. Core computational patterns

One recurrent pattern is retrieval over a candidate set. “The wallpaper is ugly” treats localization as ranking candidate panoramic images in a mapped scan by CLIP similarity, then applying a softmax over those scores (Pate et al., 2024). Text2Loc also starts with retrieval, learning a shared text-submap embedding space and training it with a symmetric contrastive loss before running a separate fine localization regressor (Xia et al., 2023). Spot follows a different route: it does not retrieve from an embedding index but instead performs neural semantic parsing from natural language to a structured graph query, which is then executed against PostgreSQL/PostGIS (Khellaf et al., 2023). These are three distinct mechanisms—embedding retrieval, contrastive place recognition, and semantic query translation—but all instantiate query-conditioned localization over a discrete search space.

A second pattern is perception coupled to explicit spatial grounding. NaviSense first detects a requested object in 2D with Moondream 2B, then converts the returned 2D object location into a 3D spatial point using ARKit and LiDAR depth data, after which the point is continuously tracked and used for spoken and haptic guidance (Sridhar et al., 23 Sep 2025). FindAnything aggregates CLIP features from eSAM-generated segments into object-centric volumetric submaps, allowing open-vocabulary queries to be matched against mapped objects and then converted into 3D target cubes around object centroids (Laina et al., 11 Apr 2025). Finder, by contrast, reasons over exploration rather than direct grasping: it maintains one spatial score channel per target object, builds Scene-to-Object and Object-to-Object score maps, fuses them, and uses the result to rank frontiers in an unknown environment (Choi et al., 2024).

A third pattern is generative localization by token prediction. LocateAnything replaces coordinate-token serialization with Parallel Box Decoding, treating each geometric element as an atomic block of length $L=6$, and factorizing over blocks as $$P(\mathbf{B}\mid \mathcal{Z}, \mathcal{E}) = \prod_{i=1}^{N} P(b_i \mid b_{<i}, Z, \mathcal{E})$$ (Wang et al., 26 May 2026). LocateAnything3D extends the same philosophy into metric 3D perception: its Chain-of-Sight sequence is $$\mathcal{S} = (\mathbf{q}_1, \mathbf{b}_1, \mathbf{q}_2, \mathbf{b}_2, \ldots, \langle \mathrm{eos}\rangle)$$, with 2D boxes emitted before 3D boxes and objects ordered near-to-far (Man et al., 25 Nov 2025). In both cases, the localization act is cast as structured next-token prediction rather than as a separate detection head.

Older work contributes a fourth pattern: contextual scoring and inference by structured reasoning rather than large pretrained VLMs. The ImageNet window-localization system scores each candidate window in the context of all other windows by combining appearance similarity and image-plane spatial relations such as overlap, part, and container (Vezhnevets et al., 2015). SecureFind does something analogous in a distributed sensing regime: it turns object finding into a multi-round polling problem over mobile detectors, hides the true tag with dummy responses, and lets only the object owner infer the likely detector set (Sun et al., 2015). These systems are not open-vocabulary in the modern sense, but they are part of the same lineage of query-conditioned localization.

4. Major domains of application

Assistive retrieval is one of the clearest embodied manifestations of the topic. NaviSense integrates conversational AI, a cloud VLM, ARKit, LiDAR, speech recognition, text-to-speech, and distance-sensitive haptics in an iOS application for blind and low-vision users, with a finite-state machine comprising Idle, Listening, Processing, Speaking, Scanning, and Guiding states (Sridhar et al., 23 Sep 2025). The wearable product-retrieval system for shopping uses YOLO-World for proposal generation, MobileNetV3-Small embeddings plus CIELAB histogram matching for target identification, MediaPipe Hands for fingertip tracking, and VLM-based navigation and correction prompts to address the “last-meter problem” on grocery shelves (Ruan et al., 18 Jan 2026). In both cases, localization is inseparable from multimodal guidance and verification after contact.

Robotic search and exploration form a second cluster. Finder addresses Multi-Object Search in unknown indoor environments by combining YOLOv7, Grounding DINO, Mobile-SAM, BLIP2 embeddings, occupancy mapping, semantic mapping, and a frontier planner over fused score maps (Choi et al., 2024). FindAnything generalizes open-vocabulary mapping and query-guided exploration to volumetric submaps and resource-constrained MAVs, coupling CLIP semantics to a SLAM-updated 3D occupancy representation (Laina et al., 11 Apr 2025). “Fast LiDAR Informed Visual Search in Unseen Indoor Environments” uses a map-free 2D LiDAR classifier to distinguish map from non-map returns and then biases next-best-view planning toward non-permanent scene elements before the visual detector fires (Gupta et al., 2023). These papers emphasize that LocateAnything in robotics is not only a perception problem; it is also an active search problem.

Indoor and map-based place localization form a third domain. “The wallpaper is ugly” localizes a user in a Matterport3D scan from a free-form text description of the surroundings (Pate et al., 2024). Text2Loc localizes a target position in a large 3D point-cloud map from relational textual hints by combining text-submap retrieval with matching-free fine localization (Xia et al., 2023). Spot moves the same basic idea into geospatial databases, allowing free-form descriptions of entities and relations to become executable OSM queries (Khellaf et al., 2023). “Lost in Space” addresses event-location extraction in text by deciding which location mentions are the true event locations, showing that LocateAnything can also denote document-internal grounding rather than physical navigation (Lee et al., 2016).

Remote sensing and large-scale detection extend the idea into overhead imagery. LAE defines “Locate Anything on Earth” as remote-sensing open-vocabulary detection, builds the LAE-Label Engine and LAE-1M, and trains LAE-DINO with Dynamic Vocabulary Construction and Visual-Guided Text Prompt Learning (Pan et al., 2024). Here the located entities are remote-sensing objects conditioned on text prompts, and the central problem is domain transfer from natural-image open-vocabulary detectors into Earth observation.

Finally, tagged-object systems like SecureFind occupy a distinct corner of the space. They do not reason over appearance, language, or maps in the same way, but they solve a direct version of the locate-anything problem for attachable objects by using mobile crowdsourcing, framed slotted ALOHA polling, and encrypted detector locations (Sun et al., 2015). The presence of this line of work is a reminder that the topic is broader than VLMs.

5. Empirical record

The empirical record is heterogeneous because tasks and metrics differ, but several systems report strong results within their own regimes. In assistive retrieval, NaviSense reduced search time to $15.86 \pm 5.65$ s, total time to $T=\{h_k\}_{k=1}^{N}$0 s, undesired object touches to $T=\{h_k\}_{k=1}^{N}$1, and achieved 95.37% accuracy in its main user study; in a supplementary ecological-validity evaluation over 200 sampled frames from cluttered 8-second scenes, its detection pipeline achieved 95% accuracy, with 190 correct identifications or correct rejections (Sridhar et al., 23 Sep 2025). The wearable shopping system reported 93.75% accuracy for shopping-list creation, 100% product-detection accuracy at $T=\{h_k\}_{k=1}^{N}$2 and $T=\{h_k\}_{k=1}^{N}$3 and at $T=\{h_k\}_{k=1}^{N}$4 and $T=\{h_k\}_{k=1}^{N}$5, 94.4% at $T=\{h_k\}_{k=1}^{N}$6 and $T=\{h_k\}_{k=1}^{N}$7, VLM-based navigation accuracy up to 94.4%, and correction accuracy above 86% under the best model configurations (Ruan et al., 18 Jan 2026).

For mapped indoor localization, finetuned CLIP in “The wallpaper is ugly” reached 55% success, 14% Hits@1, 47% same-room accuracy, $T=\{h_k\}_{k=1}^{N}$8 m error, and MRR $T=\{h_k\}_{k=1}^{N}$9 across the full gold dataset; on a simplified 20-choice task it achieved 63% success versus a 57% human baseline, matched humans at 38% Hits@1, and reduced mean localization error to $\mathcal{T}$0 m (Pate et al., 2024). Text2Loc improved top-1 localization recall on the KITTI360Pose test set to $\mathcal{T}$1 for $\mathcal{T}$2 m, compared with RET at $\mathcal{T}$3, and reached retrieval recall $\mathcal{T}$4 at Top-1/3/5 (Xia et al., 2023). “Lost in Space” improved over the dictionary baseline from 51 to 75 accuracy on China, from 59 to 84 on DRC, and from 57 to 75 on Syria, which is the basis for the claim of improvement by as much as 25% (Lee et al., 2016).

In robot exploration, Finder achieved Success Rate 63.4% and MSPL 0.389 on HM3D, and Success Rate 55.4% and MSPL 0.344 on MP3D, outperforming MultiON and several VLM-guided single-object-search baselines adapted to multi-object search (Choi et al., 2024). FindAnything achieved 48.80 mAcc and 62.91 f-mIoU on Replica closed-set semantic evaluation, surpassing the listed comparison methods in that table (Laina et al., 11 Apr 2025). The LiDAR-informed search system reported 86.19% test scan-classification accuracy, and in simulation achieved 100% success in all four settings with average times of 34 s, 90 s, 24 s, and 105 s, closely tracking an oracle-label variant and outperforming NBVP, RRT, and a multisensor frontier baseline (Gupta et al., 2023).

For generative grounding and 3D detection, LocateAnything reported 12.7 Boxes Per Second in Hybrid Mode and 15.3 in Fast Mode, while improving strict-IoU quality relative to Rex-Omni-3B, including LVIS [email protected] from 20.7 to 31.1 and COCO [email protected] from 15.9 to 19.3 (Wang et al., 26 May 2026). LocateAnything3D achieved 49.89 AP$\mathcal{T}$5 on Omni3D, surpassing the previous best by +15.51 absolute improvement even when that baseline was given ground-truth 2D boxes, and reached 52.1 $\mathcal{T}$6 on Omni3D$\mathcal{T}$7 (Man et al., 25 Nov 2025). In remote sensing, LAE-DINO pretrained on LAE-1M reached 85.5 $\mathcal{T}$8 on DIOR, 46.8 $\mathcal{T}$9 on DOTAv2.0, and 20.2 $\mathcal{V}_{test}$0 on LAE-80C, while natural-image-pretrained GLIP and GroundingDINO collapsed to values such as 1.1, 0.3, 0.2, and 0.1 on the same open-set benchmarks (Pan et al., 2024).

Not every line of work is equally mature quantitatively. Spot is presented as a proof-of-concept with a first working system, explicit evaluation criteria for validity of the intermediate format and semantic accuracy of extracted information, but no exact quantitative accuracy, F1, or retrieval-success values in the reported version (Khellaf et al., 2023). SecureFind reports extensive simulations and qualitative efficiency/security trends, but its evaluation is framed around simulations and ranking-based object-security analysis rather than modern end-to-end retrieval benchmarks (Sun et al., 2015).

6. Limits of the current paradigm

A central limitation is that LocateAnything remains fragmented into constrained subproblems. Some systems require mapped environments and discrete candidate views (Pate et al., 2024). Others require structured point-cloud maps and paired text-position supervision (Xia et al., 2023). Geospatial approaches like Spot currently target “a broad, but finite set of possible query structures,” rely on a selected subset of OSM tags, and do not report a ranking strategy for ambiguous matches (Khellaf et al., 2023). SecureFind works only for objects that carry a Bluetooth tag and under an honest-but-curious, non-colluding provider model (Sun et al., 2015). The term therefore overstates present capability if interpreted as unrestricted localization from arbitrary descriptions in arbitrary environments.

Perception and deployment constraints are equally significant. NaviSense is limited by a small-scale shelf-centered evaluation, challenges with occluded objects and visually similar packaging, line-of-sight requirements, steady handling of the phone, and dependence on cloud-based LLM/VLM services (Sridhar et al., 23 Sep 2025). The wearable shopping system depends on Open Food Facts reference images, was tested only on a controlled grocery shelf, and degrades at oblique views and larger distances (Ruan et al., 18 Jan 2026). FindAnything assumes static scenes, requires that the robot first observe target-related evidence before semantics can meaningfully guide exploration, and is bottlenecked by CLIP and eSAM latency on embedded hardware (Laina et al., 11 Apr 2025). Finder assumes static target objects, RGB-D sensing, and object-name queries rather than rich relational descriptions (Choi et al., 2024).

Language and reasoning remain narrower than the topic label suggests. Text2Loc is sensitive to changes in the query text (Xia et al., 2023). Event-geolocation pipelines still rely heavily on engineered contextual cues such as n-gram patterns and mention frequency (Lee et al., 2016). LAE’s semi-automatic LAE-COD branch is explicitly coarse-grained, and DOTAv2.0 is evaluated with horizontal boxes rather than oriented ones (Pan et al., 2024). Generative box models still require fallback policies for malformed blocks or spatial ambiguity in dense scenes, as in LocateAnything’s Hybrid Mode (Wang et al., 26 May 2026). LocateAnything3D, despite strong results, remains single-image, monocular, and sensitive to unusual focal lengths, layouts, and limited 3D annotation diversity (Man et al., 25 Nov 2025).

Taken together, these papers suggest that a fuller LocateAnything system would need to combine several capabilities that are currently scattered across subfields: open-vocabulary perception, spatial grounding into stable 2D and 3D coordinates, active search in partially observed environments, verification after contact or arrival, support for dynamic scenes, and more robust handling of ambiguous or underspecified language. The literature shows that many of these components are already effective in isolation; the unresolved problem is their unification.