---
title: 'LocateAnything: Query-Driven Spatial Grounding'
url: https://www.emergentmind.com/topics/locateanything
type: topic
---

# LocateAnything: Query-Driven Spatial Grounding

Searching arXiv for recent papers related to "LocateAnything" and adjacent formulations.
arxiv_search(query="LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR \"Find Everything\" OR \"Text2Loc\" OR \"The wallpaper is ugly\"", max_results=10, sort_by="submittedDate")

arxiv_search(query="LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR Text2Loc", max_results=10, sort_by="submittedDate")

arxiv_search({"query":"LocateAnything OR \"Locate Anything\" OR FindAnything OR NaviSense OR Text2Loc", "max_results":10, "sort_by":"submittedDate"})

LocateAnything is a broad research framing for systems that map an open-ended query to a concrete spatial hypothesis, such as a 2D box, a 3D box, an object location, a user viewpoint, a map entity, or an approximate tagged-object region. Recent work uses the term explicitly for unified vision-language grounding and 3D detection [2605.27365] [2511.20648], and implicitly for related problems including assistive object retrieval, indoor vision-language localization, geospatial querying over OpenStreetMap, robot exploration in unknown environments, remote-sensing open-vocabulary detection, and privacy-preserving tagged-object finding [2509.18672] [2410.03900] [2311.08093] [2504.08603] [2408.09110] [1503.07932]. This suggests that LocateAnything is best understood as a family of query-conditioned spatial grounding problems rather than as a single benchmark or algorithm.

## 1. Conceptual scope

Across the literature, the common structure is not a shared dataset or output format, but a shared question: given a query and an environment representation, what spatial entity should be returned, and how should that entity be reached, ranked, or verified? In some papers the answer is an image-space box or point; in others it is a 3D box in camera coordinates, a discrete panoramic viewpoint in a mapped building, a 2D planar pose in a point-cloud map, an executable geospatial database query, or a small set of candidate mobile detectors that likely cover a Bluetooth tag [2410.03900] [2311.15977] [2311.08093] [1503.07932].

A plausible taxonomy is therefore organized by output space rather than by input modality alone.

| Formulation | Query form | Localized output |
|---|---|---|
| Unified visual grounding [2605.27365] | Natural-language prompt | Boxes or points in an image |
| Monocular 3D detection [2511.20648] | Free-form text description | 3D boxes |
| Assistive retrieval [2509.18672] | Spoken request | 3D target position with guidance |
| Indoor vision-language localization [2410.03900] | Free-form description | Discrete mapped viewpoint |
| Point-cloud localization [2311.15977] | Textual hints | Planar target position |
| Geospatial search [2311.08093] | Natural-language query | Structured graph query and map results |
| Crowdsourced tagged finding [1503.07932] | Tag identifier | Approximate object location |

The phrase can also denote task-specific extensions. "Locate Anything on Earth" reformulates remote-sensing open-vocabulary detection as detecting any novel concepts on Earth [2408.09110]. "FindAnything" denotes open-world mapping and exploration with object-centric volumetric submaps [2504.08603]. "Find Everything" studies multi-object search in unknown environments using multi-channel score maps [2410.00388]. The breadth of these formulations is itself a defining property of the topic.

## 2. Query modalities and localization targets

The dominant query modality is natural language, but the literature uses it in materially different ways. In assistive retrieval, queries can be spontaneous spoken requests such as “Find my coffee cup,” “I am looking for rotini pasta,” or “Find my keys with the red keychain,” with GPT-4o-mini used as a conversational reasoning layer that can ask clarification questions if needed [2509.18672]. In geospatial search, the query may describe multiple objects, attributes, and spatial relations, as in “Please show me all cafes in Ouagadougou on rue pavee that are within 2 miles from a moat, that is nearby an office building that is 35m high,” which Spot translates into an intermediate graph-database format [2311.08093]. In mapped indoor localization, the input is a description of the user’s surroundings and the output is a distribution over candidate views, \(P(v_m \mid d, i_1,\ldots,i_M)\) [2410.03900].

Other systems use more structured textual inputs. Text2Loc represents a query as a set of hints \(T=\{h_k\}_{k=1}^{N}\), where each hint describes the spatial relationship between the target position and one object instance [2311.15977]. Event geolocation in news treats every location mention as a candidate and predicts whether it is the event-occurring location, making the query effectively the document context around each mention rather than a direct search string [1611.04837]. Remote-sensing LAE uses a provided text prompt \(\mathcal{T}\) over a large test vocabulary \(\mathcal{V}_{test}\), with no restriction on that test vocabulary [2408.09110].

Not all LocateAnything systems are text-only. The wearable product-localization system begins with speech input for brand name, product name, and optional quantity, but the downstream target representation is a reference image downloaded from the Open Food Facts database, and the retrieval stage is image-to-image rather than text-to-image [2601.12486]. SecureFind dispenses with semantic description entirely and instead starts from a unique Bluetooth tag identifier \(\widehat{ID}\), embedded in an object-finding request \(\langle H(\widehat{ID}\|r),\, r,\, \mathsf{PK}\rangle\) [1503.07932]. This diversity is important: LocateAnything does not imply language-only grounding, even when language is the most visible interface.

The localized target likewise varies sharply across systems. Some return image-space detections, some return camera-frame 3D boxes, some return candidate map regions, and some return approximate physical zones to be searched in a second stage. This is why a common misconception—that LocateAnything is simply “open-vocabulary detection”—is too narrow. The literature includes retrieval, navigation, verification, and embodied access layers in addition to recognition.

## 3. Core computational patterns

One recurrent pattern is retrieval over a candidate set. “The wallpaper is ugly” treats localization as ranking candidate panoramic images in a mapped scan by CLIP similarity, then applying a softmax over those scores [2410.03900]. Text2Loc also starts with retrieval, learning a shared text-submap embedding space and training it with a symmetric contrastive loss before running a separate fine localization regressor [2311.15977]. Spot follows a different route: it does not retrieve from an embedding index but instead performs neural semantic parsing from natural language to a structured graph query, which is then executed against PostgreSQL/PostGIS [2311.08093]. These are three distinct mechanisms—embedding retrieval, contrastive place recognition, and semantic query translation—but all instantiate query-conditioned localization over a discrete search space.

A second pattern is perception coupled to explicit spatial grounding. NaviSense first detects a requested object in 2D with Moondream 2B, then converts the returned 2D object location into a 3D spatial point using ARKit and LiDAR depth data, after which the point is continuously tracked and used for spoken and haptic guidance [2509.18672]. FindAnything aggregates CLIP features from eSAM-generated segments into object-centric volumetric submaps, allowing open-vocabulary queries to be matched against mapped objects and then converted into 3D target cubes around object centroids [2504.08603]. Finder, by contrast, reasons over exploration rather than direct grasping: it maintains one spatial score channel per target object, builds Scene-to-Object and Object-to-Object score maps, fuses them, and uses the result to rank frontiers in an unknown environment [2410.00388].

A third pattern is generative localization by token prediction. LocateAnything replaces coordinate-token serialization with Parallel Box Decoding, treating each geometric element as an atomic block of length \(L=6\), and factorizing over blocks as \(P(\mathbf{B}\mid \mathcal{Z}, \mathcal{E}) = \prod_{i=1}^{N} P(b_i \mid b_{<i}, Z, \mathcal{E})\) [2605.27365]. LocateAnything3D extends the same philosophy into metric 3D perception: its Chain-of-Sight sequence is \(\mathcal{S} = (\mathbf{q}_1, \mathbf{b}_1, \mathbf{q}_2, \mathbf{b}_2, \ldots, \langle \mathrm{eos}\rangle)\), with 2D boxes emitted before 3D boxes and objects ordered near-to-far [2511.20648]. In both cases, the localization act is cast as structured next-token prediction rather than as a separate detection head.

Older work contributes a fourth pattern: contextual scoring and inference by structured reasoning rather than large pretrained VLMs. The ImageNet window-localization system scores each candidate window in the context of all other windows by combining appearance similarity and image-plane spatial relations such as overlap, part, and container [1501.01181]. SecureFind does something analogous in a distributed sensing regime: it turns object finding into a multi-round polling problem over mobile detectors, hides the true tag with dummy responses, and lets only the object owner infer the likely detector set [1503.07932]. These systems are not open-vocabulary in the modern sense, but they are part of the same lineage of query-conditioned localization.

## 4. Major domains of application

Assistive retrieval is one of the clearest embodied manifestations of the topic. NaviSense integrates conversational AI, a cloud VLM, ARKit, LiDAR, speech recognition, text-to-speech, and distance-sensitive haptics in an iOS application for blind and low-vision users, with a finite-state machine comprising Idle, Listening, Processing, Speaking, Scanning, and Guiding states [2509.18672]. The wearable product-retrieval system for shopping uses YOLO-World for proposal generation, MobileNetV3-Small embeddings plus CIELAB histogram matching for target identification, MediaPipe Hands for fingertip tracking, and VLM-based navigation and correction prompts to address the “last-meter problem” on grocery shelves [2601.12486]. In both cases, localization is inseparable from multimodal guidance and verification after contact.

Robotic search and exploration form a second cluster. Finder addresses Multi-Object Search in unknown indoor environments by combining YOLOv7, Grounding DINO, Mobile-SAM, BLIP2 embeddings, occupancy mapping, semantic mapping, and a frontier planner over fused score maps [2410.00388]. FindAnything generalizes open-vocabulary mapping and query-guided exploration to volumetric submaps and resource-constrained MAVs, coupling CLIP semantics to a SLAM-updated 3D occupancy representation [2504.08603]. “Fast LiDAR Informed Visual Search in Unseen Indoor Environments” uses a map-free 2D LiDAR classifier to distinguish map from non-map returns and then biases next-best-view planning toward non-permanent scene elements before the visual detector fires [2309.14150]. These papers emphasize that LocateAnything in robotics is not only a perception problem; it is also an active search problem.

Indoor and map-based place localization form a third domain. “The wallpaper is ugly” localizes a user in a Matterport3D scan from a free-form text description of the surroundings [2410.03900]. Text2Loc localizes a target position in a large 3D point-cloud map from relational textual hints by combining text-submap retrieval with matching-free fine localization [2311.15977]. Spot moves the same basic idea into geospatial databases, allowing free-form descriptions of entities and relations to become executable OSM queries [2311.08093]. “Lost in Space” addresses event-location extraction in text by deciding which location mentions are the true event locations, showing that LocateAnything can also denote document-internal grounding rather than physical navigation [1611.04837].

Remote sensing and large-scale detection extend the idea into overhead imagery. LAE defines “Locate Anything on Earth” as remote-sensing open-vocabulary detection, builds the LAE-Label Engine and LAE-1M, and trains LAE-DINO with Dynamic Vocabulary Construction and Visual-Guided Text Prompt Learning [2408.09110]. Here the located entities are remote-sensing objects conditioned on text prompts, and the central problem is domain transfer from natural-image open-vocabulary detectors into Earth observation.

Finally, tagged-object systems like SecureFind occupy a distinct corner of the space. They do not reason over appearance, language, or maps in the same way, but they solve a direct version of the locate-anything problem for attachable objects by using mobile crowdsourcing, framed slotted ALOHA polling, and encrypted detector locations [1503.07932]. The presence of this line of work is a reminder that the topic is broader than VLMs.

## 5. Empirical record

The empirical record is heterogeneous because tasks and metrics differ, but several systems report strong results within their own regimes. In assistive retrieval, NaviSense reduced search time to \(15.86 \pm 5.65\) s, total time to \(31.75 \pm 8.11\) s, undesired object touches to \(0.52 \pm 0.85\), and achieved 95.37% accuracy in its main user study; in a supplementary ecological-validity evaluation over 200 sampled frames from cluttered 8-second scenes, its detection pipeline achieved 95% accuracy, with 190 correct identifications or correct rejections [2509.18672]. The wearable shopping system reported 93.75% accuracy for shopping-list creation, 100% product-detection accuracy at \(0.5\,\mathrm{m}\) and \(0^\circ\) and at \(1.0\,\mathrm{m}\) and \(0^\circ\), 94.4% at \(1.5\,\mathrm{m}\) and \(0^\circ\), VLM-based navigation accuracy up to 94.4%, and correction accuracy above 86% under the best model configurations [2601.12486].

For mapped indoor localization, finetuned CLIP in “The wallpaper is ugly” reached 55% success, 14% Hits@1, 47% same-room accuracy, \(7.30\) m error, and MRR \(0.28\) across the full gold dataset; on a simplified 20-choice task it achieved 63% success versus a 57% human baseline, matched humans at 38% Hits@1, and reduced mean localization error to \(5.90\) m [2410.03900]. Text2Loc improved top-1 localization recall on the KITTI360Pose test set to \(0.33/0.48/0.52\) for \(\epsilon < 5/10/15\) m, compared with RET at \(0.16/0.25/0.29\), and reached retrieval recall \(0.28/0.49/0.58\) at Top-1/3/5 [2311.15977]. “Lost in Space” improved over the dictionary baseline from 51 to 75 accuracy on China, from 59 to 84 on DRC, and from 57 to 75 on Syria, which is the basis for the claim of improvement by as much as 25% [1611.04837].

In robot exploration, Finder achieved Success Rate 63.4% and MSPL 0.389 on HM3D, and Success Rate 55.4% and MSPL 0.344 on MP3D, outperforming MultiON and several VLM-guided single-object-search baselines adapted to multi-object search [2410.00388]. FindAnything achieved 48.80 mAcc and 62.91 f-mIoU on Replica closed-set semantic evaluation, surpassing the listed comparison methods in that table [2504.08603]. The LiDAR-informed search system reported 86.19% test scan-classification accuracy, and in simulation achieved 100% success in all four settings with average times of 34 s, 90 s, 24 s, and 105 s, closely tracking an oracle-label variant and outperforming NBVP, RRT, and a multisensor frontier baseline [2309.14150].

For generative grounding and 3D detection, LocateAnything reported 12.7 Boxes Per Second in Hybrid Mode and 15.3 in Fast Mode, while improving strict-IoU quality relative to Rex-Omni-3B, including LVIS F1@0.95 from 20.7 to 31.1 and COCO F1@0.95 from 15.9 to 19.3 [2605.27365]. LocateAnything3D achieved 49.89 AP\(_{3D}\) on Omni3D, surpassing the previous best by +15.51 absolute improvement even when that baseline was given ground-truth 2D boxes, and reached 52.1 \(\mathrm{AP}^{out}_{3D}\) on Omni3D\(_{OUT}\) [2511.20648]. In remote sensing, LAE-DINO pretrained on LAE-1M reached 85.5 \(AP_{50}\) on DIOR, 46.8 \(mAP\) on DOTAv2.0, and 20.2 \(mAP\) on LAE-80C, while natural-image-pretrained GLIP and GroundingDINO collapsed to values such as 1.1, 0.3, 0.2, and 0.1 on the same open-set benchmarks [2408.09110].

Not every line of work is equally mature quantitatively. Spot is presented as a proof-of-concept with a first working system, explicit evaluation criteria for validity of the intermediate format and semantic accuracy of extracted information, but no exact quantitative accuracy, F1, or retrieval-success values in the reported version [2311.08093]. SecureFind reports extensive simulations and qualitative efficiency/security trends, but its evaluation is framed around simulations and ranking-based object-security analysis rather than modern end-to-end retrieval benchmarks [1503.07932].

## 6. Limits of the current paradigm

A central limitation is that LocateAnything remains fragmented into constrained subproblems. Some systems require mapped environments and discrete candidate views [2410.03900]. Others require structured point-cloud maps and paired text-position supervision [2311.15977]. Geospatial approaches like Spot currently target “a broad, but finite set of possible query structures,” rely on a selected subset of OSM tags, and do not report a ranking strategy for ambiguous matches [2311.08093]. SecureFind works only for objects that carry a Bluetooth tag and under an honest-but-curious, non-colluding provider model [1503.07932]. The term therefore overstates present capability if interpreted as unrestricted localization from arbitrary descriptions in arbitrary environments.

Perception and deployment constraints are equally significant. NaviSense is limited by a small-scale shelf-centered evaluation, challenges with occluded objects and visually similar packaging, line-of-sight requirements, steady handling of the phone, and dependence on cloud-based LLM/VLM services [2509.18672]. The wearable shopping system depends on Open Food Facts reference images, was tested only on a controlled grocery shelf, and degrades at oblique views and larger distances [2601.12486]. FindAnything assumes static scenes, requires that the robot first observe target-related evidence before semantics can meaningfully guide exploration, and is bottlenecked by CLIP and eSAM latency on embedded hardware [2504.08603]. Finder assumes static target objects, RGB-D sensing, and object-name queries rather than rich relational descriptions [2410.00388].

Language and reasoning remain narrower than the topic label suggests. Text2Loc is sensitive to changes in the query text [2311.15977]. Event-geolocation pipelines still rely heavily on engineered contextual cues such as n-gram patterns and mention frequency [1611.04837]. LAE’s semi-automatic LAE-COD branch is explicitly coarse-grained, and DOTAv2.0 is evaluated with horizontal boxes rather than oriented ones [2408.09110]. Generative box models still require fallback policies for malformed blocks or spatial ambiguity in dense scenes, as in LocateAnything’s Hybrid Mode [2605.27365]. LocateAnything3D, despite strong results, remains single-image, monocular, and sensitive to unusual focal lengths, layouts, and limited 3D annotation diversity [2511.20648].

Taken together, these papers suggest that a fuller LocateAnything system would need to combine several capabilities that are currently scattered across subfields: open-vocabulary perception, spatial grounding into stable 2D and 3D coordinates, active search in partially observed environments, verification after contact or arrival, support for dynamic scenes, and more robust handling of ambiguous or underspecified language. The literature shows that many of these components are already effective in isolation; the unresolved problem is their unification.

Source: https://www.emergentmind.com/topics/locateanything