LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

Abstract: Vision-LLMs (VLMs) commonly formulate visual grounding and detection as a coordinate-token generation problem, serializing each 2D box into multiple 1D tokens that are learned and decoded largely independently. This token-by-token decoding mismatches the coupled structure of box geometry and creates a practical inference bottleneck due to strictly sequential generation. We introduce LocateAnything, a unified generative grounding and detection framework based on Parallel Box Decoding (PBD). By decoding geometric elements such as bounding boxes and points as atomic units in a single step, LocateAnything preserves intra-box geometric coherence and unlocks substantial parallelism. We show that PBD improves both decoding throughput and localization accuracy. We further develop a scalable data engine and curate LocateAnything-Data, a large-scale dataset with more than 138 million training samples, substantially increasing data diversity for high-precision localization. Extensive evaluations show that LocateAnything advances the speed-accuracy frontier, achieving significantly higher decoding throughput while improving high-IoU localization quality across diverse benchmarks. The results highlight the complementary benefits of Parallel Box Decoding and large-scale training data in enabling efficient and precise unified visual grounding and detection.

• The paper introduces Parallel Box Decoding to redefine bounding box generation as block-based atomic prediction, significantly enhancing grounding speed and accuracy.
• It integrates a hybrid inference mode that adaptively switches between parallel and sequential decoding to mitigate format irregularities and spatial ambiguities.
• The method achieves 12.7 boxes/sec and superior F1 scores on benchmarks, demonstrating robust, multi-domain vision-language performance.

LocateAnything: Unified Fast and High-Quality Vision-Language Grounding via Parallel Box Decoding

Motivation and Background

Vision-LLMs (VLMs) have become integral in interactive and embodied systems, particularly for tasks requiring robust grounding and detection from natural language queries. Traditional generative paradigms serialize spatial coordinates as 1D token sequences—either as textual digits or quantized tokens—culminating in token-level sequential generation (Next-Token Prediction, NTP). This approach produces a mismatch with the inherently structured nature of bounding boxes, incurs significant inference latency, and undermines geometric coherence. Attempts to improve throughput via Multi-Token Prediction (MTP) have thus far relied on sequence chunking that is agnostic to spatial structure, generating unaligned or irregular outputs.

LocateAnything addresses these issues by leveraging Parallel Box Decoding (PBD), proposing a new model formulation and training paradigm that predicts full geometric entities (bounding boxes and points) as atomic blocks. The approach is complemented by a large-scale, multi-domain dataset construction pipeline designed to support high-precision, open-world grounding.

Figure 1: LocateAnything enables diverse localization tasks under a unified vision-LLM, contrasting PBD’s parallel decoding of geometric units with slower token-by-token approaches.

Parallel Box Decoding: Methodology and Model Architecture

LocateAnything redefines bounding box generation as a block-based atomic prediction task. The model architecture employs a Moon-ViT vision encoder and a Qwen2.5 language decoder, connected through an MLP projector. Visual tokens are extracted at native resolution, preserving spatial granularity for precise localization.

The output is reformulated as a sequence of fixed-length blocks, each representing a bounding box, negative signal, semantic phrase, or end token. Blocks are consistently padded and standardized to enable parallel decoding, eschewing typical sequential prediction used in NTP. The joint probability is factorized over block predictions, maintaining causal inter-block dependencies and bidirectional intra-block attention.

Figure 2: Comparison of decoding methods highlights PBD’s structured, box-aligned and atomic output contrasted with the irregular, unstructured patterns produced by standard NTP and MTP approaches.

Figure 3: LocateAnything’s architecture shows the block-based output with defined block types ensuring unified prediction of entities and termination states.

The training paradigm aligns NTP and block-level MTP via dual-formulation supervision, enforcing causal reasoning while enabling block-aligned atomic predictions. Special attention mask design isolates NTP/MTP streams while allowing both to leverage the same visual and textual context.

Hybrid Inference and Decoding Robustness

Parallel decoding encounters failure modes such as format irregularity (malformed block syntax at category boundaries) and spatial ambiguity (blurred boundaries in dense layouts). LocateAnything mitigates these by implementing an on-demand Hybrid Mode: parallel decoding is the default, but local fallback to NTP is triggered upon detection of unreliable outputs, based on syntactic integrity and spatial token confidence thresholds.

Figure 4: Schematic of the NTP fallback mechanism for robustness against format irregularity and spatial ambiguity.

This adaptive inference mechanism ensures robust predictions with maximal speed benefit. Three modes are supported: Fast (max throughput, parallel), Slow (max precision, sequential NTP), and Hybrid (adaptive fallback).

LocateAnything-Data: Dataset Scale and Diversity

The LocateAnything-Data corpus provides massive coverage, containing 12M unique images, 138M queries, and 785M bounding boxes. The dataset spans six distinct domains: general object detection, GUI element grounding, referring expression comprehension, text localization, scene/document layout grounding, and point-based localization. Explicit negative queries are included to minimize hallucination tendencies.

Dataset construction leverages advanced data engines that synthesize queries from both labeled and unlabeled images using LLMs. All candidate boxes are post-verified for quality control.

Figure 5: Dataset breakdown, illustrating distributions across queries, bounding boxes, images, and task categories.

Figure 6: Multi-target data engine: queries are synthesized from ground-truth boxes or directly from images, leveraging LLMs and segmentation models for point-to-box conversion and final verification.

Figure 7: Long-tailed distribution of targets per query, indicating multi-object and compositional grounding scenarios across domains.

Figure 8: Variation in linguistic query length demonstrates task-specific natural language diversity.

Experimental Evaluation and Results

LocateAnything advances both speed and accuracy frontiers. Under the Hybrid decoding mode, it achieves 12.7 boxes/sec, 2.5× faster than state-of-the-art VLM baselines (e.g., Rex-Omni 5.0 BPS; Qwen3-VL 1.1 BPS), and simultaneously delivers superior mean F1 scores: +3.8% (LVIS), +1.8% (COCO), and pronounced improvements in dense detection (VisDrone, Dense200).

Qualitative and task-specific results confirm compositional grounding (spatial/attribute-based reasoning), robust dense localization, and structural coherence under clutter, occlusion, and complex multi-target settings.

Figure 9: Ablation confirms optimal box ordering strategies and throughputs across decoding modes and representations.

Figure 10: Qualitative outputs across test cases: accurate localization across sparse/crowded scenes, free-form queries, and diverse box scales.

Figure 11: Superior compositional grounding for referring expression comprehension compared to Rex-Omni and Qwen3-VL.

Figure 12: Dense object detection comparison: LocateAnything maintains compact, well-separated bounding boxes in highly overlapping environments.

Figure 13: Optical character recognition: LocateAnything yields high-precision, coherent box predictions across distinct text regions and layouts.

LocateAnything demonstrates high recall for open-world tasks, outperforming specialized and generalist baselines in GUI grounding, document analysis, and complex referring queries.

Implications and Future Directions

LocateAnything substantiates the utility of block-aligned parallel decoding for structured vision-language grounding, showing measurable improvements in speed/accuracy tradeoffs. The adaptive Hybrid inference mode is a practical solution for production pipelines, enabling real-time embodied perception in latency-sensitive settings.

The method is not tethered to a specific backbone, as PBD improves throughput and accuracy across multiple VLM architectures. The approach sets a foundation for scalable, precise multi-task grounding, opening paths for RL-based optimization and further exploitation of block-level supervision in dense, high-variance scenarios.

Conclusion

LocateAnything integrates Parallel Box Decoding into vision-language grounding, achieving robust geometric coherence, substantial throughput gains, and high localization accuracy. The block-level supervision and hybrid inference design overcome key bottlenecks of traditional sequential decoding, offering practical scalability for real-time perception applications. Expansion through RL-based policy optimization and further diversification of training modalities is anticipated to enhance robustness and adaptability for increasingly complex multi-domain tasks.

(2605.27365)