Text-Grounded Object Localization

• Text-grounded object localization is the process of linking natural language expressions to explicit spatial regions in visual data, such as bounding boxes, masks, and 3D centroids.
• It spans diverse applications including image-text verification, video segmentation, remote sensing, document grounding, and active robotic search to ensure language aligns with visual cues.
• Advancements in supervision regimes, modeling paradigms, and evaluation metrics demonstrate that precise mapping of text to visuals enhances localization accuracy and explanation quality.

Text-grounded object localization denotes the family of problems in which a natural-language expression is mapped to explicit spatial evidence in visual data. In the cited literature, that evidence may be a horizontal bounding box, an oriented bounding box, a segmentation mask, a temporal trajectory, a document text box, or a 3D centroid; in some formulations it is coupled to verification and explanation rather than localization alone. The topic therefore spans phrase grounding, grounded caption verification, referring video object segmentation, remote-sensing visual grounding, document understanding, active robotic search, and 3D grounding, with a common requirement that linguistic content be tied to a concrete visual region or object representation (Gao et al., 25 Jun 2026, Zhou et al., 2024, Grant et al., 9 Mar 2026, Wang et al., 2023).

1. Task formulations and output spaces

A central formulation is given by GAVEL, where each datapoint is a tuple $(I, D, E, G)$ consisting of an image $I$, a single-sentence description $D$, a free-form explanation $E$ of any image–text misalignment, and a grounding region $G$ given by bounding-box coordinates. In that setting, localization is not isolated: the model must verify inconsistency, explain it in natural language, and localize the visual evidence. The paper explicitly frames this as a fine-grained benchmark for systematically evaluating and training vision-LLMs to reason about alignment and discrepancy while also providing spatial evidence (Gao et al., 25 Jun 2026).

Other formulations distribute the same grounding requirement across different upstream tasks. ActivityNet Entities Object Localization evaluates how grounded or faithful a description is through two sub-tasks: localization with ground-truth sentences and localization with generated sentences. Weakly-supervised video object grounding treats a sentence as specifying the set of objects to be localized across a video segment even though no training bounding boxes are available. Referring video object segmentation moves from boxes to masks and requires localization throughout an entire video. Remote-sensing visual grounding further expands the output space to HBBs, OBBs, and masks in a single framework. Document understanding formulations, by contrast, ground textual content itself, representing localized text as <text>[x_{\min}, y_{\min}, x_{\max}, y_{\max}]. Pose-free 3D localization adds per-view segmentation masks and 3D object centroids derived from world-space coordinates (Ruan et al., 2021, Zhou et al., 2018, Lin et al., 2024, Zhou et al., 2024, Wang et al., 2023, Grant et al., 9 Mar 2026).

Setting	Representative output	Example source
Image–text verification	Bounding box plus free-form explanation	GAVEL
Video grounding	Frame-level bounding box	Weakly-supervised video object grounding
RVOS	Temporal-consistent mask sequence	GroPrompt
Remote sensing	HBB, OBB, or mask	GeoGround
Document grounding	Text box coordinates	TGDoc
3D grounding	Segmentation mask and 3D centroid	TrianguLang

This range of output spaces undercuts a common simplification that treats text-grounded localization as synonymous with 2D phrase grounding over axis-aligned boxes. The surveyed work instead presents it as a broader grounding interface between language and spatial structure.

2. Supervision regimes, datasets, and annotation design

The supervision spectrum is unusually broad. At one end, GAVEL uses human-annotated supervision for verification, explanation, and localization. Negative image–caption pairs are collected from both image-to-text and text-to-image pipelines; automated alignment scores filter likely misaligned pairs; annotators verify misalignment, write explanations, and annotate bounding boxes; annotation is then verified through a multi-stage review process. The reported scale is approximately 30,000 training images with 35,000 pairs and approximately 2,500 test images with 5,000 pairs. The benchmark includes diverse hallucination types such as object, attribute, color, locational, and counting hallucinations; the average is 1.32 boxes per image; only approximately 7.3% of training pairs are “no hallucination” pairs; and sentence length averages roughly 25 words with more than 27,000 unique vocabulary items (Gao et al., 25 Jun 2026).

At the opposite end are weakly supervised or annotation-efficient settings. Weakly-supervised video object grounding assumes only paired video segments and sentences during training, with no object bounding boxes, while the set of possible objects is known beforehand. GroPrompt explicitly targets referring video object segmentation from weak supervision using only box supervisions rather than dense mask annotations. ZSOL is framed as zero-shot object localization guided by prompt words in the absence of labeled samples for target categories. TrianguLang removes camera calibration at inference and operates feed-forward on uncalibrated multi-view RGB images plus a single text query (Zhou et al., 2018, Lin et al., 2024, Wang et al., 2024, Grant et al., 9 Mar 2026).

Several datasets are designed around domain-specific grounding constraints. ActivityNet Entities provides frame-caption pairs for caption-conditioned grounding in video, and the Team RUC_AIM3 system fine-tunes MDETR on it for 5 epochs with batch size 2 (Ruan et al., 2021). GeoGround introduces refGeo at the scale of 161,000 image-text pairs over more than 80,000 remote-sensing images, with every object annotated using HBB, OBB, and segmentation mask, and with masks bootstrapped from boxes or OBBs using SAM before further refinement (Zhou et al., 2024). TGDoc pre-trains on 99K PowerPoint presentations and fine-tunes with 12K multi-turn QA conversations featuring text locations in text-rich images (Wang et al., 2023). TReCS exploits Localized Narratives, where detailed descriptions are paired with mouse traces that provide sparse, fine-grained visual grounding for phrases (Koh et al., 2020).

These dataset choices reflect a recurrent design decision: grounding annotations are often embedded inside a larger communicative task—caption verification, caption generation, task instructions, or question answering—rather than treated as an isolated box-regression problem.

3. Modeling paradigms

The cited work presents several distinct modeling paradigms. A classic line is MIL-style grounding. In weakly-supervised video grounding, object queries and region proposals are projected into a shared feature space, and segment-level ranking losses are adapted to frame-wise grounding by estimating per-frame confidence and reweighting supervision. The full model combines visual-semantic matching confidence with language-derived confidence from self-attention over object queries, yielding the frame-wise loss

$$L = \frac{1}{T} \sum_{t=1}^{T} \left[ \lambda \frac{1}{2}(C_t + C^{t_s}_{\text{lang}}) L^{t}_{\text{rank}} - (1-\lambda)\log(C_t + C^{t_s}_{\text{lang}}) \right].$$

The stated motivation is that naive extension of image MIL to video lets a single successful match satisfy the loss even when the object appears only sparsely across frames (Zhou et al., 2018).

A second paradigm separates language generation and grounding. Team RUC_AIM3 divides caption generation and object grounding into two stages: UMPM generates event descriptions with rich objects, and MDETR performs object grounding, followed by IoU voting and co-reference unification. The paper argues that this separation permits each module to leverage domain-specific architectures and improvements, rather than constraining both in a single jointly trained framework (Ruan et al., 2021).

A third paradigm treats localization itself as sequence generation. GAVEL augments the tokenizer with coordinate tokens $\mathcal{P}=\{\tau_i\mid i\in[-512,512]\}$, normalizes and quantizes coordinates via

$$q(v,S)=\mathrm{round}\left(512\cdot \mathrm{clip}\left(\frac{v-S/2}{S/2},-1,1\right)\right),$$

and serializes a box as

$$(\tau_{q(x_1,W)},\tau_{q(y_1,H)},\tau_{q(x_2,W)},\tau_{q(y_2,H)}).$$

The model is then supervised to autoregressively generate both coordinate tokens and free-form explanation text in a unified sequence (Gao et al., 25 Jun 2026).

A fourth paradigm textualizes the output space so that a general VLM can emit localization signals without specialized heads. GeoGround converts HBBs, OBBs, and masks into textual sequences, including a Text-Mask paradigm that downsamples a binary mask to an $N\times N$ grid and compresses it using R-RLE. It further introduces prompt-assisted learning,

$I$0

and geometry-guided learning,

$I$1

to encourage consistency across HBB, OBB, and mask outputs (Zhou et al., 2024).

Other models center the cross-modal representation itself. GRILL constructs hybrid sequences in which referring words are replaced by patch-aggregated region embeddings and trains with PrefixLM, MaskedLM, and a discriminative true/false alignment objective. ZSOL combines CLIP with a Text Self-Similarity Matching module that refines the text embedding before density-map prediction. In egocentric grounding, symbolic knowledge is injected into GLIP through per-object aggregation masking so that object phrases, pre/post-conditions, and descriptions are aligned separately. GroPrompt learns box prompts rather than dense masks and uses TextCon and ModalCon triplet losses to make prompts sentence-specific and video-aware (2305.14676, Wang et al., 2024, Wu et al., 2023, Lin et al., 2024).

A broad methodological pattern emerges: localization improves when text is not treated as a single global sentence embedding but is decomposed into object phrases, geometry, temporal cues, or auxiliary symbolic descriptions.

4. Objectives, metrics, and consistency criteria

Intersection-over-Union remains a principal criterion for box grounding. In GAVEL, IoU between predicted and ground-truth boxes is defined as

$I$2

and localization accuracy at threshold $I$3 is

$I$4

with typical thresholds $I$5. GAVEL jointly evaluates localization and explanation, using GPT-5 as an LLM-as-judge for explanation quality on a 1–5 scale that is then scaled to 0–100 (Gao et al., 25 Jun 2026).

ActivityNet Entities uses related but task-specific metrics. In sub-task I, localization is correct if IoU exceeds $I$6, averaged across unique object categories. In sub-task II, correctness requires that the object word be both generated and correctly localized, and performance is reported using $I$7, $I$8, $I$9, and per-sentence variants such as $D$0 (Ruan et al., 2021).

Mask-based tasks shift the emphasis from box overlap to region overlap and contour fidelity. GroPrompt reports Region Similarity $D$1, Contour Accuracy $D$2, and their mean $D$3 on Ref-YouTube-VOS and Ref-DAVIS17, while A2D Sentences and JHMDB Sentences use Precision@K and overall or mean IoU. Remote-sensing grounding adds cross-signal consistency through the BBox Consistency Score,

$D$4

which measures agreement among HBB, OBB, and mask predictions for the same object (Lin et al., 2024, Zhou et al., 2024).

In scene-generation settings, evaluation can depart further from direct IoU. The text-to-3D scene generation work introduces ASTS,

$D$5

which the paper reports as strongly correlated with human judgments. This suggests that, in compositional grounding tasks, alignment may be evaluated at the level of object-template correspondence rather than only pixel or box overlap (Chang et al., 2015).

5. Reported empirical findings

The empirical picture is mixed: grounding remains difficult, but task-specific supervision and geometry-aware modeling can produce large gains. GAVEL reports that even strong closed-source models struggle. For localization, GPT-5 obtains 34.1 at [email protected], 17.4 at [email protected], and 6.5 at [email protected]; GPT-5-mini obtains 32.5, 14.7, and 4.2; Qwen3-VL-30B obtains 30.8, 22.8, and 16.4; and InternVL-3.5-30B obtains 23.5, 15.8, and 9.4. On explanation quality, GPT-5 scores 70 and Qwen3-VL-30B scores 62. Fine-tuning the VisionLLM baseline on GAVEL raises [email protected] from 15.1 to 42.6, [email protected] from 12.1 to 30.2, and [email protected] from 9.1 to 18.4. The error analysis further reports no systematic size bias in predicted boxes; inaccuracies stem from localization imprecision rather than simple scale effects (Gao et al., 25 Jun 2026).

In ActivityNet Entities, the two-stage UMPM plus fine-tuned MDETR system reports a final testing-set localization accuracy of 72.57 on sub-task I and 0.2477 $D$6 on the hidden testing set of sub-task II. The same report attributes gains to separation of captioning and grounding, Spacy-based filtering of abstract nouns, model fusion, and co-reference unification (Ruan et al., 2021). In weakly-supervised video grounding, the full model improves test Box Accuracy from 30.80 for the DVSA baseline to 31.73, while the proposal upper bound remains 58.56, indicating remaining dependence on proposal quality (Zhou et al., 2018).

Cross-domain studies show that explicit geometric or multi-signal grounding can match or surpass specialized systems. GeoGround reports 77.73% HBB [email protected] on DIOR-RSVG-Test, 59.72% OBB [email protected] on GeoChat*, and 60.97% mask [email protected] with 54.92% mIoU on RRSIS-D test, with further gains after a SAM refiner. TrianguLang reports 62.4% mIoU and 77.4% mAcc on ScanNet++, 94.6% mIoU and 98.3% mAcc on uCO3D, and 58.1% mIoU with 83.5% localization accuracy on LERF-OVS while remaining feed-forward and pose-free at inference (Zhou et al., 2024, Grant et al., 9 Mar 2026).

Instruction-aware and embodied settings report similarly strong effects from grounding cues. In egocentric vision, symbolic world knowledge leads to more than 54% improvements in all standard metrics on TREK-150-OPE-Det, more than 7% improvements on TREK-150-OPE, and more than 3% AP on Ego4D SCOD. WoMAP reports more than $D$7 and $D$8 higher success rates than VLM and diffusion-policy baselines, respectively, in zero-shot active object localization (Wu et al., 2023, Yin et al., 2 Jun 2025).

These results support a narrow but important conclusion: localization can improve substantially when the training signal specifies not only what the object is, but also how language, geometry, temporal structure, or error evidence should map onto space.

6. Broader extensions, misconceptions, and mechanistic interpretations

Text-grounded localization is tightly entangled with generation and control. GLIGEN conditions text-to-image diffusion on captions plus grounding entities, freezes the entire pre-trained pipeline, and injects grounding through gated self-attention, reporting zero-shot performance on COCO and LVIS that outperforms supervised layout-to-image baselines on several metrics. ST-LDM addresses text-grounded object generation in real images using a Swin-Transformer-based framework with deformable multimodal alignment and training-free backward guidance; on its benchmark, the text-only ST-LDM result is reported as FID 6.16, LPIPS 15.54, HPS 64.28, CLIPScore 81.97, and VISOR 92.39. TGT conditions video generation on trajectories paired with localized text descriptions and reports EPE 25.11 with CLIP-T (Local) 0.2531. TReCS uses fine-grained user attention traces to retrieve and place segmentation masks before synthesis, reporting IS 21.3, FID 48.7, and strong human preference over AttnGAN on realism and alignment (Li et al., 2023, Xue et al., 2024, Zhang et al., 16 Oct 2025, Koh et al., 2020).

A related misconception is that strong general VLM competence automatically implies reliable localization. GAVEL directly reports that even strong closed-source models struggle, and mechanistic analysis provides a possible explanation. The interpretability study of LLaVA-1.5 and InternVL-3.5 argues that localization is driven by a “containerization” mechanism in which object-aligned tokens define spatial extent while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads is reported to mediate the causal effect, with classification and localization sharing some early processing but depending on largely distinct specialized heads (Gao et al., 25 Jun 2026, Schaumlöffel et al., 19 May 2026).

The 3D literature adds a second correction: semantic compatibility alone is insufficient when geometric consistency is weak. ViL3DRel explicitly conditions self-attention on pairwise distances and orientations between objects in point clouds, and TrianguLang’s Geometry-Aware Semantic Attention suppresses semantically plausible but geometrically inconsistent cross-view matches without requiring camera calibration at inference. This suggests that explicit spatial structure remains a primary design axis even in large multimodal systems (Chen et al., 2022, Grant et al., 9 Mar 2026).

Across these works, text-grounded object localization appears less as a single benchmark than as a unifying operation: binding language to visual entities through boxes, masks, trajectories, or 3D coordinates, and doing so under conditions that may require explanation, temporal reasoning, geometry, controllable generation, or physical action. The cited results collectively indicate that progress depends not only on stronger encoders or larger VLMs, but on the precision of the grounding interface itself.