REIRCOCO: Benchmark for Instance Retrieval

• REIRCOCO is a comprehensive benchmark for referring expression instance retrieval that bridges image retrieval and object localization across large galleries.
• It employs a generation–filtering pipeline using MS COCO data to create diverse, context-rich referring expressions emphasizing attributes and relations.
• The associated CLARE baseline leverages a dual-stream architecture with a CLIA objective to achieve superior joint retrieval and localization performance.

Searching arXiv for REIRCOCO and closely related papers to ground the article. REIRCOCO is a large-scale benchmark for Referring Expression Instance Retrieval (REIR), a vision–language task in which a system receives a natural-language referring expression and must search a gallery of images to both retrieve the correct image and localize the target object instance with a bounding box. It was introduced together with the task formalization and a strong end-to-end baseline in "Referring Expression Instance Retrieval and A Strong End-to-End Baseline" (Hao et al., 23 Jun 2025). The benchmark is positioned between Text–Image Retrieval (TIR), which retrieves images from image-level descriptions, and Referring Expression Comprehension (REC), which localizes objects inside a known image. REIRCOCO addresses the regime in which queries are instance-level, gallery-wide, and explicitly require both ranking and localization.

1. Definition and task formulation

REIR is defined as a unified retrieval-and-grounding problem: given a query $x_{\text{text}}$ and a gallery $\{x_{\text{img}}^i\}_{i=1\ldots N}$, the model outputs a bounding box $B=[x,y,w,h]$ that must lie in the unique image containing the referred instance (Hao et al., 23 Jun 2025). This formulation departs from two established tasks.

Text–Image Retrieval ranks images from an image-level caption, but does not return an object-level location and often fails on fine-grained, instance-specific queries. Referring Expression Comprehension localizes an instance in a single known image, but does not scale to gallery-wide search and lacks a cross-image ranking mechanism. The motivation for REIR is that real applications such as surveillance, forensics, and photo search require both instance-level retrieval and precise localization across large galleries, with expressions rich in attributes and relations such as “the brown dog lies on a cushion near the fireplace with a red rug underneath” (Hao et al., 23 Jun 2025).

A central claim of the benchmark paper is that naïvely composing TIR and REC is insufficient. TIR is too coarse for object-level semantics, while REC-style early-fusion pipelines are computationally prohibitive across galleries and produce scores that are not comparable between images. This suggests that REIR is not merely a combination of prior tasks, but a separate evaluation setting with its own benchmark design and retrieval-grounding metrics (Hao et al., 23 Jun 2025).

2. Benchmark construction and corpus characteristics

REIRCOCO is constructed from MS COCO Detection and the RefCOCO/RefCOCO+/RefCOCOg family. COCO contributes dense instance boxes and category labels across 80 object categories, while the RefCOCO variants motivate coverage of relational semantics through human-written short expressions (Hao et al., 23 Jun 2025).

The benchmark uses a Generation–Filtering pipeline. In the generation stage, GPT-4o is prompted with structured information for each annotated instance: the instance bounding box and category, the image’s global caption from MS COCO Captions when available, and contextual information about nearby objects and scene layout. GPT-4o is instructed to generate 5–10 diverse, context-rich referring expressions per instance that emphasize appearance, attributes, and relations, avoid absolute coordinates, and ensure discriminative uniqueness across the gallery. If an instance is too small or occluded, the prescribed output is the placeholder “Instance quality is poor.” In the filtering stage, DeepSeek R1 verifies expression quality from the caption, candidate expressions, and structured metadata, rejecting ambiguous, inaccurate, unverifiable, or non-discriminative expressions (Hao et al., 23 Jun 2025).

The resulting dataset statistics are as follows.

Item	Value
Images	30,106
Object instances	215,835
Verified referring expressions	613,548

The benchmark covers 80 MS COCO object classes and is designed so that expressions are unambiguous and uniquely tied to a single instance, with strong emphasis on attributes and relations. The paper qualitatively stresses relational richness, including spatial context and interactions, but does not report average expression length or exact distributional summaries. It evaluates on standardized train/val/test splits aligned with COCO/RefCOCO images, but the specific split counts are not enumerated (Hao et al., 23 Jun 2025).

Annotations are instance-centered. A typical JSON entry contains image_id, instance_id, category_id, bbox, and a list of verified expressions. The benchmark paper gives a concrete example for image 391895, instance 391895_07, category 18, with bounding box [472, 210, 85, 64] and three expressions describing a brown dog by appearance and relation to the fireplace, rug, and iron grate (Hao et al., 23 Jun 2025).

3. Evaluation protocol and metrics

REIRCOCO evaluates both retrieval quality and localization quality. Localization follows standard REC practice through Intersection-over-Union,

$$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$

and [email protected], which counts a prediction as correct if $\mathrm{IoU}(B_{\text{pred}}, B_{\text{gt}}) > 0.5$ (Hao et al., 23 Jun 2025).

Retrieval is measured by Recall@k,

$$\mathrm{Recall}@k = \frac{1}{N}\sum_{i=1\ldots N} \mathbf{1}[\,GT(i)\in \mathrm{Top}\text{-}k(i)\,].$$

The benchmark’s defining metric is the joint retrieval-and-localization measure BoxRecall@k at IoU threshold $\tau$:

$$\mathrm{BoxRecall}@k(\tau) = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\,\mathrm{Inst}_i \in \mathrm{Top}\text{-}k(i)\ \wedge\ \mathrm{IoU}(B_i^{\mathrm{pred}}, B_i^{\mathrm{gt}})>\tau\,].$$

Here, $\mathrm{Top}\text{-}k(i)$ ranks object candidates across the gallery by similarity to the query, and the prediction is counted only if the correct instance is retrieved and localized above threshold (Hao et al., 23 Jun 2025).

This metric design is significant because it prevents a system from receiving full credit for finding the correct image while localizing the wrong object, or for localizing well in an incorrectly retrieved image. A plausible implication is that REIRCOCO enforces a stricter notion of end-to-end correctness than either TIR or REC alone.

4. CLARE: the benchmark’s end-to-end baseline

The benchmark paper introduces CLARE, short for Contrastive Language-Instance Alignment with Relation Experts, as a strong end-to-end baseline (Hao et al., 23 Jun 2025). CLARE uses a dual-stream architecture.

The text branch applies a SigLIP text encoder to produce a global sentence embedding $F_t$, then refines it with MORE (Mix of Relation Experts) to obtain a relation-aware text embedding $\{x_{\text{img}}^i\}_{i=1\ldots N}$0. The image branch uses a SigLIP vision encoder, followed by a Deformable-DETR-like object extractor with 6 encoder + 6 decoder layers that outputs $\{x_{\text{img}}^i\}_{i=1\ldots N}$1 proposals and instance-level features $\{x_{\text{img}}^i\}_{i=1\ldots N}$2 (Hao et al., 23 Jun 2025).

MORE contains two expert types. Shared experts are always active and model general semantics; routed experts specialize in spatial, relational, and comparative reasoning. The text representation is formed as

$\{x_{\text{img}}^i\}_{i=1\ldots N}$3

with $\{x_{\text{img}}^i\}_{i=1\ldots N}$4 shared expert, $\{x_{\text{img}}^i\}_{i=1\ldots N}$5 routed experts, and Top-2 selection per query in the best-performing configuration (Hao et al., 23 Jun 2025).

Cross-modal matching is driven by the Contrastive Language-Instance Alignment (CLIA) objective. The paper also presents an InfoNCE-style example,

$\{x_{\text{img}}^i\}_{i=1\ldots N}$6

but the actual training loss is a Sigmoid-based CLIA extending SigLIP to instance-level pairs across images:

$\{x_{\text{img}}^i\}_{i=1\ldots N}$7

This casts expression–object matching as binary classification over all cross-image pairs, enabling gallery-wide negative sampling (Hao et al., 23 Jun 2025).

Training proceeds in two stages. Stage 1 is grounding pretraining on COCO detection and RefCOCO/+/g REC:

$\{x_{\text{img}}^i\}_{i=1\ldots N}$8

Stage 2 is REIRCOCO finetuning:

$\{x_{\text{img}}^i\}_{i=1\ldots N}$9

The focal term is

$B=[x,y,w,h]$0

with $B=[x,y,w,h]$1 and $B=[x,y,w,h]$2, while the box regression term combines $B=[x,y,w,h]$3 and GIoU losses (Hao et al., 23 Jun 2025).

At inference time, CLARE precomputes and caches instance features for all gallery images, encodes the query once, scores all instances by the dot product $B=[x,y,w,h]$4, ranks candidates, and predicts or refines the box for the top match using the DETR head. This directly addresses the inefficiency of applying REC independently to every image in the gallery (Hao et al., 23 Jun 2025).

5. Empirical results and ablations

On REIRCOCO, CLARE substantially outperforms two-stage TIR+REC baselines across IoU thresholds. For CLARE-ViT-B, the paper reports BR@1/5/10 of 28.47 / 49.50 / 56.85 at $B=[x,y,w,h]$5, 25.96 / 42.85 / 48.57 at $B=[x,y,w,h]$6, and 19.38 / 31.53 / 35.48 at $B=[x,y,w,h]$7. For CLARE-ViT-L, the corresponding results are 29.53 / 50.56 / 56.98, 28.03 / 47.07 / 53.34, and 21.12 / 35.80 / 40.35. A best two-stage baseline such as SigLIP-ViT-L + DeepSeek-VL2 reaches only 18.52 / 33.22 / 39.48 at $B=[x,y,w,h]$8 (Hao et al., 23 Jun 2025).

The paper argues that REC alone is unsuitable for REIR for two explicit reasons. First, pairing each query with every image scales as $B=[x,y,w,h]$9 forward passes, illustrated by 25M calls for 5k queries $$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$0 5k images. Second, REC outputs are not calibrated across images, so gallery ranking is impossible (Hao et al., 23 Jun 2025).

Ablation studies identify three components as especially important. Two-stage training is necessary: Pretrain only yields BR@10 = 18.01, Finetune only yields 39.83, and Pretrain $$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$1 Finetune yields 53.28 at $$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$2. MORE routing improves BR@10 from 53.28 without routed experts to 56.85 with $$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$3, Top-2 active. CLIA is critical: without CLIA, BR@10 = 21.24; with mini-batch CLIA, 48.62; with full CLIA, 53.28. The comparison between a focal-style alternative and SigLIP-style CLIA gives 52.04 versus 53.28, favoring CLIA (Hao et al., 23 Jun 2025).

The baseline also shows transfer beyond REIR itself. On REIRCOCO viewed as an image retrieval problem, CLARE-ViT-L achieves R@1/5/10 = 36.40 / 65.76 / 75.99, compared with 23.25 / 43.10 / 52.01 for SigLIP-ViT-L-384. On conventional REC benchmarks, CLARE is described as competitive; for example, the paper reports 91.40 on RefCOCO val with ViT-L under [email protected] (Hao et al., 23 Jun 2025).

6. Relation to adjacent benchmarks and naming issues

REIRCOCO is closely related to, but distinct from, the RefCOCO family. Work on Interactive Referring Expression Generation operates on RefCOCO, RefCOCO+, and RefCOCOg and explicitly states that it does not introduce any dataset named “REIRCOCO”; its experiments remain within the standard RefCOCO-family benchmarks (Ye et al., 2023). Likewise, subsequent work on zero-shot REC over RefCOCO, RefCOCO+, and RefCOCOg treats those datasets as REC benchmarks derived from MS COCO and does not redefine them as REIRCOCO (Liu et al., 12 Sep 2025).

The distinction matters methodologically. REC assumes the image is already known and predicts a box inside that image. REIR, as instantiated by REIRCOCO, must search across a gallery and jointly rank and localize. This difference is why REIRCOCO introduces BoxRecall@k rather than relying only on REC accuracy or image retrieval recall (Hao et al., 23 Jun 2025).

A common misconception is that REIRCOCO is merely RefCOCO used with a different evaluation script. The benchmark paper rejects that interpretation indirectly by defining a new task, a new dataset construction pipeline, instance-level gallery annotations, and joint retrieval-localization metrics. A plausible implication is that models optimized only for REC or only for TIR will generally be misaligned with REIRCOCO’s evaluation target unless they explicitly support gallery-wide instance ranking.

7. Limitations, release status, and research directions

The benchmark paper identifies several limitations. Although REIRCOCO emphasizes attributes and relations, it does not report exact quantitative distributions such as average expression length or relation-type histograms. Complex relational constructs remain difficult, especially in cluttered scenes with multiple similar objects, subtle relation keywords, rare compositions, small or occluded instances, and overlapping objects. The generation-and-filtering procedure uses captions and structured metadata, but DeepSeek R1 does not ingest pixels, so ambiguity filtering is not vision-grounded in the strictest sense (Hao et al., 23 Jun 2025).

The authors state that they will publicly release code and the REIRCOCO benchmark, but license details and a final download URL will be provided upon release (Hao et al., 23 Jun 2025). For reproduction, the paper reports training on 8× NVIDIA A800 80GB GPUs, with SigLIP ViT-B and ViT-L backbones, AdamW, an initial learning rate of $$\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},$$4, weight decay 0.05 in pretraining and 0.0001 in finetuning, and multi-scale resizing with short side 480–800 and long side at most 1333 (Hao et al., 23 Jun 2025).

Future extensions named in the paper include harder compositional expressions, broader category coverage beyond the 80 COCO classes, open-vocabulary targets such as LVIS/Objects365, improved handling of small or heavily occluded objects, and stronger vision-aware or human-in-the-loop validation. These directions follow directly from the benchmark’s core objective: enabling scalable research on gallery-wide, instance-specific, language-conditioned retrieval and localization (Hao et al., 23 Jun 2025).