---
title: 'REIRCOCO: Benchmark for Instance Retrieval'
url: https://www.emergentmind.com/topics/reircoco
type: topic
---

# REIRCOCO: Benchmark for Instance Retrieval

Searching arXiv for REIRCOCO and closely related papers to ground the article.
REIRCOCO is a large-scale benchmark for **Referring Expression Instance Retrieval (REIR)**, a vision–language task in which a system receives a natural-language referring expression and must search a gallery of images to both retrieve the correct image and localize the target object instance with a bounding box. It was introduced together with the task formalization and a strong end-to-end baseline in "Referring Expression Instance Retrieval and A Strong End-to-End Baseline" [2506.18246]. The benchmark is positioned between **Text–Image Retrieval (TIR)**, which retrieves images from image-level descriptions, and **Referring Expression Comprehension (REC)**, which localizes objects inside a known image. REIRCOCO addresses the regime in which queries are instance-level, gallery-wide, and explicitly require both ranking and localization.

## 1. Definition and task formulation

REIR is defined as a unified retrieval-and-grounding problem: given a query $x_{\text{text}}$ and a gallery $\{x_{\text{img}}^i\}_{i=1\ldots N}$, the model outputs a bounding box $B=[x,y,w,h]$ that must lie in the unique image containing the referred instance [2506.18246]. This formulation departs from two established tasks.

**Text–Image Retrieval** ranks images from an image-level caption, but does not return an object-level location and often fails on fine-grained, instance-specific queries. **Referring Expression Comprehension** localizes an instance in a single known image, but does not scale to gallery-wide search and lacks a cross-image ranking mechanism. The motivation for REIR is that real applications such as surveillance, forensics, and photo search require both instance-level retrieval and precise localization across large galleries, with expressions rich in attributes and relations such as “the brown dog lies on a cushion near the fireplace with a red rug underneath” [2506.18246].

A central claim of the benchmark paper is that naïvely composing TIR and REC is insufficient. TIR is too coarse for object-level semantics, while REC-style early-fusion pipelines are computationally prohibitive across galleries and produce scores that are not comparable between images. This suggests that REIR is not merely a combination of prior tasks, but a separate evaluation setting with its own benchmark design and retrieval-grounding metrics [2506.18246].

## 2. Benchmark construction and corpus characteristics

REIRCOCO is constructed from **MS COCO Detection** and the **RefCOCO/RefCOCO+/RefCOCOg** family. COCO contributes dense instance boxes and category labels across 80 object categories, while the RefCOCO variants motivate coverage of relational semantics through human-written short expressions [2506.18246].

The benchmark uses a **Generation–Filtering pipeline**. In the generation stage, **GPT-4o** is prompted with structured information for each annotated instance: the instance bounding box and category, the image’s global caption from MS COCO Captions when available, and contextual information about nearby objects and scene layout. GPT-4o is instructed to generate **5–10 diverse, context-rich referring expressions per instance** that emphasize appearance, attributes, and relations, avoid absolute coordinates, and ensure discriminative uniqueness across the gallery. If an instance is too small or occluded, the prescribed output is the placeholder **“Instance quality is poor.”** In the filtering stage, **DeepSeek R1** verifies expression quality from the caption, candidate expressions, and structured metadata, rejecting ambiguous, inaccurate, unverifiable, or non-discriminative expressions [2506.18246].

The resulting dataset statistics are as follows.

| Item | Value |
|---|---|
| Images | 30,106 |
| Object instances | 215,835 |
| Verified referring expressions | 613,548 |

The benchmark covers **80 MS COCO object classes** and is designed so that expressions are **unambiguous and uniquely tied to a single instance**, with strong emphasis on attributes and relations. The paper qualitatively stresses relational richness, including spatial context and interactions, but does **not** report average expression length or exact distributional summaries. It evaluates on standardized train/val/test splits aligned with COCO/RefCOCO images, but the specific split counts are **not enumerated** [2506.18246].

Annotations are instance-centered. A typical JSON entry contains `image_id`, `instance_id`, `category_id`, `bbox`, and a list of verified `expressions`. The benchmark paper gives a concrete example for image `391895`, instance `391895_07`, category `18`, with bounding box `[472, 210, 85, 64]` and three expressions describing a brown dog by appearance and relation to the fireplace, rug, and iron grate [2506.18246].

## 3. Evaluation protocol and metrics

REIRCOCO evaluates both retrieval quality and localization quality. Localization follows standard REC practice through **Intersection-over-Union**,
$$
\mathrm{IoU}(B_p, B_g) = \frac{|B_p \cap B_g|}{|B_p \cup B_g|},
$$
and **Precision@0.5**, which counts a prediction as correct if $\mathrm{IoU}(B_{\text{pred}}, B_{\text{gt}}) > 0.5$ [2506.18246].

Retrieval is measured by **Recall@k**,
$$
\mathrm{Recall}@k = \frac{1}{N}\sum_{i=1\ldots N} \mathbf{1}[\,GT(i)\in \mathrm{Top}\text{-}k(i)\,].
$$
The benchmark’s defining metric is the joint retrieval-and-localization measure **BoxRecall@k** at IoU threshold $\tau$:
$$
\mathrm{BoxRecall}@k(\tau) = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\,\mathrm{Inst}_i \in \mathrm{Top}\text{-}k(i)\ \wedge\ \mathrm{IoU}(B_i^{\mathrm{pred}}, B_i^{\mathrm{gt}})>\tau\,].
$$
Here, $\mathrm{Top}\text{-}k(i)$ ranks object candidates across the gallery by similarity to the query, and the prediction is counted only if the correct instance is retrieved and localized above threshold [2506.18246].

This metric design is significant because it prevents a system from receiving full credit for finding the correct image while localizing the wrong object, or for localizing well in an incorrectly retrieved image. A plausible implication is that REIRCOCO enforces a stricter notion of end-to-end correctness than either TIR or REC alone.

## 4. CLARE: the benchmark’s end-to-end baseline

The benchmark paper introduces **CLARE**, short for **Contrastive Language-Instance Alignment with Relation Experts**, as a strong end-to-end baseline [2506.18246]. CLARE uses a **dual-stream architecture**.

The **text branch** applies a **SigLIP text encoder** to produce a global sentence embedding $F_t$, then refines it with **MORE (Mix of Relation Experts)** to obtain a relation-aware text embedding $T \in \mathbb{R}^D$. The **image branch** uses a **SigLIP vision encoder**, followed by a **Deformable-DETR-like object extractor** with **6 encoder + 6 decoder layers** that outputs **$M=900$ proposals** and instance-level features $\{O_j\}\in\mathbb{R}^D$ [2506.18246].

MORE contains two expert types. **Shared experts** are always active and model general semantics; **routed experts** specialize in spatial, relational, and comparative reasoning. The text representation is formed as
$$
T = \sum_{j=1\ldots N_s}\mathrm{FFN}^{(s)}_j(F_t) + \sum_{k=1\ldots N_r}\hat{g}_k \cdot \mathrm{FFN}^{(r)}_k(F_t),
$$
with **$N_s=1$ shared expert**, **$N_r=4$ routed experts**, and **Top-2 selection per query** in the best-performing configuration [2506.18246].

Cross-modal matching is driven by the **Contrastive Language-Instance Alignment (CLIA)** objective. The paper also presents an InfoNCE-style example,
$$
L_{\mathrm{CLIA\text{-}InfoNCE}} = - \log \frac{\exp(s(q,v^+)/\tau)}{\sum_j \exp(s(q,v_j)/\tau)},
$$
but the actual training loss is a **Sigmoid-based CLIA** extending SigLIP to instance-level pairs across images:
$$
L_{\mathrm{CLIA}} = - \frac{1}{B\cdot N} \sum_{i=1}^{B}\sum_{k=1}^{B}\sum_{l=1}^{N_k}
\log\!\left(\frac{1}{1+\exp\!\big(z_{i,k,l}(-t\langle T_i,O_k^l\rangle+b)\big)}\right).
$$
This casts expression–object matching as binary classification over all cross-image pairs, enabling gallery-wide negative sampling [2506.18246].

Training proceeds in two stages. **Stage 1** is grounding pretraining on COCO detection and RefCOCO/+/g REC:
$$
L_{\mathrm{pretrain}} = L_{\mathrm{focal}} + L_{\mathrm{bbox}}.
$$
**Stage 2** is REIRCOCO finetuning:
$$
L_{\mathrm{finetune}} = L_{\mathrm{CLIA}} + L_{\mathrm{focal}} + L_{\mathrm{bbox}}.
$$
The focal term is
$$
L_{\mathrm{focal}}(t) = -\alpha_t(1-t)^\gamma \log(t),
$$
with $\gamma=2$ and $\alpha_t=0.25$, while the box regression term combines $L_1$ and GIoU losses [2506.18246].

At inference time, CLARE precomputes and caches instance features for all gallery images, encodes the query once, scores all instances by the dot product $\langle T,O_j\rangle$, ranks candidates, and predicts or refines the box for the top match using the DETR head. This directly addresses the inefficiency of applying REC independently to every image in the gallery [2506.18246].

## 5. Empirical results and ablations

On REIRCOCO, CLARE substantially outperforms two-stage TIR+REC baselines across IoU thresholds. For **CLARE-ViT-B**, the paper reports **BR@1/5/10** of **28.47 / 49.50 / 56.85** at $\tau=0.5$, **25.96 / 42.85 / 48.57** at $\tau=0.7$, and **19.38 / 31.53 / 35.48** at $\tau=0.9$. For **CLARE-ViT-L**, the corresponding results are **29.53 / 50.56 / 56.98**, **28.03 / 47.07 / 53.34**, and **21.12 / 35.80 / 40.35**. A best two-stage baseline such as **SigLIP-ViT-L + DeepSeek-VL2** reaches only **18.52 / 33.22 / 39.48** at $\tau=0.5$ [2506.18246].

The paper argues that REC alone is unsuitable for REIR for two explicit reasons. First, pairing each query with every image scales as $N\times N$ forward passes, illustrated by **25M calls for 5k queries $\times$ 5k images**. Second, REC outputs are not calibrated across images, so gallery ranking is impossible [2506.18246].

Ablation studies identify three components as especially important. **Two-stage training** is necessary: **Pretrain only** yields **BR@10 = 18.01**, **Finetune only** yields **39.83**, and **Pretrain $\rightarrow$ Finetune** yields **53.28** at $\tau=0.5$. **MORE routing** improves **BR@10** from **53.28** without routed experts to **56.85** with **$N_r=4$, Top-2 active**. **CLIA** is critical: **without CLIA**, **BR@10 = 21.24**; with **mini-batch CLIA**, **48.62**; with **full CLIA**, **53.28**. The comparison between a focal-style alternative and SigLIP-style CLIA gives **52.04** versus **53.28**, favoring CLIA [2506.18246].

The baseline also shows transfer beyond REIR itself. On REIRCOCO viewed as an image retrieval problem, **CLARE-ViT-L** achieves **R@1/5/10 = 36.40 / 65.76 / 75.99**, compared with **23.25 / 43.10 / 52.01** for **SigLIP-ViT-L-384**. On conventional REC benchmarks, CLARE is described as competitive; for example, the paper reports **91.40** on **RefCOCO val** with **ViT-L** under **Precision@0.5** [2506.18246].

## 6. Relation to adjacent benchmarks and naming issues

REIRCOCO is closely related to, but distinct from, the **RefCOCO** family. Work on **Interactive Referring Expression Generation** operates on **RefCOCO, RefCOCO+, and RefCOCOg** and explicitly states that it does **not** introduce any dataset named “REIRCOCO”; its experiments remain within the standard RefCOCO-family benchmarks [2308.09977]. Likewise, subsequent work on zero-shot REC over RefCOCO, RefCOCO+, and RefCOCOg treats those datasets as REC benchmarks derived from MS COCO and does not redefine them as REIRCOCO [2509.09958].

The distinction matters methodologically. **REC** assumes the image is already known and predicts a box inside that image. **REIR**, as instantiated by REIRCOCO, must search across a gallery and jointly rank and localize. This difference is why REIRCOCO introduces **BoxRecall@k** rather than relying only on REC accuracy or image retrieval recall [2506.18246].

A common misconception is that REIRCOCO is merely RefCOCO used with a different evaluation script. The benchmark paper rejects that interpretation indirectly by defining a new task, a new dataset construction pipeline, instance-level gallery annotations, and joint retrieval-localization metrics. A plausible implication is that models optimized only for REC or only for TIR will generally be misaligned with REIRCOCO’s evaluation target unless they explicitly support gallery-wide instance ranking.

## 7. Limitations, release status, and research directions

The benchmark paper identifies several limitations. Although REIRCOCO emphasizes attributes and relations, it does **not** report exact quantitative distributions such as average expression length or relation-type histograms. Complex relational constructs remain difficult, especially in cluttered scenes with multiple similar objects, subtle relation keywords, rare compositions, small or occluded instances, and overlapping objects. The generation-and-filtering procedure uses captions and structured metadata, but **DeepSeek R1 does not ingest pixels**, so ambiguity filtering is not vision-grounded in the strictest sense [2506.18246].

The authors state that they **will publicly release code and the REIRCOCO benchmark**, but **license details and a final download URL will be provided upon release** [2506.18246]. For reproduction, the paper reports training on **8× NVIDIA A800 80GB GPUs**, with **SigLIP ViT-B and ViT-L backbones**, **AdamW**, an initial learning rate of **$2\times10^{-4}$**, weight decay **0.05** in pretraining and **0.0001** in finetuning, and multi-scale resizing with short side **480–800** and long side at most **1333** [2506.18246].

Future extensions named in the paper include harder compositional expressions, broader category coverage beyond the **80 COCO classes**, open-vocabulary targets such as **LVIS/Objects365**, improved handling of small or heavily occluded objects, and stronger vision-aware or human-in-the-loop validation. These directions follow directly from the benchmark’s core objective: enabling scalable research on gallery-wide, instance-specific, language-conditioned retrieval and localization [2506.18246].

Source: https://www.emergentmind.com/topics/reircoco