Referring Expression Instance Retrieval

• Referring Expression Instance Retrieval (REIR) is a vision-and-language task that retrieves images and precisely localizes target object instances from large unstructured galleries using natural language.
• REIR unifies text-image retrieval and referring expression comprehension to support fine-grained, instance-level search in diverse applications such as surveillance, forensics, and personal photo search.
• Key methodologies include dual-stream architectures, contrastive alignment losses, and relation expert routing modules that improve both accuracy and scalability across large-scale datasets.

Referring Expression Instance Retrieval (REIR) is a vision-and-language task that aims to retrieve both the correct image and the precise object instance within it, based on a natural language referring expression. REIR unifies and extends classical Text-Image Retrieval (TIR) and Referring Expression Comprehension (REC) by supporting fine-grained, instance-level search and localization over large unstructured galleries, using natural language that typically refers to specific attributes, context, or relationships.

1. Task Definition and Motivation

REIR arises from the limitations of established tasks:

• Text-Image Retrieval (TIR) retrieves images from a gallery using global captions, but struggles with fine-grained object queries or diverse scenes.
• Referring Expression Comprehension (REC) localizes an object given an expression, but is formulated within a single image and cannot search large galleries.

REIR is defined as: given a free-form, instance-level referring expression, the system must retrieve the image containing the target and also localize (e.g., with a bounding box or mask) the correct object instance within that image (2506.18246).

This unified setting better reflects requirements in real-world applications such as surveillance (finding an individual given a description in a large video archive), forensics (identifying a specific object in massive datasets), or personal photo search (locating “my daughter on a pony in a red dress” in a large collection).

2. Benchmark Datasets

Progress in REIR necessitates large-scale, high-quality datasets that provide fine-grained natural language referring expressions anchored to specific object instances across thousands of images:

• REIRCOCO (2506.18246) is a benchmark specifically constructed for REIR. It is based on MSCOCO detection annotations and the RefCOCO family (RefCOCO, RefCOCO+, and RefCOCOg).
• To generate expressions, advanced vision-LLMs (e.g., GPT-4o) are prompted with structured object and scene information to create multiple diverse, unambiguous expressions per object.
• A secondary LLM-based filtration stage reviews and removes ambiguous or inaccurate expressions, yielding a corpus of over 200,000 high-quality, uniquely grounded descriptions for more than 30,000 images.
• Each instance in REIRCOCO is annotated with bounding box coordinates and at least one referring expression, enabling both retrieval and localization evaluation.

The construction pipeline ensures coverage of diverse categories, spatial contexts, and linguistic styles while supporting rigorous evaluation of both sub-tasks (retrieval and localization).

3. End-to-End Baseline: The CLARE Architecture

A representative baseline model for REIR is CLARE (Contrastive Language-Instance Alignment with Relation Experts) (2506.18246). Its key components are:

• Dual-Stream Architecture: Separate text and visual encoders process the referring expression and candidate object regions, respectively.
  • The textual branch embeds the query using a transformer encoder and the Mix Of Relation Experts (MORE) module, which dynamically activates specialized “experts” to process different types of relational or attribute information. The final sentence embedding is a sum of outputs from shared and routed expert MLPs, governed by a gating network.
  • The visual branch uses a vision encoder (SigLIP backbone) and Deformable-DETR to produce instance-level object features, encoding both appearance and relational context.
• Contrastive Language-Instance Alignment (CLIA): A contrastive learning objective aligns paired referring-expression and instance features, such that positives (correct instance-expression pairs) are closer than negatives. The CLIA loss extends the SigLIP sigmoid-based loss to every instance in every image of a batch.
• Relation Expert Routing: The MORE module allows the model to dynamically select which specialized expert networks to use for a given expression-instance pair. These experts are trained to model different facets, such as object attributes or relations with other objects, improving fine-grained discrimination.

The training process occurs in two stages. First, grounding capabilities are developed via pretraining on detection and REC data, and then REIR performance is optimized via CLIA and bounding box regression losses.

4. Algorithmic and Theoretical Foundations

REIR spans both formal logic-based and neural approaches:

• Logic-based perspectives: Early work in generating and distinguishing referring expressions (1006.4621) analyzes the expressive power of logical languages. Theoretical results specify how expressiveness, model structure, and computational resources interact to determine when unique instance descriptions can be constructed efficiently—critical for scalable REIR systems.
  • Algorithms based on simulator sets and graph substructure search provide polynomial-time procedures under certain constraints, but face exponential growth of expression size in highly expressive logics.
• Neural and contrastive models: Modern REIR systems employ contrastive alignment losses for cross-modal training and integrate advanced context modeling through attention, graph convolutional networks, and region-level feature propagation.

The dual challenge of balancing expressiveness (ability to generate/understand discriminative references) and computational tractability (scalability with gallery size and linguistic diversity) underpins algorithmic choices in REIR design.

5. Evaluation Metrics and Empirical Results

Assessment of REIR methods requires metrics that jointly capture retrieval and localization performance:

• BoxRecall@k: The proportion of queries for which the correct instance (within the correct image, at sufficient IoU overlap) appears within the top-k ranked results. This metric unifies Recall@k for image retrieval and bounding box precision for localization.
• Experiments demonstrate that end-to-end REIR architectures such as CLARE generally outperform two-stage methods (retrieving images before running REC), especially on both fine-grained queries and cluttered galleries (2506.18246).
• Additional analysis may include accuracy stratified by expression type, object category, and complexity of relational reasoning required.

6. Practical Implications, Challenges, and Future Directions

Practical deployment of REIR systems faces multiple challenges and avenues for development:

• Relational and Compositional Language: Many referring expressions require resolving relationships among objects (e.g., “the woman holding a green umbrella next to the blue car”). Models must represent not only object attributes but also compositional spatial and semantic relations (1906.04464, 2506.18246).
• Generalization and Ambiguity: Expressions may be ambiguous, compositional, or context-dependent, especially in open-world settings. Future benchmarks and models will require robust handling of rare compositions and ambiguous queries.
• Dataset Bias and Evaluation: Empirical studies show that high apparent performance sometimes masks reliance on dataset biases or shallow cues (1805.11818), necessitating careful dataset construction and adversarial testing.
• Efficiency: Scalability to galleries of millions of images/instances and real-time inference efficiency are practical concerns, motivating continued research into efficient instance proposal generation, indexing, and negative sampling for contrastive learning.
• Interactive and Joint Learning: The potential for interactive frameworks—where expression generation (REG) and comprehension (REC) components communicate and jointly optimize for retrieval—and dialog-based disambiguation represent promising directions, especially when extended to multimodal conversational agents (2308.09977).
• Application Domains: REIR supports use cases in surveillance, forensics, robotics, content-based image search, and accessible technologies, where instance-level accuracy and interpretability are critical.

7. Summary Table: REIR vs. Prior Tasks

Task	Query Scope	Response	Evaluates
Text-Image Retrieval (TIR)	Image-level	Images	Recall@k
Referring Expression Comprehension	Within single image	Object instance (location)	Accuracy (IoU)
Referring Expression Instance Retrieval (REIR)	Instance-level (gallery-wide)	(Image, Instance Location)	BoxRecall@k

In summary, REIR formalizes and operationalizes the joint retrieval and grounding of visual instances from large-scale galleries using fine-grained natural language. The REIRCOCO benchmark and models such as CLARE establish strong baselines, though challenges of relational reasoning, ambiguity, and scalability invite further research. Equations and modules for contrastive alignment, relation expert routing, and dual-stream modeling are central to state-of-the-art performance (2506.18246, 1006.4621).