Instance-Level Composed Image Retrieval

Abstract: The progress of composed image retrieval (CIR), a popular research direction in image retrieval, where a combined visual and textual query is used, is held back by the absence of high-quality training and evaluation data. We introduce a new evaluation dataset, i-CIR, which, unlike existing datasets, focuses on an instance-level class definition. The goal is to retrieve images that contain the same particular object as the visual query, presented under a variety of modifications defined by textual queries. Its design and curation process keep the dataset compact to facilitate future research, while maintaining its challenge-comparable to retrieval among more than 40M random distractors-through a semi-automated selection of hard negatives. To overcome the challenge of obtaining clean, diverse, and suitable training data, we leverage pre-trained vision-and-LLMs (VLMs) in a training-free approach called BASIC. The method separately estimates query-image-to-image and query-text-to-image similarities, performing late fusion to upweight images that satisfy both queries, while down-weighting those that exhibit high similarity with only one of the two. Each individual similarity is further improved by a set of components that are simple and intuitive. BASIC sets a new state of the art on i-CIR but also on existing CIR datasets that follow a semantic-level class definition. Project page: https://vrg.fel.cvut.cz/icir/.

• The paper introduces an instance-centric CIR evaluation dataset that resolves annotation ambiguities by focusing on physical object instances under diverse modifications.
• The paper presents FreeDom, a training-free baseline leveraging pretrained VLMs with bias removal, semantic subspace projection, and multiplicative fusion to achieve a macro-mAP of 31.6%.
• The paper demonstrates that careful dataset curation and explicit modality treatment can yield competitive, scalable retrieval performance without the need for retraining.

Instance-Level Composed Image Retrieval: Methods and Benchmarking

Introduction and Motivation

The paper "Instance-Level Composed Image Retrieval" (2510.25387) addresses significant limitations in the evaluation and development of composed image retrieval (CIR), where the objective is to retrieve database images matching both a query image and an associated textual modification. Previous CIR datasets largely operate at a class/semantic level, suffer from ambiguous positive pairs, and are vulnerable to unimodal shortcuts. This work introduces an instance-centric CIR evaluation protocol and dataset, providing a strong empirical baseline named FreeDom—a training-free method leveraging pretrained vision-LLMs (VLMs) with principled modality fusion.

Dataset Design: Instance-Level CIR

The central contribution is the construction of an instance-level CIR dataset, focusing on retrieval of images depicting the same physical instance of an object (e.g., a particular landmark or product), under diverse textual modifications (e.g., "at sunset", "from an aerial viewpoint"). The dataset curation process involves selection of meaningful modifications per instance, manual verification of positive images, and systematic mining of hard negatives: visual (matching the instance, not the modifier), textual (matching the modification, not the instance), and composed (almost—but not fully—matching both).

The design facilitates precise evaluation by:

• Preventing annotation ambiguity and false negatives,
• Including multiple, challenging negative types,
• Supporting photo-real and context-shifted retrieval scenarios,
• Balancing compactness (average 3.7K images/database) and retrieval difficulty (equivalent to >40M random distractors in LAION, as demonstrated experimentally).

FreeDom: A Training-Free, Strong Baseline for CIR

FreeDom is built to exploit pretrained VLMs such as CLIP. It dispenses with labor-intensive or synthetic triplet mining, instead computing and fusing modality-specific similarities (image-to-image and text-to-image) directly on frozen representations. The key technical innovations and system components are as follows:

• Bias Removal via Centering: Mean feature subtraction mitigates modality-specific distributional bias, computed on large external datasets (e.g., LAION for vision, a curated concept corpus for text).
• Semantic Subspace Projection: Image embeddings are projected onto a subspace optimized to retain object-specific (not style or context) information. This subspace is constructed from textual representations with positive and negative corpora, using top-k eigenvectors of a contrastive covariance matrix. This step enhances instance discrimination and modifier-invariance.
• Textual Query Contextualization: To bridge CLIP's pretraining mismatch (full captions versus partial modifier text), queries are augmented using a subject corpus, forming caption-like queries for more robust text embedding.
• Score Normalization and Fusion: Modal scores are min-normalized (to account for range imbalances), then multiplied (logical AND) and regularized with a Harris-corner-inspired criterion to penalize results only strong in a single modality.

  Figure 2: Overview of the FreeDom pipeline, including centering, semantic projection, text contextualization, modality scoring, and multiplicative Harris-regularized fusion.

All these operations are efficient and query-side, requiring no re-embedding or backpropagation on the database side and thus are suitable for large-scale, real-world image retrieval systems.

Experimental Results and Empirical Analysis

The authors provide extensive benchmarking across their instance-level CIR dataset and multiple class-level legacy datasets. FreeDom consistently demonstrates superior or competitive performance compared to a suite of strong baselines and published state-of-the-art methods, including training-based, pseudo-triplet, and zero-shot approaches.

Key empirical findings include:

• Substantial absolute performance gains: FreeDom achieves macro-mAP of 31.6% on the proposed instance-level task, which notably outperforms all prior zero-shot and training-based CIR methods, including large-scale generative approaches and domain-specialized models.
• Strong per-category robustness: FreeDom outperforms specialized methods across most object and modifier categories, with particularly pronounced margins on highly compositional instances (fictional, product, technology).
• Compact-yet-hard benchmark: The challenge of the dataset, as empirically evidenced, is highly competitive—a 3.7K-image hard-negative database is as difficult as 40M random images from LAION for strong unimodal fusion baselines.
• Genuine modality composition: Experiments sweeping the text/image fusion weight show the proposed dataset yields synergistic gains for multimodal fusion, whereas legacy datasets often favor text-only matching.
• Transparent, cumulative ablation: Each FreeDom component, from centering and projection to regularization and contextualization, is shown to incrementally and robustly enhance macro-mAP, affirming the necessity of multi-faceted representation treatment in practical CIR.

Hyperparameter Sensitivity

The paper presents an analysis of FreeDom's robustness to its core hyperparameters: the contrastive scaling parameter ($\alpha$) in semantic projection, the number of PCA components ($k$), and the Harris regularization weight ($\lambda$). Results indicate broad plateaus, supporting the method's transferability and low risk of overfitting.

Figure 4: Sensitivity of FreeDom to hyperparameters: variation of mAP with respect to $\alpha$, $k$, and $\lambda$ across multiple datasets.

Implications and Future Considerations

The contributions of this paper have implications for both practical application and theoretical development in multimodal retrieval:

• Benchmark design: By addressing annotation ambiguity, hard-negative construction, and instance granularity, the dataset sets a new standard for CIR evaluation, discouraging unimodal shortcuts and supporting broader modifier diversity.
• Training-free methods: FreeDom's architecture, relying on analytically derived projection and centering/normalization, suggests that retraining or finetuning large VLMs is not strictly necessary to attain strong CIR accuracy, especially when modal representations are explicitly treated for bias and compositionality.
• Transfer and customization: The use of external corpora for semantic subspace projection opens further avenues for domain adaptation and task-specific tailoring in retrieval pipelines without full retraining.

Potential developments include leveraging generative methods to expand the coverage of rare modifier-instance pairs, adapting projection and contextualization for video or temporally evolving object instances, and integrating learned fusion mechanisms that extend beyond AND-based logic, especially in more entangled textual-visual tasks.

Conclusion

"Instance-Level Composed Image Retrieval" (2510.25387) advances CIR by formalizing an instance-centric evaluation benchmark featuring challenging negatives and by proposing FreeDom, a simple yet surprisingly effective, training-free CIR baseline that sets a new empirical standard. This work highlights the importance of data curation and explicit modality treatment for evaluating and developing VLM-based retrieval systems, and provides a principled foundation for compositional retrieval in both research and applied multimedia scenarios.