---
title: Composed Image Retrieval (CIR)
url: https://www.emergentmind.com/topics/composed-image-retrieval-cir
type: topic
---

# Composed Image Retrieval (CIR)

Composed Image Retrieval (CIR) is a multimodal retrieval paradigm in which the system retrieves a target image in response to a user query composed of a reference image and a natural language modification text. CIR enables more precise and interactive image search by supporting queries that express detailed modifications to the reference, addressing the limitations of single-modality (image- or text-only) retrieval. CIR research draws on and advances multi-modal representation learning, vision-language pretraining, and cross-modal fusion methodologies.

## 1. Task Definition and Motivation

CIR is formally defined as: Given a reference image $I_r$ and a modification text $T_m$ (also called the relative caption), retrieve the target image $I_t$ such that $I_t$ is visually similar to $I_r$ but also satisfies the modifications specified in $T_m$. This search protocol allows users to start with a specific instance (e.g., a photo of an item) and then articulate desired changes (e.g., “make this dress red and sleeveless”). CIR has direct applications in e-commerce product search, creative content retrieval, and interactive digital assistants.

The CIR formulation supports much finer control than traditional visual search, which typically retrieves visually similar images without incorporating user's editing intents, and goes beyond text-only retrieval, which often fails to capture the nuances of visual reference and grounded semantics.

## 2. Methodological Taxonomy

CIR models are systematically categorized across several dichotomies and architectural axes [2502.18495]:

**Supervised CIR**: Models trained on triplet data consisting of $(I_r, T_m, I_t)$ annotated examples.

- **Feature Extraction**: Early approaches utilized separate CNNs/RNNs [2502.18495]. Recent methods adopt Vision-Language Pre-trained (VLP) models such as CLIP or BLIP for unified multimodal representation extraction.
- **Image–Text Fusion**: Fusion strategies are commonly classified as:
    - *Explicit combination-based*: Linear or MLP-based transformation of feature concatenations; residual editing [2309.01366, 2310.05473].
    - *Neural network–based*: Cross-attention modules, self-attention, or graph-attention networks for deep fusion [2405.19149, 2507.12823].
    - *Prototype image generation*: Generation of a query prototype synthesized in image space, converted into a matching problem in embedding space.

- **Query Composition**: The challenge is to determine, attribute by attribute, which information from $I_r$ must be “kept” and which should be “replaced” per $T_m$. Advanced models (e.g., TG-CIR) address this with explicit mask-based “keep-and-replace” mechanisms using teacher-student branches (target-aware guidance during training, target-free inference) and knowledge distillation [2309.01366].

- **Target Matching**: Contrastive objectives such as batch-based classification, soft triplet loss, or metric learning incorporating KL-divergence between similarity distributions are used to align composed queries with target image embeddings [2309.01366, 2404.11317].

**Zero-shot CIR (ZS-CIR)**: Methods that avoid reliance on annotated triplets, seeking to enable CIR with minimal or no manual labeling [2502.18495, 2405.02951, 2503.19296, 2404.15516].

- *Textual inversion approaches*: The reference image is mapped to a pseudo-word token in the pre-trained language embedding space (e.g., CLIP), combined with modification text, and processed by the text encoder; this reduces CIR to text→image retrieval [2405.02951, 2503.19296].
- *Pseudo-triplet or synthetic data approaches*: LLMs and image captioning models are leveraged to automatically generate $(I_r, T_m, I_t)$ triplets at scale, which are then used to finetune or pretrain CIR models, potentially with minimal supervision [2404.11317, 2404.15516, 2503.19910, 2507.05970].
- *Training-free and prompting methods*: Query unification through raw-data fusion, prompt engineering, or weaving key attributes directly into input data for pre-trained VLP encoders, thereby fully exploiting the multimodal alignment without expensive fusion modules [2404.15875].

The design landscape also encompasses *semi-supervised* and *few-shot* approaches, where synthetic or pseudo-labeled triplets augment a small set of labeled data to improve scalability [2404.15516, 2407.06001].

## 3. Key Architectural Principles

The principal challenges in CIR model design are (1) effective multimodal fusion, (2) conflict/consistency modeling between vision and language, and (3) robust metric learning for fine-grained ranking.

### Multimodal Fusion Strategies

- **Raw-Data Level Fusion**: Approaches such as DQU-CIR concatenate a caption generated from $I_r$ with $T_m$ or embed modification keywords visually in $I_r$ (“writing on” key phrases), then encode each “unified query” with the respective VLP encoders, combining outputs adaptively via a learnable linear weighting [2404.15875].
- **Feature-Level Fusion**: Most CIR systems concatenate or fuse representations of $I_r$ and $T_m$ at the feature level, using MLPs or attention mechanisms. Nonlinear fusions can introduce embedding drift from the VLP-aligned semantic space; orthogonal regularization or attention-based cross-modal mapping is used to counteract this [2309.01366, 2405.19149].
- **Cross-Attention and Knowledge Distillation**: Dual-branch designs with teacher and student composition modules, as seen in TG-CIR, use contrastive learning not only for query-target matching but for mimicking target-based ideal fusion during training, thereby improving generalization at inference [2309.01366].

### Conflict Relationship and Attribute Selection

A fundamental problem is to resolve whether to “keep” (inherit from $I_r$) or “replace” (adopt from $T_m$) each attribute dimension in the composed query (e.g., color, pattern, sleeve shape). Models implement mask prediction for attribute selection (e.g., via MLPs), supervised via knowledge distillation against a teacher using the actual target image during training [2309.01366].

### Adaptive Matching and Metric Learning

- **Attribute-wise similarity**: Attribute-specific cosine similarity is used to compute matching degrees between composed query and candidates; these degrees are aggregated (via summation in teacher or mean pooling in student branches) [2309.01366].
- **KL-divergence Regularization**: Distributions of similarity between composed queries and a batch of candidate images are regularized against the visual similarity distribution between the target image and candidates, promoting adaptive ranking beyond simple hard positives/negatives [2309.01366].
- **Contrastive and Classification Losses**: InfoNCE batch-based contrastive loss is ubiquitous for aligning composed queries with target features, sometimes combined with additional auxiliary alignment or reconstruction losses [2310.05473, 2404.11317].

## 4. Data Generation, Augmentation, and Benchmarking

Manual annotation of CIR triplets is costly and time-consuming, limiting the scalability of supervised CIR. Prominent augmentation and pseudo-labeling techniques include:

- **LLM- and MLLM-driven Synthetic Triplet Generation**: Automated pipelines use LLMs for difference captioning and T2I generative models for image synthesis, filtered by multimodal models for consistency and quality. The CIRHS dataset represents a large-scale, synthetic triplet collection produced in this manner [2507.05970].
- **Pseudo-triplet Guided Few-shot Learning**: Methods generate triplets by masking and captioning unlabeled images or by captioning plausible target images, ranking candidate pairs by “challenge” and randomly sampling the most informative for fine-tuning [2407.06001].
- **Benchmark Dataset Curation**: Dedicated CIR benchmarks include FashionIQ, Shoes, CIRR, and open-domain benchmarks such as CIRCO (which features multiple ground truths and semantic categories per query) [2405.02951].
- **Evaluation Metrics**: Standard metrics include Recall@k, subset recall for challenging negatives, and mAP@k (for datasets with multiple ground truths). Loss functions used during training are commonly batch-based classification/contrastive variants, e.g.,
  $$
  L_{\mathrm{BBC}} = \frac{1}{B} \sum_{i=1}^B -\log \left( \frac{\exp( \kappa(\phi^{(i)}, x_t^{(i)})/\tau ) }{ \sum_{j=1}^B \exp( \kappa(\phi^{(i)}, x_t^{(j)})/\tau ) } \right)
  $$
  where $\phi^{(i)}$ is the composed query, $x_t^{(i)}$ is the target image, $\kappa$ is the similarity (usually cosine), and $\tau$ a temperature.

| Dataset    | Domain       | Distinctive Features         |
|------------|--------------|-----------------------------|
| FashionIQ  | Fashion      | Attribute-rich, triplet labels|
| CIRR       | Open domain  | Fine-grained reasoning, single ground truth|
| CIRCO      | Open domain  | Multi-ground truth, semantic categories |

## 5. Recent Advances and Experimental Findings

TG-CIR [2309.01366] exemplifies the best practices in contemporary CIR: It achieves strong Recall@k (e.g., $>$29% improvement on Shoes R@1 over previous baselines), validates the efficacy of integrating global and local attribute extraction, orthogonal regularization, and target-aware composition. Ablation studies confirm all components' contributions, including the teacher-student strategy and KL-divergence regularization.

Sentence-level prompt augmentation using BLIP-2 and Q-Former further improves CIR, especially for multi-object or complex modification queries [2310.05473], with up to 14 accuracy points improvement in R@10 (FashionIQ Shirt).

Plug-and-play data scaling (both positive and negative examples), as proposed in contrastive learning frameworks, raises ceiling performances for multiple baselines, significantly improving R@k on both FashionIQ and CIRR [2404.11317].

A plausible implication is that the future of CIR will further blend large-scale synthetic data generation, prompt-driven modification capture, and adaptive fusion strategies—potentially closing the performance gap between zero-shot/few-shot and fully supervised systems.

## 6. Open Challenges and Research Directions

Several research challenges and frontiers in CIR remain open:

- **Scalability and Annotation Efficiency**: Despite synthetic triplet generation, ensuring diversity, naturalness, and coverage in modifications—without incurring annotation bias—is an ongoing concern [2404.11317, 2507.05970].
- **Fine-grained Attribute and Conflict Resolution**: Modeling nuanced attribute substitution (especially spatial attributes, view/angle changes, or subjective/non-salient modifications) is still not consistently addressed [2310.05473].
- **Interpretability and User Control**: Techniques that can explain or visualize attribute-level “keep/replace” decisions promise to enhance trust and usability, but are not yet widely adopted.
- **Generalization across Domains and Modalities**: Robustness to out-of-distribution queries, extension to video, dialog, sketch-and-text queries, and dialog-based iterative search remain under active development [2502.18495].
- **Evaluation Benchmark Expansion**: Datasets with richer ground truth, more diverse concepts, and semantic labeling (as in CIRCO) are needed for reliable evalution and to minimize false negatives [2405.02951].
- **LLM/VLM Embedding Quality and Fusion Strategy**: How to optimally fuse image and text at the embedding or token level without incurring excessive semantic drift or modality bias is a rapidly evolving question, given the proliferation of new MLLMs and complex retrieval pipelines [2405.19149, 2503.19910, 2507.12823].

In summary, CIR research has rapidly advanced in methodological sophistication and empirical capability, shifting from basic feature concatenations to attribute-aware, teacher-student, and prompt-augmented architectures, aided by automated data generation. Open challenges include achieving optimal fusion, robust generalization, data efficiency, and extending evaluation to richer, more realistic benchmarks.

Source: https://www.emergentmind.com/topics/composed-image-retrieval-cir