TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

Abstract: Composed Image Retrieval (CIR) is an important image retrieval paradigm that enables users to retrieve a target image using a multimodal query that consists of a reference image and modification text. Although research on CIR has made significant progress, prevailing setups still rely simple modification texts that typically cover only a limited range of salient changes, which induces two limitations highly relevant to practical applications, namely Insufficient Entity Coverage and Clause-Entity Misalignment. In order to address these issues and bring CIR closer to real-world use, we construct two instruction-rich multi-modification datasets, M-FashionIQ and M-CIRR. In addition, we propose TEMA, the Text-oriented Entity Mapping Architecture, which is the first CIR framework designed for multi-modification while also accommodating simple modifications. Extensive experiments on four benchmark datasets demonstrate that TEMA's superiority in both original and multi-modification scenarios, while maintaining an optimal balance between retrieval accuracy and computational efficiency. Our codes and constructed multi-modification dataset (M-FashionIQ and M-CIRR) are available at https://github.com/lee-zixu/ACL26-TEMA/.

• The paper introduces TEMA, a method that integrates an LLM-based Parsing Assistant and an Entity Mapping module to address multi-modification queries.
• It leverages enhanced dataset annotations with longer, multi-modification texts to improve alignment between textual modifications and visual features.
• Experiments on M-FashionIQ and M-CIRR demonstrate superior retrieval performance, efficiency, and robustness in handling fine-grained, multi-entity modifications.

TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval

Problem Definition and Motivation

Composed Image Retrieval (CIR) leverages a reference image in conjunction with modification text to retrieve a target image from large-scale collections. This paradigm outperforms unimodal retrieval by synthesizing visual priors (from the image) and user-driven intent (from the text), enabling fine-grained control over the retrieval targets. However, existing CIR models predominantly focus on simple, short modification expressions that are ill-suited for real-world scenarios where users may specify multiple, overlapping, or fine-grained modifications. The paper identifies two critical challenges in practical CIR:

1. Insufficient Entity Coverage: Current models and datasets often miss entities that require modification due to limited annotation granularity.
2. Clause-Entity Misalignment: In many cases, multiple modification clauses may refer to the same entity, or one clause may affect multiple entities, but models lack the explicit alignment capabilities to resolve these relations.

Empirical evidence demonstrates a substantial drop in retrieval performance for multi-modification scenarios in established CIR models (Figure 1).

Figure 1: (a) Illustration of conventional CIR setup; (b) Performance degradation in multi-modification scenarios across baselines, highlighting the core limitation addressed by this paper.

Dataset Construction: M-FashionIQ and M-CIRR

To bridge the gap between prototypical CIR benchmarks and complex, real-world user intent, the authors introduce two new datasets—M-FashionIQ and M-CIRR—expanding the FashionIQ and CIRR datasets with Multi-Modification Texts (MMT). These datasets are distinguished by:

• Annotation Pipeline: Reference and target images are paired, and detailed modification text is generated using a Multimodal LLM (Llama 3.2), followed by post-processing with GPT-4o for hallucination detection/removal and a manual review for consistency, accuracy, diversity, and linguistic quality.
• Coverage: The expanded annotations demonstrate a drastic increase in average modification text length (152.7 tokens in M-FashionIQ vs. 24.7 in FashionIQ; 319.4 in M-CIRR vs. 12.8 in CIRR) and incorporate multiple entities, complex relations, and fine-grained details, aligning more closely with real-world query requirements.

  Figure 2: End-to-end pipeline for generating robust multi-modification annotations, comprising staged LLM prompting, hallucination filtering, and expert human validation.

Methodology: Text-oriented Entity Mapping Architecture (TEMA)

TEMA is designed to explicitly address the inadequacies of CIR models in handling multi-modification queries. The architecture consists of two principal modules:

1. MMT Parsing Assistant (PA)

• Purpose: Enhances to-be-modified entity coverage during training.
• Mechanics: Employs an LLM-based summarizer to extract key entities from MMT and a consistency detector (LLM-based) to verify exhaustive coverage against the MMT. Upon detecting omissions or hallucinations, summaries are iteratively refined (used only during training).
• Output: A concise summary embedding capturing all to-be-modified entities.

2. MMT-oriented Entity Mapping (EM)

• Purpose: Resolves clause-entity misalignments in multi-modification contexts.
• Mechanics:
  • Introduces learnable queries as dynamic proxies for entities.
  • Aggregates multiple MMT clauses per entity using a Transformer architecture, guided by the summary embedding from PA (textual mapping) and global/local image features (visual mapping).
  • Enforces summary-guided distillation loss, encouraging aggregation tokens to align semantically with identified entities in the summary.
  • Orthogonal regularization is imposed to ensure semantic independence between entity features.
• Query Composition: The model concatenates global features and mapped entity features before matching against candidate target image embeddings.

  Figure 3: TEMA architecture diagram showing the interplay between MMT Parsing Assistant, Entity Mapping modules, and the final multimodal embedding composition.

Experimental Results

Quantitative Analysis

• Datasets: Experiments are conducted on M-FashionIQ and M-CIRR.
• Metrics: R@K for various K (FashionIQ: R@10/50, CIRR: R@1/5/10, R_subset).
• Findings:
  • TEMA achieves state-of-the-art performance under both multi-modification and original CIR conditions. R@10 on M-FashionIQ improved from 47.38% (prior SOTA) to 50.59%. The model also generalized to traditional CIR tasks with best-in-class results.
  • The largest gains are observed in scenarios requiring multiple entity modifications and clause aggregation.
  • Ablations show significant performance degradation when removing the PA or EM modules, highlighting their necessity for complex modification reasoning.

Efficiency

• Computational Cost: TEMA delivers superior efficiency—lower FLOPs, faster training and inference, and reduced GPU memory footprint compared to strong baselines.

Sensitivity and Robustness

• Hyperparameters: Sensitivity analysis demonstrates optimal model behavior at intermediate values of summary-guided distillation loss weight and number of learnable queries.
• LLM Interchangeability: The modular nature of PA supports replacement with various open-source or proprietary LLMs, with negligible impact on final performance.

  Figure 4: Sensitivity of TEMA to summary loss scaling and number of learnable entity queries.

Qualitative Analysis and Visualization

• Entity Summarization: The PA module produces summaries that consistently cover all necessary entities, as visually highlighted (Figure 5). Attention maps further validate that both textual and visual entity representations are accurately learned.

  Figure 5: PA-generated summaries, with colored highlights marking to-be-modified entities extracted from MMT.

• Prompt Design: Diverse prompt strategies yielded richer and more comprehensive MMTs, with best results arising from prompts emphasizing multi-entity granularity (Figure 6).

  Figure 6: MMT examples generated using different prompts, indicating variability and coherence in annotation via prompt engineering.

• Reduction of False Negatives: Annotating previously ambiguous queries with MMTs reduced prevalence of false negatives, thereby improving learning signal quality (Figures 8 and 9).

  Figure 7: Illustration of false-negative mitigation; MMT relabeling increases specificity and disambiguates retrieved results.

• Entity Attention Alignment: Attention visualization confirms robust model alignment between summarized entities and visual counterparts in both fashion and open-domain settings (Figures 10, 11).

  Figure 8: Attention heatmaps demonstrating the model's focus on all relevant spatial regions corresponding to summary entities (M-FashionIQ).

  

  Figure 9: Analogous attention visualization for M-CIRR, underlining domain generalization.

Implications and Future Directions

TEMA systematically advances the CIR field towards practical deployment by enabling robust multi-modification query handling with explicit entity correspondence and aggregation. The model's dual emphasis on annotation rigor (via MMTs) and architectural adaptability (via PA and EM modules) paves the way for:

• Fine-grained, instruction-following CIR systems usable in complex real-world scenarios (e.g., e-commerce, HCI).
• Downstream vision-language reasoning tasks benefiting from explicit entity and clause disentanglement.
• Research on interactive or multi-turn retrieval, as current models—including TEMA—address only single-turn settings.
• Studies on annotation scalability and automated quality assurance via advanced LLM-based curation.

The dual process of leveraging LLMs for annotation and internal summarization aligns CIR research with recent progress in multimodal AI and instruction tuning, with potential impact on other multimodal understanding and generation tasks.

Conclusion

TEMA addresses longstanding limitations in composed image retrieval by delivering the first framework optimized for multi-modification queries and by constructing high-quality, real-world-aligned benchmarks. Through the integration of MMT Parsing Assistant and Entity Mapping modules, TEMA achieves superior accuracy and efficiency, robustly counting all modified entities and resolving clause-entity misalignments. This work establishes a new paradigm for CIR, with direct applicability to multimodal retrieval systems and vision-language reasoning workflows, while highlighting the importance of dataset design and modular model architecture for advancing real-world AI solutions.