COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations

Abstract: Composed Image Retrieval (CIR) represents a challenging retrieval task that targets locating specific images through multimodal inputs. Despite recent progress in CIR techniques, prior approaches often overlook cases where images appear visually alike yet differ in attributes, potentially undermining both multimodal feature fusion and similarity modeling. To mitigate this limitation, we design a unified representation of cross-modal features based on attribute prototypes. Nevertheless, the task is far from straightforward, owing to three core issues: (1) entanglement in attribute-level semantics, (2) inconsistency across modalities, and (3) supervised signal missing. To tackle the above obstacles, we introduce a COMposed image retrieval network guided By attrIbute-based NEighbor Relations (COMBINER). Specifically, we first design an Adaptive Semantic Disentanglement module, which is capable of disentangling attribute features based on multimodal primitive features. Secondly, we propose a Unified Prototype-based Composition module, which can construct cross-modal unified prototypes (CUP) and facilitate multimodal feature composition. Finally, we introduce a Dual Relations Modeling module, which can mine pairwise and neighbor relations based on attribute similarity. Compared to traditional neighbor relations modeling CIR methods, COMBINER represents the first study addressing the phenomenon of visually similar but attribute-unrelated samples. It achieves a more accurate understanding of the semantic relations among samples by employing an attribute prototype-based similarity metric. Comprehensive experiments conducted on three benchmark datasets confirm the effectiveness of our proposed COMBINER. The implementation of our method will be accessed at https://github.com/Lee-zixu/COMBINER

• The paper introduces COMBINER, a novel framework using adaptive semantic disentanglement to extract attribute prototypes from image and text inputs.
• It unifies cross-modal features via prototype-based composition and employs dual relations modeling to distinguish attribute-relevant neighbors from visually similar distractors.
• Experimental results show state-of-the-art gains on FashionIQ, Shoes, and CIRR, validating robust metric learning and practical low-latency retrieval deployment.

Composed Image Retrieval Guided by Attribute-based Neighbor Relations: An Expert Analysis of COMBINER

Problem Motivation and Challenges in CIR

Composed Image Retrieval (CIR) constitutes a pivotal multimodal retrieval task, requiring the retrieval of a target image conditioned on a query comprising both a reference image and a modification text. Prevailing CIR systems typically fail in scenarios where visually similar images are attribute-irrelevant, i.e., where distractor candidates are close in visual space but semantically disjoint from the compositional attribute specified by the textual modification (Figure 1).

Figure 1: Example of (a) Pairwise Relations, (b) Neighbor Relations, and (c) Visually Similar Images in Relations Modeling. Visually similar but attribute-unrelated samples ($C_2$) confound traditional neighbor relations modeling, deteriorating retrieval precision.

These failures reveal three core modeling challenges:

• Attribute-level semantic entanglement: Visual and textual information often contains entangled attributes, impeding fine-grained cross-modal alignment.
• Modal inconsistency: Features sourced from disparate modalities lack a unified attribute taxonomy, undermining compositionality.
• Absence of explicit supervision: Ground-truth for attribute-aware neighbor relations is missing, complicating metric learning.

COMBINER: Architecture and Methodological Advances

COMBINER ("COMposed image retrieval network guided By attrIbute-based NEighbor Relations") introduces a solution through explicit attribute prototype mining, semantic disentanglement, unified cross-modal prototypes, and dual relations modeling.

Adaptive Semantic Disentanglement (ASD)

This module utilizes a novel Semantic Attribute Attention (SAA) mechanism to adaptively extract attribute prototypes from both image and text inputs. By deploying attention-based filters on CLIP-extracted features (global and local), ASD generates disentangled, interpretable prototype vectors for each modality, forming the foundation for prototype-based matching.

Unified Prototype-based Composition (UPC)

Addressing the modal inconsistency problem, the UPC module constructs Cross-modal Unified Prototypes (CUP), an attribute-aligned subspace derived by re-aligning prototypes from both modalities through further semantic attention. This shared semantic anchor (Figure 2) enables robust and algebraically-compositional combination of reference and modification attributes.

Figure 2: Schematic of the similarity measure method based on attribute prototypes, underpinning unified cross-modal semantic representation.

Dual Relations Modeling (DRM)

DRM fuses supervised pairwise modeling (classic query–target matching) with unsupervised neighbor modeling, the latter implemented through attribute-aware K-means clustering over prototype representations. Unlike prior methods that rely solely on visual feature space proximity, DRM leverages CUP to simultaneously cluster attribute-relevant neighbors and explicitly exclude visually similar but attribute-irrelevant samples. DRM is further regularized by KL-divergence-based mutual learning, aligning the similarity distributions between query (composed) and target (ground-truth) prototypes.

Figure 3: Overall framework of COMBINER, delineating (a) Adaptive Semantic Disentanglement, (b) Unified Prototype-based Composition, and (c) Dual Relations Modeling.

Experimental Analysis

Quantitative Performance

COMBINER establishes new state-of-the-art (SOTA) metrics on all three standard CIR datasets: FashionIQ, Shoes, and CIRR. On the largest open-domain CIRR dataset, COMBINER attains an R@1 absolute improvement of 1.2% over the previous SOTA, and achieves 2.1%/2.9% gains on FashionIQ-Avg-R@10 and Shoes-R@1, respectively. These results are robust across backbone architectures (ResNet50, ViT-B/L/H, BLIP-2), with method-level improvements decoupled from mere architectural scaling.

Component and Design Justification

Ablation studies validate the indispensability of each component:

• Removing ASD (replacing prototype attention with average pooling) markedly degrades performance, confirming the centrality of adaptive attribute disentanglement.
• Omitting UPC (simple feature addition) or DRM leads to further drops, demonstrating that both cross-modal prototype unification and dual relation modeling (especially attribute-aware neighbor discrimination) are critical.
• Alternative clustering algorithms (DBSCAN, Spectral) or regularization choices (Jensen-Shannon, Wasserstein) yield inferior results, confirming the design rationale for K-means and KL-based mutual learning.

  Figure 4: Influence of hyperparameters ($\rho$: cluster-oriented loss weight, $\kappa$: cluster-based regularization, $\mu$: pool-based regularization) on model performance on FashionIQ and CIRR.

  

  Figure 5: Model sensitivity to attribute prototype number ($U$) and semantic cluster number ($H$) on representative datasets.

Qualitative Diagnostics

Case studies show COMBINER's robustness in retrieving targets with complex attribute modifications, outperforming SOTA rivals such as SPRC (Figure 6). Attention and cluster visualizations substantiate that the model correctly identifies regions and clusters aligned with modification semantics, quantitatively and qualitatively surpassing previous neighbor-based CIR models.

Figure 6: Case study analysis on FashionIQ, Shoes, CIRR, including failure cases analyzing dataset false negatives and prototype capacity.

Figure 7: Similarity matrix visualization on FashionIQ shows pronounced diagonal dominance in COMBINER, confirming enhanced discrimination.

Figure 8: GradCAM-based attention visualizations on different datasets reveal prototype alignment with targeted attributes.

Figure 9: Visualization of semantic cluster neighbors affirms that DRM groups attribute-relevant samples, excluding attribute-irrelevant confounders.

Theoretical and Practical Implications

COMBINER demonstrates that attribute prototype learning with explicit attention-based disentanglement, followed by prototype alignment and attribute-aware neighbor relation modeling, is crucial for robust multimodal composition. The approach directly addresses the core limitation of visually similar but semantically irrelevant negatives, a persistent challenge in retrieval with fine-grained semantics, and generalizes across open-domain and fashion-domain CIR tasks.

Practically, the system outperforms prior baselines without external LLMs/VLMs, incurring negligible overhead at inference time, thus suiting deployment in real-world, low-latency visual search and e-commerce interfaces. The model is robust to hyperparameter variations (attribute cluster sizes, regularization weights), auguring scalability to unseen domains and modifications.

Future Directions

This work opens several technical avenues:

• Extension to unbounded open-vocabulary attribute spaces and compositional generalization.
• Integration with LLM-based pseudo-labeling only for unsupervised domain transfer.
• Joint visual-language generative augmentation for compositional training data synthesis.
• Transposition to video and cross-modal retrieval scenarios involving temporal and entity-level attributes.

Conclusion

COMBINER presents a rigorously validated, attribute prototype-based framework for composed image retrieval. Through adaptive disentanglement, prototype-based fusion, and dual attribute-aware relational modeling, it achieves robust metric learning even in the presence of visually deceptive negatives. This work sets a new methodological benchmark for the CIR task and presents a foundation for future extensions to compositional vision-language understanding (2606.04604).