HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

Abstract: Composed Image Retrieval (CIR) is a flexible image retrieval paradigm that enables users to accurately locate the target image through a multimodal query composed of a reference image and modification text. Although this task has demonstrated promising applications in personalized search and recommendation systems, it encounters a severe challenge in practical scenarios known as the Noise Triplet Correspondence (NTC) problem. This issue primarily arises from the high cost and subjectivity involved in annotating triplet data. To address this problem, we identify two central challenges: the precise estimation of composed semantic discrepancy and the insufficient progressive adaptation to modification discrepancy. To tackle these challenges, we propose a cHrono-synergiA roBust progressIve learning framework for composed image reTrieval (HABIT), which consists of two core modules. First, the Mutual Knowledge Estimation Module quantifies sample cleanliness by calculating the Transition Rate of mutual information between the composed feature and the target image, thereby effectively identifying clean samples that align with the intended modification semantics. Second, the Dual-consistency Progressive Learning Module introduces a collaborative mechanism between the historical and current models, simulating human habit formation to retain good habits and calibrate bad habits, ultimately enabling robust learning under the presence of NTC. Extensive experiments conducted on two standard CIR datasets demonstrate that HABIT significantly outperforms most methods under various noise ratios, exhibiting superior robustness and retrieval performance. Codes are available at https://github.com/Lee-zixu/HABIT

• The paper introduces HABIT, a novel framework that uses mutual knowledge estimation and dual-consistency learning to mitigate noise in composed image retrieval.
• It leverages batch-relative Transition Rate and KL-divergence based losses to refine semantic alignment, outperforming baselines on FashionIQ and CIRR benchmarks.
• The framework demonstrates robust retrieval performance under high-noise conditions, enabling improved adaptation of fine-grained visual and textual queries.

HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval

Introduction and Problem Motivation

Composed Image Retrieval (CIR) presents a flexible paradigm for semantic visual search, utilizing queries composed of both a reference image and textual modifications. This multimodal query framework facilitates fine-grained target localization, addressing user intent beyond simple visual similarity. However, CIR suffers from practical limitations arising from the high cost and subjectivity of triplet annotations, leading to pervasive Noise Triplet Correspondence (NTC). These challenges manifest as either partial or total misalignment between query modifications and target images, exacerbated by annotation brevity and hallucinations in large models (Figure 1).

Figure 1: (a) An example of the CIR paradigm. (b) Unmentioned visual discrepancies in CIR triplets. (c) The Chrono-Synergia Mechanism for robust learning.

Two central technical bottlenecks are identified: (1) precise quantification of the semantic discrepancy between the composed (image, text) query and the target, and (2) robust progressive adaptation to triplet-level noise. Previous robust learning approaches in multimodal retrieval, such as TME, focus primarily on pairwise similarity metrics and do not adequately capture triplet-level semantic discrepancies unique to CIR. The HABIT framework addresses these limitations through mutual information-based sample evaluation and a dual-consistency progressive learning strategy.

Methodology

The core architecture of HABIT is bifurcated into two primary modules: Mutual Knowledge Estimation (MKE) and Dual-consistency Progressive Learning (DPL), illustrated in Figure 2.

Figure 2: HABIT's architecture comprising (a) Mutual Knowledge Estimation and (b) Dual-consistency Progressive Learning modules.

Mutual Knowledge Estimation (MKE)

MKE leverages mutual information between composed features and their targets to derive a measure of sample cleanliness, thus enabling noise-aware learning. Features are extracted with a Q-Former encoder from both the composed multimodal query and the target image. The mutual knowledge ($\operatorname{MK}$) is calculated as the sum over all query-target feature pairs of the joint log probability ratio, thus capturing fine-grained semantic co-alignment.

The central innovation is the introduction of the Transition Rate (TR): a batch-relative margin between each sample’s mutual knowledge and that of the batch's "standard" (lowest-loss) correspondence. Cleanliness estimation ($\mathbb{E}$) is produced by aggregating TR values from both query and target perspectives, enabling reliability-aware sample selection that transcends crude feature-space similarity counts.

Dual-consistency Progressive Learning (DPL)

DPL is designed to retain beneficial model habits (robust semantic matching capabilities) and calibrate residual misclassification errors throughout the training dynamics. This module includes two complementary mechanisms:

• Chrono-Synergia Noise Discrimination: By tracking the temporal sequencing of cleanliness estimations across iterations and clustering (DBSCAN) outliers, DPL identifies samples systematically deviating from normal semantic matching and applies a noise mask during optimization.
• Time-Flux Knowledge Updating: Dual consistency between historical and current similarity matrices is enforced via a KL-divergence loss on clean samples, preserving learned alignment. Additionally, margin-based soft estimation and robust contrastive losses, with dynamic margins informed by cleanliness scores, perform difficult negative mining and correction of persistent “bad habits”.

The final loss is a weighted sum of the robust contrastive loss, KL-consistency, and the soft estimation margin components.

Empirical Evaluation

HABIT is evaluated on FashionIQ and CIRR, established CIR benchmarks with varying levels of annotation noise. The empirical comparison spans both ordinary CIR methods (SSN, CALA, SPRC) and recent robust learning baselines (RCL, RDE, TME).

Key findings:

• Superior robustness: HABIT consistently achieves top performance across all tested noise levels. For CIRR at high noise ($\sigma=0.5, 0.8$), it surpasses TME by $1.16\%$ and $1.28\%$ average recall, and on FashionIQ the gain is $0.94\%$ and $1.31\%$ respectively.
• As noise severity increases, the performance gap between robust (e.g., TME, HABIT) and non-robust methods widens considerably, confirming the necessity of explicit noise modeling.
• Ablation studies confirm that removal or weakening of either MKE or DPL components—such as discarding the transition rate in noise estimation or forgoing dual historical-current consistency—results in significant degradation of both average recall and robustness metrics.

A comparative overview of performance rank across noise ratios is visualized in Figure 3.

Figure 3: Comprehensive performance ranking of various models on CIRR and FashionIQ across different noise ratios.

Sensitivity analysis reveals optimal tradeoffs for the balancing coefficients governing the KL and soft losses (Figure 4). Case studies on semantic-rich queries demonstrate HABIT's improved retrieval for complex multi-object and attribute compositions (Figure 5).

Figure 5: Qualitative comparison of top-ranked retrieval results on CIRR and FashionIQ showing superior semantic alignment via HABIT.

Figure 4: Hyperparameter sensitivity analysis for (a) $\kappa$ (KL tradeoff) and (b) $\gamma$ (soft margin loss weight).

Failure analysis (Figure 6) indicates that even in highly ambiguous or false-negative-prone scenarios, HABIT's top-k results maintain higher semantic relevance than other methods.

Figure 6: Failure case examples on CIRR and FashionIQ, highlighting remaining annotation ambiguities.

Qualitative Analysis and Cleanliness Estimation

HABIT’s cleanliness estimation is further visualized in Figure 7, displaying the model’s ability to sharply distinguish clean and noisy triplets even under substantial NTC ratios.

Figure 7: Visualization of MKE-based cleanliness estimation across clean, noisy, and ambiguous CIR triplets.

Moreover, the similarity matrices produced by HABIT, in contrast to state-of-the-art TME, show sharper diagonal dominance and superior suppression of negative matches (Figure 8), manifesting the impact of mutual knowledge-based intra-batch calibration and dual-consistency regularization.

Figure 8: Comparison of TME and HABIT similarity matrices on CIRR, showing improved positive-negative separation with HABIT.

Implications and Future Directions

HABIT’s progressive, collaborative learning regime delivers a modular solution to triplet-level noise in CIR, integrating mutual information-driven estimation with longitudinal self-correction. By establishing a batch-relative, distribution-aware margin for correspondence reliability, it advances noise-robust retrieval under resource-constrained annotation conditions, a scenario prevalent in real-world deployment of CIR systems.

The architectural approach—separating noise estimation from adaptation/correction and incorporating historical consistency—provides a blueprint for extending robust learning to other strongly compositional, multimodal settings, including open-vocabulary video retrieval and intent-inference (e.g., LLM-augmented CIR). Further, the reliance on mutual information, as opposed to pure feature-space similarity, suggests the potential for integration with uncertainty modeling, probabilistic embeddings, and causal matching strategies.

Future exploration may consider:

• Scaling HABIT with larger vision-language backbones and open-domain annotation methods.
• Integrating uncertainty quantification for active annotation and semi-supervised extension.
• Generalizing the Chrono-Synergia mechanism to continual learning or federated retrieval tasks, where temporal consistency mitigates cross-source annotation noise.

Conclusion

The HABIT framework systematically addresses the core challenge of noisy triplet correspondence in composed image retrieval through a principled combination of mutual knowledge-driven estimation and dual-consistency progressive adaptation. Its empirical superiority, demonstrated across standard CIR benchmarks and strong ablation support, establishes a robust baseline for future research into noise-aware multimodal retrieval systems (2604.18037).