INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval

Abstract: Composed Image Retrieval (CIR) is a challenging image retrieval paradigm that enables to retrieve target images based on multimodal queries consisting of reference images and modification texts. Although substantial progress has been made in recent years, existing methods assume that all samples are correctly matched. However, in real-world scenarios, due to high triplet annotation costs, CIR datasets inevitably contain annotation errors, resulting in incorrectly matched triplets. To address this issue, the problem of Noisy Triplet Correspondence (NTC) has attracted growing attention. We argue that noise in CIR can be categorized into two types: cross-modal correspondence noise and modality-inherent noise. The former arises from mismatches across modalities, whereas the latter originates from intra-modal background interference or visual factors irrelevant to the coarse-grained modification annotations. However, modality-inherent noise is often overlooked, and research on cross-modal correspondence noise remains nascent. To tackle above issues, we propose the Invariance and discrimiNaTion-awarE Noise neTwork (INTENT), comprising two components: Visual Invariant Composition and Bi-Objective Discriminative Learning, specifically designed to handle the two-aspect noise. The former applies causal intervention on the visual side via Fast Fourier Transform (FFT) to generate intervened composed features, enforcing visual invariance and enabling the model to ignore modality-inherent noise during composition. The latter adopts collaborative optimization with both positive and negative samples, and constructs a scalable decision boundary that dynamically adjusts decisions based on the loyalty degree, enabling robust correspondence discrimination. Extensive experiments on two widely used benchmark datasets demonstrate the superiority and robustness of INTENT.

• The INTENT framework reduces noise in CIR by focusing on both modality-inherent and cross-modal correspondence noise.
• Visual Invariant Composition (VIC) uses causal intervention with FFT to suppress irrelevant image factors while maintaining semantic integrity.
• Bi-Objective Discriminative Learning (BiODL) refines decision boundaries using dynamic loyalty degree estimation to enhance retrieval accuracy.

Summary of "INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval"

Introduction

The paper "INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval" (2604.18051) addresses the challenge of composed image retrieval (CIR), which utilizes multimodal input consisting of reference images and textual modifications to retrieve target images. Standard CIR methods suffer from annotation errors, leading to noisy triplet correspondence (NTC) due to incorrect matches across datasets. The authors categorize CIR noise into modality-inherent noise (background or irrelevant factors in images) and cross-modal correspondence noise (mismatches between modalities). Existing approaches predominantly focus on the latter, neglecting modality-inherent noise.

Proposed Method: INTENT Framework

The paper proposes the INTENT framework, consisting of Visual Invariant Composition (VIC) and Bi-Objective Discriminative Learning (BiODL) modules.

Visual Invariant Composition (VIC)

The VIC module generates counterfactual images by applying causal intervention through Fast Fourier Transform (FFT) on reference images. This intervention alters frequency components to suppress modality-inherent noise while preserving essential semantics.

Figure 1: Visualization of the effects of different intervention operations in the VIC module, including our FFT-based intervention, Random Mask, Patch Shuffle, Gaussian Blur, and Style Transfer.

The paper embraces causal relations to enhance the multimodal composition, allowing models to focus on invariant semantics and ignore irrelevant noise during the composition process.

Bi-Objective Discriminative Learning (BiODL)

BiODL utilizes collaborative optimization with positive and negative samples, constructing a scalable decision boundary adjusted dynamically according to the loyalty degree.

Figure 2: (a) shows typical Modality-inherent Noise in CIR. (b) reveals Cross-modal Correspondence Noise. (c) presents two cases: left successfully retrieves target with low confidence (50\%), while right fails with high confidence (70\%), illustrating varying decision boundaries in retrievals.

The authors address hard decision boundaries prevalent in CIR by promoting contrast between samples and using loyalty degree estimation to refine the decision process. This ensures improved discrimination amidst semantic complexity and variability.

Experiments and Results

The results demonstrate INTENT's robustness and effectiveness over several benchmarks, showing improved recall and precision metrics compared to existing methods. On datasets such as FashionIQ and CIRR, INTENT consistently achieves higher accuracy, notably under higher noise ratios, affirming the framework's ability to mitigate NTC.

Figure 3: The causal graph of CIR. Solid arrows present the cause effect. Dash arrows mean there exist correlations.

In ablation studies, components such as FFT-based intervention and scalable decision boundaries markedly enhanced performance, indicating their pivotal role in noise mitigation.

Implications and Future Directions

INTENT establishes a framework for effectively addressing both modality-inherent and cross-modal correspondence noise, paving the way for more precise CIR. By leveraging causal interventions, INTENT achieves stable semantic composition and robust retrieval outcomes. Future research could extend these methodologies to other multimodal learning contexts or explore alternate noise mitigation strategies, such as advanced causal inference or adaptive learning techniques tailored to broader applications in AI.

Conclusion

INTENT represents a strategic advance in composed image retrieval, emphasizing invariance learning and adaptive discrimination practices to manage complex noise environments prevalent in CIR tasks. The framework not only addresses intrinsic and extrinsic noise dynamics but also suggests a broader applicability of causal interventions within multimodal systems.