Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning

Abstract: Large-scale web-harvested datasets have fueled the progress of cross-modal retrieval but inevitably suffer from noisy correspondence, which severely degrades model generalization. Existing methods primarily address this by filtering out noise or seeking a substitute label, yet they predominantly remain bound by a "Discrete Selection" paradigm. We argue that relying on a single discrete proxy induces Single-Point Fragility and Discretization Error. To overcome these limitations, we propose a novel framework, Intra-modal Neighbor-aware Noise Rectification (IN2R), which shifts the paradigm from searching for a substitute to synthesizing a reliable supervision target. Leveraging the intrinsic geometric stability of intra-modal data, IN2R employs a Graph Refiner to perform relational reasoning over neighbors retrieved from a dynamic Cross-Model Memory. Instead of propagating discrete labels, our method synthesizes a continuous, soft prototype that reflects the consensus of the local semantic neighborhood, effectively rectifying inter-modal misalignment. Extensive experiments on Flickr30K, MS-COCO, and CC152K demonstrate that IN2R significantly outperforms state-of-the-art methods. Our code and pre-trained models are publicly available at https://github.com/liuyyy111/IN2R.

• The paper presents IN²R, which replaces discrete neighbor selection with continuous prototype synthesis from intra-modal consensus to mitigate noise.
• The framework employs a co-training architecture with cross-model memory and graph-guided self-attention to enhance vision-language alignment.
• Empirical results on Flickr30K, MS-COCO, and CC152K show significant retrieval performance gains, especially under high-noise conditions.

Intra-Modal Neighbor-Aware Noise Rectification: Shifting from Discrete Selection to Continuous Prototype Synthesis in Vision-Language Alignment

Motivation and Limitations of Discrete Proxy-Based Noise Rectification

Training vision-LLMs on large-scale datasets scraped from the web is fundamentally limited by the prevalence of noisy correspondences—instances where images and their paired captions are mismatched or irrelevant. Prior approaches for noise mitigation are largely rooted in "Discrete Selection": filtering out suspected noisy samples or substituting a noisy label with a discrete proxy (neighbor) from the existing dataset. However, these approaches are beset by two core deficiencies: (1) Single-Point Fragility, whereby the chosen proxy is itself unreliable—a single outlier can skew supervision; (2) Discretization Error, as the true semantic alignment often resides between discrete dataset samples and is inevitably lost in a finite candidate pool.

The central claim of the paper "Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning" (2606.04061) is that semantic truth is better modeled as a continuous prototype synthesized from the local intra-modal consensus rather than any discrete surrogate. Leveraging intrinsic intra-modal geometric structure enables collective, stable, and fine-grained correction of label noise, which the authors operationalize via their Intra-modal Neighbor-aware Noise Rectification (IN$^2$R) framework.

Figure 1: Contrast between the traditional discrete selection paradigm, which chooses a single substitute from a finite set (prone to noise), and IN$^2$R, which leverages intra-modal topological structure and aggregates via a graph refiner for robust, continuous prototype synthesis.

IN$^2$R: Architecture and Theoretical Underpinnings

Co-Training with Cross-Model Dynamic Memory

IN$^2$R is instantiated using a co-training architecture with two peer networks, ensuring decoupling between the source of neighbors and the model being updated, thereby mitigating confirmation bias. High-confidence, clean samples are dynamically curated into Cross-Model Memory Queues, and serve as a continually refined manifold reference. The small-loss hypothesis is used to bootstrap robust training, leading to a two-phase process: (i) robust pretraining using Symmetric Cross-Entropy (to avoid overfitting at initialization), and (ii) dynamic partitioning of data into clean and noisy subsets via GMM-fitted loss distributions.

Manifold Stabilization and Geometric Consistency

Clean pairs are used not only for inter-modal alignment—via bi-directional triplet ranking loss—but also for explicit intra-modal constraints, regularizing the feature space to enable reliable neighbor retrieval. Positive pairs are synthesized through random dropout, imposing ranking penalties within image and text features to foster a compact and semantically faithful geometric backbone.

Graph-Guided Continuous Rectification

Noisy samples are rectified by retrieving Top-$K$ nearest intra-modal neighbors from the peer memory bank. Unlike hard assignment (top-1 selection) or naive averaging, the method uses a learnable Graph Refiner—specifically, a single-layer multi-head self-attention mechanism over the neighbor set—to synthesize a soft prototype as a supervision target. This approach dynamically attenuates outlier neighbors and accentuates local semantic consensus, folding information from both relational density and global context into prototype generation.

Figure 2: The IN$^2$R framework—(top) clean pairs drive manifold stabilization and feed elite representations into Cross-Model Memory; (bottom) for noisy pairs, a graph refiner aggregates neighbor evidence to synthesize robust supervision targets.

The synthesized prototypes are converted into soft targets using a softmax over cosine similarity scores, and robust supervision is enforced using Symmetric Cross-Entropy with reverse cross-entropy regularization, which enhances tolerance to miscalibration.

Empirical Performance and Detailed Ablations

Robustness under Simulated and Real-World Noise

Extensive experiments are performed on Flickr30K, MS-COCO, and CC152K. IN$^2$R consistently achieves SOTA text-to-image and image-to-text retrieval scores, particularly excelling as the noise ratio increases:

• Flickr30K (80% noise): IN$^2$R yields an rSum of 458.8, outstripping the strongest discrete rectification baseline (PCSR) by +21.3.
• MS-COCO (80% noise): IN$^2$R achieves 501.3 (+7.6 over PCSR).
• CC152K (Real-World Web Noise): IN$^2$R sets a new SOTA with an rSum of 380.8, demonstrating efficacy without explicit noise priors.

Crucially, IN$^2$0R does not simply outperform in high-noise regimes—improvements persist even with 20-40% synthetic noise, highlighting the benefit of continuous prototype-driven supervision in feature structuring and generalization.

Component Contribution and Mechanism Analysis

Comprehensive ablation on Flickr30K (60% noise) reveals:

• Adding intra-modal geometric constraints alone improves rSum by +6.1.
• Applying only the graph-guided rectification yields +14.6.
• Combining both achieves +26.1, underlining their orthogonality.

The superiority of graph refiner-based continuous rectification is reflected in the following delta over alternatives: +15.6 compared to top-1 discrete selection and +10.0 compared to mean pooling.

Refiner Architecture and Memory Management

Multi-head self-attention (MHSA) outperforms graph convolution (GCN) and graph attention (GAT) architectures, with MHSA's global modeling capacity being decisive for robust prototype synthesis.

Decoupled cross-model memory—where networks retrieve only from their peer's memory, not their own—ameliorates confirmation bias. The elitist rolling update, filtering low-confidence samples, further ensures that only trustworthy representations participate in neighbor synthesis.

Figure 3: Hyperparameter sensitivity shows R@1 improves monotonically with memory size $^2$1 (saturating at 32k) and peaks for neighbor count $^2$2, confirming that moderate outreach balances semantic consensus and noise suppression.

Feature Distribution Visualization

t-SNE projections show that, compared to discrete selection, IN$^2$3R feature spaces have significantly tighter clusters and clearer visual-semantic class boundaries, validating the geometric advantage conferred by continuous prototype supervision.

Figure 4: t-SNE visualization on Flickr30K. IN$^2$4R (left) yields compact, well-aligned image/text clusters, compared to scattered distributions with discrete selection (right).

Practical Implications and Theoretical Outlook

IN$^2$5R closes the gap between discrete candidate search and ideal, continuous supervision by harnessing the relational topology of intra-modal neighbors. Practically, this confers superior robustness to web-derived noise, reducing dependency on expensive manual annotation, and therefore democratizing scalable vision-LLM development.

Theoretically, the results challenge the sufficiency of traditional discrete proxy-based correction and motivate further investigation into neighborhood-driven reasoning, prototype learning, and memory bank dynamics. The interplay between local consensus and global semantics invites research into fairness: if the data manifold encodes societal bias, local consensus may entrench such biases unless corrected with out-of-distribution interventions.

Conclusion

The proposed IN$^2$6R framework demonstrates that continuous, graph-synthesized prototypes yield substantial robustness against noisy correspondence in image-text retrieval and general vision-language alignment tasks. By replacing single-point proxies with consensus-driven, attention-refined supervision targets, IN$^2$7R both elevates empirical performance and redefines methodological standards for learning in noisy, large-scale, multi-modal regimes. Further extensions will likely integrate explicit fairness modeling and adapt the paradigm to broader multi-modal and open-world scenarios.