IN2R: Intra-modal Noise Rectification

• The paper introduces IN2R, a framework that synthesizes continuous supervision targets from intra-modal neighbor consensus to rectify noisy image-text pairs.
• It employs a co-training strategy with dynamic cross-model memory, robust warm-up, and graph refinement to mitigate the pitfalls of discrete selection.
• Empirical results on Flickr30K, MS-COCO, and CC152K demonstrate enhanced retrieval performance under high synthetic and real-world noise conditions.

Searching arXiv for the specified method and closely related papers to ground the article in current literature. Intra-Modal Neighbor-aware Noise Rectification (IN$^2$R) is a framework for cross-modal retrieval under noisy image-text correspondence that rectifies corrupted inter-modal supervision by exploiting the relative stability of intra-modal neighborhood geometry. Rather than filtering noisy pairs, reweighting them, or replacing them with a single substitute sample, IN$^2$R shifts correction toward the synthesis of a continuous supervision target from local semantic consensus. It does so through a co-training framework with two peer networks, a dynamic Cross-Model Memory, and a Graph Refiner that performs relational reasoning over retrieved intra-modal neighbors to construct soft prototypes for rectification (Liu et al., 2 Jun 2026).

1. Problem formulation and conceptual position

IN$^2$R studies cross-modal retrieval on datasets of image-text pairs,

$\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$

where the objective is to train an image encoder $f_\theta(\cdot)$ and a text encoder $g_\phi(\cdot)$ so that matched pairs are close and mismatched pairs are far apart in a shared embedding space (Liu et al., 2 Jun 2026). Its motivating failure mode is inter-modal noisy correspondence: an image is paired with a caption that does not actually describe it, or vice versa. Under standard ranking or contrastive training, such corruption directly misguides optimization by reinforcing false semantic associations.

The framework is explicitly positioned against three families of prior noisy-correspondence methods: filtering / sample selection, consistency / reweighting, and correction-based approaches (Liu et al., 2 Jun 2026). The paper argues that many correction-based methods remain within a Discrete Selection paradigm, in which a noisy pair is repaired by selecting a single proxy from the dataset. Two critiques structure this argument.

The first is Single-Point Fragility: if supervision depends on a single selected proxy, rectification can fail when that proxy is itself noisy, semantically partial, or otherwise suboptimal. The second is Discretization Error: the true semantic target may lie on a continuous manifold and need not coincide with any one existing sample. This motivates the central shift in IN$^2$R from discrete replacement to continuous target synthesis (Liu et al., 2 Jun 2026).

The paper encapsulates its core intuition in the statement “Intra-Modal Neighbors Never Lie.” This does not mean every neighbor is correct in isolation. Rather, it asserts that while cross-modal pairings may be corrupted, the neighborhood structure among images or among texts is often much more stable. A plausible implication is that intra-modal topology can serve as a more trustworthy carrier of semantic evidence than the original noisy correspondence itself (Liu et al., 2 Jun 2026).

2. Core architecture and training pipeline

IN$^2$R uses a co-training framework with two peer networks, denoted $\mathcal{M}_A$ and $\mathcal{M}_B$, and divides training into a warm-up phase and a co-training phase with dynamic partition (Liu et al., 2 Jun 2026).

During warm-up, the model is trained on the full noisy dataset using a robust symmetric loss based on Symmetric Cross Entropy (SCE). The stated purpose is to form an initial discriminative embedding space and make clean/noisy samples separable in terms of per-sample loss. After warm-up, each epoch computes per-sample losses, fits a two-component Gaussian Mixture Model (GMM) to the loss distribution, and partitions data into a labeled clean subset $^2$0 and an unlabeled noisy subset $^2$1 (Liu et al., 2 Jun 2026).

The clean and noisy branches are optimized differently.

For the clean-data branch, IN$^2$2R applies Manifold Stabilization. Clean samples are trained with inter-modal alignment via bidirectional triplet ranking and with intra-modal geometric consistency induced by dropout-generated positive views within each modality. These high-confidence representations are then pushed into the memory module (Liu et al., 2 Jun 2026).

For the noisy-data branch, the original pair is not trusted. Instead, the model encodes a noisy image or text query, retrieves top-$^2$3 nearest neighbors in the same modality from the peer network’s memory, collects the paired target-modality features of those neighbors, refines them with the Graph Refiner, and mean-pools the refined nodes to synthesize a Soft Textual Prototype or Soft Visual Prototype. That prototype is then converted into a soft target distribution and used in a robust symmetric rectification loss $^2$4 (Liu et al., 2 Jun 2026).

This yields the pipeline: $^2$5 followed either by clean training with $^2$6 or noisy rectification through neighbor retrieval, graph refinement, prototype synthesis, and soft-target supervision (Liu et al., 2 Jun 2026).

3. Mathematical formulation

For a clean batch $^2$7, normalized embeddings are

$^2$8

with similarity

$^2$9

(Liu et al., 2 Jun 2026).

The warm-up robust objective uses symmetric cross entropy,

$^2$0

where $^2$1 is the predicted temperature-scaled softmax distribution of similarity scores and $^2$2 is the target distribution. In warm-up, $^2$3 is one-hot ground truth, with $^2$4-smoothing applied in the reverse cross-entropy term for stability. The bidirectional warm-up loss is

$^2$5

(Liu et al., 2 Jun 2026).

For clean samples, the bidirectional triplet ranking loss is

$^2$6

with inter-modal and intra-modal components

$^2$7

$^2$8

and total clean loss

$^2$9

(Liu et al., 2 Jun 2026).

For noisy samples, a query retrieves top-$\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$0 intra-modal neighbors from the peer memory. For an image query $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$1, the method retrieves nearest visual neighbors from the stored visual keys, then gathers their paired textual values

$\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$2

(Liu et al., 2 Jun 2026).

The Graph Refiner stacks these neighbor features as $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$3 and applies Multi-Head Self-Attention: $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$4 followed by

$\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$5

(Liu et al., 2 Jun 2026).

The continuous prototype is then synthesized by uniform mean pooling over refined features: $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$6 The same procedure symmetrically yields $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$7 for text-to-image rectification. The prototype induces a soft target distribution over the current batch: $\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$8 (Liu et al., 2 Jun 2026).

The rectification loss is

$\mathcal{D}=\{(I_i, T_i)\}_{i=1}^N,$9

and the total training objective for each peer network is

$f_\theta(\cdot)$0

(Liu et al., 2 Jun 2026).

4. Dynamic Cross-Model Memory and graph-based intra-modal reasoning

The memory module is central to IN$f_\theta(\cdot)$1R because rectification depends on retrieving trustworthy intra-modal neighbors. The Cross-Model Memory is defined as

$f_\theta(\cdot)$2

where each entry stores a paired image embedding and text embedding produced by the peer network (Liu et al., 2 Jun 2026).

This memory is cross-model in the sense that $f_\theta(\cdot)$3 retrieves from $f_\theta(\cdot)$4’s memory and $f_\theta(\cdot)$5 retrieves from $f_\theta(\cdot)$6’s memory. The stated rationale is to reduce confirmation bias: a model should not validate a noisy sample using its own potentially corrupted historical beliefs (Liu et al., 2 Jun 2026).

The update strategy is an Elitist Rolling Update. At epoch $f_\theta(\cdot)$7, a dynamic confidence threshold

$f_\theta(\cdot)$8

is defined as the average confidence of the current clean set. Only samples with confidence $f_\theta(\cdot)$9 are considered elite; their embeddings are detached from the computation graph and inserted into a FIFO queue, replacing the oldest entries (Liu et al., 2 Jun 2026). The paper explicitly states dynamic high-confidence filtering, FIFO maintenance, and gradient detachment, but does not give an explicit formula for $g_\phi(\cdot)$0.

The Graph Refiner converts raw top-$g_\phi(\cdot)$1 retrieval into consensus-aware semantic evidence. The retrieved target-modality features are interpreted as nodes in a fully connected local graph, with affinities computed dynamically through self-attention. This means the graph is local, dynamic, and fully connected (Liu et al., 2 Jun 2026). The paper argues that raw nearest neighbors are insufficient because they only reflect similarity to the query, whereas refined neighbors additionally encode agreement among themselves. A plausible implication is that structurally inconsistent but query-similar outliers can be attenuated before prototype synthesis.

The appendix specifies that the Graph Refiner is implemented as a single-layer Transformer encoder with 4 heads (Liu et al., 2 Jun 2026). Ablation compares GCN, GAT, and MHSA, with MHSA performing best. The result supports the specific role of dense consensus reasoning rather than generic graph smoothing.

5. Empirical results and ablation evidence

IN$g_\phi(\cdot)$2R is evaluated on Flickr30K, MS-COCO, and CC152K (Liu et al., 2 Jun 2026). Flickr30K contains 31K images with 5 captions each, with 1K images for validation and 1K for test following VSE++. MS-COCO contains 123K images with 5 captions each, 566K training pairs, 25K validation, and 25K test, evaluated on 5-fold 1K test splits. CC152K is a subset of Conceptual Captions with real web noise, with 150K training images, 1K validation, and 1K test (Liu et al., 2 Jun 2026).

For Flickr30K and MS-COCO, synthetic symmetric noise is created by random shuffling at rates 20%, 40%, 60%, and 80%; CC152K is evaluated under real-world web noise (Liu et al., 2 Jun 2026). Retrieval metrics are R@1, R@5, R@10, and rSum, the sum of six recall values across both directions.

A compact summary of selected main results is shown below.

Dataset / noise	IN$g_\phi(\cdot)$3R result	Comparison noted
Flickr30K, 20%	rSum 511.6	SPS 507.1, GSC 505.8, PCSR 505.7
Flickr30K, 60%	rSum 495.2	SPS 484.1, GSC 478.0, PCSR 480.5
Flickr30K, 80%	rSum 458.8	PCSR 437.5, CREAM 416.3, L2RM 404.0
MS-COCO, 20%	rSum 527.3	ESC 526.8, GSC 525.7, SPS 525.5
MS-COCO, 80%	rSum 501.3	PCSR 493.7, CREAM 487.2, L2RM 482.6
CC152K, real noise	rSum 380.8	SPS 376.3, ESC 375.9, PCSR 375.3

The strongest gains are reported in high-noise synthetic settings, especially 80% noise, and on real-world web-noise data (Liu et al., 2 Jun 2026). The paper interprets this as evidence that continuous rectification becomes most useful when direct inter-modal supervision is highly unreliable.

The ablation studies are especially revealing. On Flickr30K with 60% noise, component ablation gives:

• baseline $g_\phi(\cdot)$4 only: 469.1
• • intra-modal constraints $g_\phi(\cdot)$5: 475.2
• • rectification $g_\phi(\cdot)$6 directly: 483.7
• full IN$g_\phi(\cdot)$7R: 495.2 (Liu et al., 2 Jun 2026)

This sequence indicates that manifold stabilization and rectification are complementary. The paper’s interpretation is that better geometry improves neighbor retrieval, and better retrieval improves rectification.

The comparison between continuous and discrete correction is also direct. On Flickr30K with 60% noise:

• None (discard noise): 467.7
• Hard Selection (Top-1): 479.6
• Mean Pooling (Top-$g_\phi(\cdot)$8): 485.2
• Graph Refiner: 495.2 (Liu et al., 2 Jun 2026)

This experimentally grounds the paper’s conceptual critique of discrete selection. Top-1 exhibits Single-Point Fragility, mean pooling improves robustness but cannot suppress outliers, and graph-based continuous synthesis performs best.

Additional ablations show:

• GCN: 486.1
• GAT: 490.5
• MHSA: 495.2

and for memory construction:

• Self-Memory: 482.7
• Full Queue (No Filtering): 488.1
• Elitist Cross-Memory: 495.2 (Liu et al., 2 Jun 2026)

These results support the importance of both cross-model decoupling and high-confidence elite filtering.

6. Relation to adjacent neighbor-aware noise methods

IN$g_\phi(\cdot)$9R occupies a specific position within a broader family of neighbor-aware noisy-supervision methods, but it is not interchangeable with prior approaches.

A closely related earlier direction is INN, “Integration with the Nearest Neighborhoods,” which addresses supervised classification with noisy labels by identifying clean-labeled samples via local prediction consistency in feature-space neighborhoods rather than small-loss heuristics (Kim et al., 2021). INN is strongly related in spirit because it uses nearest neighbors in learned feature space, probes the local region between a sample and its neighbors, and derives a cleanliness score from the integrated confidence assigned to the observed label: $^2$0 (Kim et al., 2021). However, INN focuses on clean-sample identification and partitioning for noisy-label classification rather than cross-modal rectification of image-text correspondence.

Another related line is Neighbor-aware Instance Refining with Noisy Labels (NIRNL) for cross-modal retrieval (Liu et al., 30 Dec 2025). NIRNL uses intra-modal nearest neighbors in the visual and text spaces separately, generates soft labels from neighbor label frequency, and partitions samples into pure, hard, and noisy subsets through cross-modal neighborhood consensus (Liu et al., 30 Dec 2025). This creates an overlap with IN$^2$1R in that same-modality neighborhood evidence is used to assess supervision quality. The difference is structural: NIRNL is a noisy-label framework operating on class-labeled image-text pairs, whereas IN$^2$2R rectifies inter-modal correspondence itself by synthesizing continuous supervision targets from paired features of retrieved neighbors (Liu et al., 2 Jun 2026, Liu et al., 30 Dec 2025).

By contrast, “Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation” addresses diffusion-based image-to-video generation and introduces a tuning-free, plug-and-play noise rectification strategy that operates in latent/noise space during denoising (Li et al., 2024). Although it uses the term “noise rectification,” the provided record explicitly states that it does not introduce a formal module called IN2R and contains no explicit “neighbor-aware” mechanism (Li et al., 2024). It is therefore terminologically related but methodologically distinct.

These comparisons help delimit IN$^2$3R precisely. It is neither a pure sample-selection method nor a direct descendant of diffusion-time noise correction. Its specific contribution is a graph-based, intra-modal, continuous-target rectification mechanism for noisy cross-modal supervision (Liu et al., 2 Jun 2026).

7. Limitations, assumptions, and significance

The paper attributes several strengths to IN$^2$4R. It is especially robust under heavy noise, uses continuous supervision rather than a brittle top-1 pseudo-target, leverages semantic neighborhood consensus, reduces self-confirmation through cross-model memory, and incurs no inference overhead because the memory bank and graph refiner are training-time only (Liu et al., 2 Jun 2026).

At the same time, the design carries clear assumptions and costs. Training is more complex than a standard retrieval model because it adds dual peer networks, GMM partitioning, large memory queues, neighbor search, and graph refinement (Liu et al., 2 Jun 2026). Memory cost can be substantial; the reported configurations use memory size $^2$5 for Flickr30K and $^2$6 for MS-COCO, with top-$^2$7, temperature $^2$8, warm-up $^2$9, and 40 co-training epochs (Liu et al., 2 Jun 2026).

The method also depends on retrieval quality. If the intra-modal manifold is not yet well formed, neighbor retrieval may be poor. This is why warm-up, manifold stabilization, and elite memory filtering are built into the framework (Liu et al., 2 Jun 2026). The paper further notes that sparse or biased neighborhoods may weaken local consensus, and its impact statement mentions possible amplification of societal bias by relying on neighborhood consensus (Liu et al., 2 Jun 2026).

A further limitation is specification granularity. The paper does not fully detail the exact confidence computation used in clean/noisy identification or the exact GMM thresholding behavior (Liu et al., 2 Jun 2026). For implementation, this leaves some lower-level choices underdetermined.

In significance, IN$^2$0R represents a shift in how noisy cross-modal learning is framed. Instead of assuming that the right correction is a more reliable discrete pair, it assumes that the more reliable signal is the local consensus structure within each modality, and that this signal should be expressed as a continuous soft prototype. Within the landscape of neighbor-aware noise handling, this is the defining distinction of IN$^2$1R: it turns intra-modal neighborhoods into cross-modal rectification targets through graph-based reasoning and prototype synthesis (Liu et al., 2 Jun 2026).