Noise-Resilient Contrastive Learning (NRCL)

• NRCL is a set of methods that uses contrastive objectives to preserve clean semantic information amid noisy labels and corrupted views.
• It employs strategies like contrastive pre-training and regularization to reduce overfitting to noisy labels and improve representation quality.
• Robust loss designs such as Reverse InfoNCE and SymNCE, along with adaptive pair construction, form key mechanisms in NRCL.

Noise-Resilient Contrastive Learning (NRCL) denotes a family of methods that use contrastive objectives, contrastive pre-training, or contrastive regularization to make learned representations and downstream predictors less sensitive to corrupted supervision, false positive pairs, noisy views, or structured background interference. In the noisy-label setting, the central premise is that deep networks trained directly with categorical or softmax cross-entropy memorize corrupted labels, whereas contrastive learning can organize the representation space before or during supervised learning so that clean semantic structure is preserved and noisy supervision has less leverage over optimization (Ciortan et al., 2021, Yi et al., 2022, Ghosh et al., 2021).

1. Problem formulation and scope

NRCL is most prominently studied in learning with noisy labels. The standard setup distinguishes the true label from the corrupted training label: clean data are sampled as $(x_i,y_i)\sim \mathrm{P}(X,c)$ or $(X,Y)$, whereas training often proceeds with noisy labels $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$ or $(X,\tilde Y)$ (Cui et al., 2 Jan 2025, Yi et al., 2022). In this regime, overparameterized classifiers trained with cross-entropy exhibit a memorization effect: they first fit easy or clean patterns and then progressively absorb corrupted labels into the representation and classifier, which degrades generalization (Yi et al., 2022, Ghosh et al., 2021).

The same robustness question appears beyond label corruption. In supervised contrastive learning, mislabeled examples create false positives among nominally same-class pairs; in self-supervised contrastive learning, overly aggressive augmentations or multimodal misalignment create noisy views with little shared information; in high-dimensional paired data, shared signal may be obscured by structured background variation; and in speech separation, background noise may be absorbed into speaker streams unless noise is explicitly represented (Chuang et al., 2022, Wu et al., 15 Nov 2025, Zhang et al., 2023). Across these settings, NRCL studies how pair construction, objective design, and representation geometry can be modified so that contrastive learning remains effective under corruption.

A recurring claim is that robustness is mediated by representation learning rather than only by the supervised loss. This perspective is explicit in noisy-label image classification, where contrastive pre-training is used to learn good feature representations before classification, and in contrastive regularization methods, where the encoder is constrained so that clusters reflect true classes rather than corrupted labels (Ciortan et al., 2021, Yi et al., 2022).

2. Algorithmic families and training pipelines

Across the literature, NRCL appears in several recurring training patterns.

Family	Core mechanism	Representative papers
Contrastive pre-training	Learn encoder first, then train or fine-tune classifier	(Ciortan et al., 2021, Ghosh et al., 2021)
Contrastive regularization	Add contrastive term to cross-entropy training	(Yi et al., 2022)
Two-view consistency triage	Use agreement across views for clean/ID/OOD handling	(Yao et al., 2021)

The two-stage pipeline is the canonical noisy-label formulation. In "A Framework using Contrastive Learning for Classification with Noisy Labels" (Ciortan et al., 2021), Phase A trains an encoder with contrastive learning on images, then trains a classifier on top of the learned representation to produce pseudo-labels. Phase B is optional: the pre-trained classifier predicts train-set pseudo-labels $\widehat y$, a 2-component Gaussian Mixture Model is fit to the per-sample loss distribution, the probability of belonging to the clean component becomes a sample weight, weighted supervised contrastive learning is applied, and a classification head is then trained or fine-tuned to produce final predictions. The paper also notes a warm-up variant in supervised training: first train only the classifier head while freezing the encoder, then train the full model (Ciortan et al., 2021).

A closely related formulation uses self-supervised pre-training followed by full end-to-end fine-tuning. "Contrastive Learning Improves Model Robustness Under Label Noise" (Ghosh et al., 2021) uses SimCLR on the same noisy-label image dataset while ignoring labels, discards the projection head after pre-training, initializes the classifier backbone with the learned encoder, initializes the classification head with zero weights and biases, and fine-tunes the entire classifier. The encoder is explicitly not frozen in this pipeline (Ghosh et al., 2021).

A different family integrates contrastive learning directly into supervised optimization. "On Learning Contrastive Representations for Learning with Noisy Labels" (Yi et al., 2022) defines the overall objective as

$$\mathcal{L} = \mathcal{L}_{\text{ce}} + \lambda \widetilde{\mathcal{L}}_{\text{ctr}},$$

with the key regularizer

$\widetilde{\mathcal{L}}_{\text{ctr}}(x_i,x_j) = \Big(\log(1-\langle \tilde q_i,\tilde z_j\rangle)+\log(1-\langle \tilde q_j,\tilde z_i\rangle)\Big)\mathbbm{1}\{p_i^\top p_j \ge \tau\}.$

Here the confidence threshold $p_i^\top p_j \ge \tau$ replaces direct reliance on noisy label agreement, and the log-barrier-like form changes the gradient behavior so that more similar pairs receive larger gradients (Yi et al., 2022).

Jo-SRC extends the two-view idea in a consistency-based direction. It generates two augmented views per sample, estimates a clean likelihood with Jensen-Shannon divergence, distinguishes in-distribution noisy from out-of-distribution noisy samples via view disagreement, relabels the three subsets differently with a mean-teacher model, and optimizes a joint classification plus consistency regularization loss (Yao et al., 2021). This is not classical pairwise embedding contrast, but it is explicitly framed as a contrastive-learning-style noisy-label method.

3. Theoretical mechanisms of robustness

Several papers provide distinct but complementary explanations for why NRCL works.

A general risk-consistency account is given in "An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise" (Cui et al., 2 Jan 2025). Under class-balanced data and symmetric label noise,

$$\widetilde{\mathcal{R}(\mathcal L;f)} = \left(1-\frac{C}{C-1}\gamma\right)^2 \mathcal{R}(\mathcal L;f) + \frac{C\gamma}{C-1}\left(2-\frac{C}{C-1}\gamma\right)\Delta \mathcal{R}(\mathcal L;f).$$

The key robustness criterion is that if there exists a constant $A$ such that $(X,Y)$0 for all representation functions $(X,Y)$1, then minimizing noisy risk is equivalent to minimizing clean risk; the loss is noise tolerant if $(X,Y)$2. A central conceptual consequence is that label noise changes the positive-pair distribution, whereas negatives remain sampled from the overall input distribution (Cui et al., 2 Jan 2025).

A representation-theoretic account appears in "Investigating Why Contrastive Learning Benefits Robustness Against Label Noise" (Xue et al., 2022). There, self-supervised contrastive learning is shown to produce a representation matrix with one prominent singular value corresponding to each sub-class and significantly smaller remaining singular values, together with large alignment between the prominent singular vectors and the clean labels of each sub-class. A linear layer trained on such representations can therefore learn the clean labels without overfitting the noise. The same work further argues that the Jacobian of a contrastively pre-trained deep network has a low-rank structure with shrunken tail singular values, which slows memorization during early fine-tuning (Xue et al., 2022).

A mutual-information account is developed in the CTRR formulation. "On Learning Contrastive Representations for Learning with Noisy Labels" (Yi et al., 2022) states that representations $(X,Y)$3 learned by minimizing $(X,Y)$4 maximize the mutual information $(X,Y)$5. Under the paper’s $(X,Y)$6-distribution assumption, the optimal representation $(X,Y)$7 satisfies

$(X,Y)$8

and

$(X,Y)$9

This formalizes the claim that the learned representation retains almost all true-label information while suppressing information tied to corrupted labels (Yi et al., 2022).

4. Robust loss design and pair construction

A large part of NRCL concerns replacing or modifying standard InfoNCE-style objectives so that false positives, mislabeled positives, or noisy views do not dominate learning.

The pre-training-plus-refinement framework in (Ciortan et al., 2021) uses standard contrastive pre-training and then introduces weighted supervised contrastive learning. The weight is the GMM-estimated probability that sample $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$0 belongs to the clean component,

$(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$1

where $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$2 is the sample loss. These weights are then used to reduce the influence of likely mislabeled samples in the supervised contrastive phase. The same paper combines this with pseudo-labeling, sample selection with Gaussian Mixture models, and optional dynamic bootstrapping with mixup (Ciortan et al., 2021).

The theoretical framework of (Cui et al., 2 Jan 2025) argues that the popular InfoNCE loss is non-robust because its additional noisy-label risk depends on the learned representation. To repair this, it defines Reverse InfoNCE and the symmetric combination

$(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$3

for which the additional risk becomes $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$4, a constant. The same framework also reinterprets nearest-neighbor sample selection and RINCE, showing that for RINCE the additional risk becomes zero when $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$5 (Cui et al., 2 Jan 2025).

"Robust Contrastive Learning against Noisy Views" (Chuang et al., 2022) develops RINCE as a drop-in replacement for InfoNCE:

$(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$6

As $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$7, RINCE approaches InfoNCE; at $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$8, it yields a fully symmetric robust form. The paper’s intuition is that InfoNCE emphasizes hard positives, which in noisy regimes often correspond to false positive pairs, whereas larger $(x_i,\tilde y_i)\sim \mathrm{P}(X,\tilde c)$9 downweights suspicious pairs implicitly and therefore improves robustness (Chuang et al., 2022).

A separate direction reinterprets augmentation itself as noise modeling. "Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise" (Zhang et al., 2024) argues that predefined augmentation in standard contrastive learning can be regarded as a point estimation of $(X,\tilde Y)$0-noise, and proposes PiNDA, which learns $(X,\tilde Y)$1 end-to-end so that the augmentation becomes task-beneficial rather than merely fixed or hand-designed (Zhang et al., 2024).

5. Empirical performance and application domains

In noisy-label image classification, the empirical pattern is consistent: contrastive pre-training improves both non-robust and robust supervised losses. "A Framework using Contrastive Learning for Classification with Noisy Labels" (Ciortan et al., 2021) evaluates on CIFAR-10, CIFAR-100, mini-Webvision, and Clothing1M, and reports that contrastive pre-training improves CE, NFL+RCE, and ELR across all noise settings; gains are especially large at 80% symmetric noise, with margins of roughly 10–75 points on CIFAR-10 and 5–30 points on CIFAR-100 in the noisiest settings. The paper further concludes that ELR with contrastive pre-training tends to perform best overall, while the extra fine-tuning stage can further improve accuracy but at the cost of additional complexity (Ciortan et al., 2021).

The pre-train-then-fine-tune formulation in (Ghosh et al., 2021) reports especially large gains under extreme noise. Under 90% symmetric noise on CIFAR-10, CCE with random initialization yields 42.7%, DivideMix yields 93.2%, and CCE with SimCLR initialization yields 82.9%. Under 90% symmetric noise on CIFAR-100, CCE with random initialization yields 10.1%, DivideMix 31.5%, and CCE with SimCLR initialization 52.11%. On Clothing1M, CCE, $(X,\tilde Y)$2, and MAE with SimCLR initialization achieve 73.27%, 73.35%, and 73.36%, respectively (Ghosh et al., 2021).

The CTRR formulation reports that NRCL achieves the best accuracy across all noise settings on CIFAR-10 and CIFAR-100, with gains becoming larger as noise increases; on CIFAR-10 with 90% symmetric noise, CTRR reaches about 81.65%, and the method also outperforms baselines on ANIMAL-10N and Clothing1M (Yi et al., 2022). SymNCE extends the supervised-contrastive line to CIFAR-10, CIFAR-100, TinyImageNet, and Clothing1M, where SupCon is strong when noise is low but degrades strongly as noise increases, whereas SymNCE consistently improves over SupCon under both symmetric and asymmetric noise and achieves the best reported performance among the compared methods on Clothing1M (Cui et al., 2 Jan 2025).

Robustness to noisy views is not confined to label-noise classification. RINCE improves image, video, and graph contrastive learning: on ImageNet it changes SimCLR from 69.3 to 70.0 top-1 and MoCo-v3 from 73.8 to 74.2; on Kinetics400 transfer it changes HMDB51 from 57.8 to 61.6 and UCF101 from 88.6 to 88.8; and on TUDataset it reaches, for example, 90.9 ± 0.6 on RDT-B versus an InfoNCE baseline of 89.9 ± 0.4 (Chuang et al., 2022).

Medical image classification provides a distinct application regime. "Contrastive-Based Deep Embeddings for Label Noise-Resilient Histopathology Image Classification" (Dedieu et al., 2024) evaluates six public histopathology datasets and shows that embedding-based methods match or outperform image-based methods across almost all datasets and noise levels, while contrastive embeddings degrade much more slowly as noise increases than non-contrastive embeddings. The effect is consistent under both uniform and asymmetric label noise, and is strongest when training a linear head on frozen embeddings (Dedieu et al., 2024).

Speech separation provides another NRCL interpretation. "Noise-Aware Speech Separation with Contrastive Learning" (Zhang et al., 2023) treats background noise as an additional output and applies patch-wise contrastive learning between noise and speaker representations. On WHAM! and LibriMix, NASS achieves 1 to 2dB SI-SNRi or SDRi over DPRNN and Sepformer with less than 0.1M parameter increase (Zhang et al., 2023).

6. Limitations, controversies, and research directions

The empirical success of NRCL does not eliminate its practical costs. The two-stage framework of (Ciortan et al., 2021) explicitly notes that additional fine-tuning improves accuracy at the cost of additional complexity and computational cost, and that hyperparameter sensitivity remains an issue, especially under severe noise. In histopathology, robustness is lower on small datasets such as BACH and MHIST, asymmetric noise is harder to resist than uniform noise, and the study uses synthetic noise rather than real-world annotation noise (Ciortan et al., 2021, Dedieu et al., 2024).

A central controversy concerns the status of InfoNCE and standard supervised contrastive learning under noise. The 2025 theoretical framework argues that InfoNCE is non-robust and that SupCon degrades strongly as noise increases, whereas robustness requires either a constant additional-risk term, nearest-neighbor positive selection, or an explicitly symmetric objective such as SymNCE (Cui et al., 2 Jan 2025). This suggests that “contrastive learning helps under noise” is too coarse a statement; the specific loss and pair-construction rule matter.

Theoretical limits under stronger corruption are stricter still. "Contrastive Learning with Nasty Noise" (Zhao, 25 Feb 2025) studies adversarial sample modification or replacement and proves that if $(X,\tilde Y)$3, no algorithm can PAC learn a non-trivial hypothesis class under nasty noise rate $(X,\tilde Y)$4 with success probability $(X,\tilde Y)$5. The same paper argues that robustness depends on VC dimension, Rademacher complexity, bounded representations, and Lipschitz losses, thereby shifting part of the NRCL agenda from objective engineering to geometric complexity control (Zhao, 25 Feb 2025).

A related direction emphasizes explicit uniformity. "PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning" (Wu et al., 15 Nov 2025) argues that robust contrastive representation learning in noisy, high-dimensional settings requires not only alignment of positive pairs but also explicit control of feature dispersion. In that linear setting, the hard uniformity constraint $(X,\tilde Y)$6 turns contrastive PCA into a generalized eigenproblem that regularizes against background interference. A plausible implication is that future NRCL systems may increasingly combine alignment, pair filtering, and explicit feature isotropy rather than relying on alignment alone (Wu et al., 15 Nov 2025).