---
title: Label Noise in Machine Learning
url: https://www.emergentmind.com/topics/label-noise
type: topic
---

# Label Noise in Machine Learning

Label noise is the corruption, uncertainty, or incompleteness affecting labels supplied with training examples. It arises when an observed label $\widetilde{Y}$ differs from an unobserved clean label $Y$, when multiple annotators disagree, or when a multi-label annotation omits present classes or includes absent ones. In supervised learning, the learner observes $(X,\widetilde{Y})$ but is generally evaluated against $(X,Y)$. Its consequences depend not only on the corruption rate but also on whether errors are class-conditional, instance-dependent, concentrated near decision boundaries, adversarially selected, associated with graph topology, or distributed over individual entries of a multi-label vector. Research on label noise therefore encompasses statistical modeling, detection, correction, robust optimization, representation learning, semi-supervised learning, and benchmark construction.

## 1. Formalization and taxonomy

Let $X$ denote an input, $Y\in\{1,\ldots,K\}$ its clean class, and $\widetilde{Y}$ its observed label. The learner has access to samples from $P_{X\widetilde{Y}}$, while the target risk is defined under $P_{XY}$. A general feature-dependent corruption mechanism is described by

$$
T(x)_{ij}=P(\widetilde{Y}=j\mid Y=i,X=x),
$$

so that, with an appropriate convention for row or column probability vectors,

$$
\widetilde{\boldsymbol{\eta}}(x)=T(x)^{\mathsf T}\boldsymbol{\eta}(x),
$$

where $\boldsymbol{\eta}(x)=P(Y\mid X=x)$ and $\widetilde{\boldsymbol{\eta}}(x)=P(\widetilde{Y}\mid X=x)$. Under class-conditional noise, $T(x)=T$ is independent of $x$; under instance-dependent noise, it varies with the individual input.

### Class-conditional and uniform noise

Uniform, or symmetric, noise retains the correct label with probability $1-\rho$ and distributes the remaining probability uniformly among the $K-1$ incorrect classes:

$$
T_{ii}=1-\rho,\qquad T_{ij}=\frac{\rho}{K-1}\quad(i\neq j).
$$

Class-dependent noise is also independent of $X$ conditional on $Y$, but its transition probabilities differ across source and destination classes. Pair-flipping noise maps one class preferentially to another, while structured class-dependent noise may reflect semantic confusions.

For uniform noise, the noisy posterior is an affine transformation of the clean posterior:

$$
\widetilde{\eta}_j(x)
=
\left(1-\rho-\frac{\rho}{K-1}\right)\eta_j(x)
+\frac{\rho}{K-1}.
$$

Consequently, the Bayes decision is preserved while

$$
\rho<\frac{K-1}{K}.
$$

This threshold is $50\%$ for binary classification and $90\%$ for ten classes. At the threshold, all noisy posterior coordinates become equal; beyond it, the ordering can reverse [2206.01106].

### Instance-dependent and feature-dependent noise

Instance-dependent noise (IDN), also called feature-dependent noise, satisfies

$$
P(\widetilde{Y}=j\mid Y=i,X=x)
$$

that varies with $x$. Two examples from the same clean class may therefore have different corruption probabilities. Ambiguous, atypical, or low-quality instances can be more likely to receive a wrong label, and the wrong class can depend on their visual or semantic content.

Feature-dependent noise is generally harder than uniform noise because it is aligned with the geometry of the data. Errors near a decision boundary can move the boundary even when the overall corruption rate is small. Under this regime, high training and noisy-validation accuracy can coexist with poor clean-test accuracy [2003.10471]. BERT classification exhibits the same phenomenon: automatic gazetteer labels in Hausa and Yoruba produce training noise rates of $50.37\%$ and $33.28\%$, respectively, while clean-test performance falls substantially relative to clean-data training [2305.16337].

### Open-set noise

Open-set noise occurs when training inputs originate from classes outside the target label space but are assigned labels from the target classes. It differs from closed-set corruption, in which every observed label belongs to the modeled class set. Experiments on CIFAR-10 using CIFAR-100 and ImageNet images demonstrate that open-set noise can be treated separately from ordinary label transitions [1910.00701].

### Multi-label noise

In multi-label classification, each example has a binary label vector $\mathbf{y}_i\in\{0,1\}^C$. Noise is entry-wise rather than necessarily image-wise.

- **Additive noise** changes $0\rightarrow1$: an absent class is annotated as present.
- **Subtractive noise** changes $1\rightarrow0$: a present class is omitted.
- **Mixed noise** combines both directions.

Subtractive noise removes positive evidence and is particularly harmful when positive labels are sparse. Mixed noise is more difficult because the learner must both suppress false positives and recover missing positives. In remote-sensing multi-label classification, uniform entry-wise corruption is dominated by additive errors because most class entries are negative [2207.13975].

### Real-valued label noise

For regression, observed labels can be modeled as

$$
y_i=f(x_i)+\varepsilon_i,
$$

with sample-specific Gaussian noise variances $\sigma_i$. Large learned variances indicate observations that are difficult to explain using the regression function and the other labels. This setting is distinct from categorical label noise because additive Gaussian residuals and continuous variances have no direct analogue for class labels [2103.11352].

## 2. Statistical effects on learning and generalization

Deep networks can fit arbitrary labels, including random labels. Under noisy supervision, they often learn consistently supported patterns before memorizing incorrect labels, but prolonged optimization can eventually fit the corrupted targets. The resulting training accuracy may remain high while clean generalization deteriorates.

The impact depends strongly on the structure of the corruption.

- **Uniform noise** produces dispersed incorrect gradients that can partially cancel, and additional data may dilute their influence.
- **Class-dependent noise** produces systematically biased gradients associated with particular class transitions.
- **Locally concentrated noise** creates corrupted regions in feature space and can induce incorrect local decision boundaries.
- **Feature-dependent noise** aligns incorrect labels with genuine feature similarities, allowing a network to learn a coherent but incorrect decision rule [2003.10471].

A theoretical feature-space analysis characterizes label noise through its scale and shape. Uniform noise can preserve the Bayes classifier up to the tipping point $\frac{K-1}{K}$, whereas class-dependent noise can change the decision boundary at substantially lower rates. Noise concentrated near a region where the boundary can move may reduce robustness at arbitrarily small total noise mass [2206.01106].

A related domain-adaptation formulation treats $P_{X\widetilde{Y}}$ as a source domain and $P_{XY}$ as a target domain with the same feature marginal but different posteriors. The relative signal strength (RSS)

$$
M(x;\eta,\widetilde{\eta})
=
\min_j
\frac{
\max_i[\widetilde{\eta}(x)]_i-[\widetilde{\eta}(x)]_j
}{
\max_i[\eta(x)]_i-[\eta(x)]_j
}
$$

measures how much clean posterior separation survives corruption. When RSS is high and the clean and noisy Bayes decisions agree, ordinary empirical risk minimization on noisy labels can be nearly minimax optimal. When RSS is low or the Bayes decisions disagree, correction cannot recover information absent from the noisy sample [2411.00079].

The same distinction appears in the theory of forward correction. If the true transition matrix is known and the base loss is proper, forward correction is population-consistent. However, finite-sample empirical risk can fall below the noisy population-risk floor because a high-capacity network memorizes the realized corruptions. For cross-entropy under symmetric noise, this floor is the entropy of the noisy posterior, motivating a noise-bounded objective that prevents the loss from decreasing below a theoretically estimated bound [2307.13100].

## 3. Label detection and estimation

Label-noise detection assigns each example a score indicating disagreement between its observed label and evidence from a model, neighboring examples, repeated annotations, or a regression posterior. Detection is conceptually distinct from subsequent filtering, relabeling, or robust training.

### Loss- and trajectory-based detection

The small-loss heuristic assumes that clean examples tend to have lower losses than noisy examples during early training. Co-teaching, DivideMix, and related methods use low-loss examples as presumed clean. This assumption fails under structured or adversarial noise. BadLabel deliberately selects examples and replacement labels so that noisy examples become easy to fit and their losses overlap with those of clean examples [2305.18377].

A recent benchmark decomposes detection methods into:

1. a label-agreement function;
2. an aggregation method over epochs or models;
3. an in-sample or out-of-sample information-gathering strategy.

The evaluated agreement functions include cross-entropy, Jensen–Shannon divergence, and the logit margin

$$
L_{\mathrm{LM}}(z_i,\widetilde{y}_i)
=
\max_{k\neq\widetilde{y}_i}z_{i,k}-z_{i,\widetilde{y}_i}.
$$

Aggregation can use the last epoch, mean agreement, mean predicted probabilities, clustering of loss trajectories, or stochastic weight averaging. Across vision and tabular settings, averaging probabilities over training and applying the logit margin—Mean Probabilities–LM—was the most consistently effective combination. Its fixed-budget evaluation reports the false negative rate (FNR) when the selected fraction equals the dataset noise rate [2510.16211].

### Graph-based and representation-based detection

NoiseRank treats detection as an unsupervised instance-ranking problem. It extracts vector representations, constructs class prototypes using $K$-means, predicts prototype labels using weighted $k$-nearest neighbors, and assigns each example a Markov Random Field score based on pairwise support and contradiction. Nearby examples with the same observed label provide negative support; nearby examples whose prototype predictions contradict their labels provide positive blame. The score is rank-equivalent to a posterior-like quantity rather than a calibrated probability. High-scoring examples are removed before retraining. Iterative NoiseRank reduces average detection error to $18.43$ on Food101N and $22.81$ on Clothing1M [2003.06729].

For graph node classification, label corruption alters neighborhood label distributions and can propagate through message passing. LENS uses leave-one-out changes in neighborhood entropy to identify noisy nodes and then relabels rather than discarding them [2610.22998]. LP4GLN reconstructs a graph with greater label compatibility, applies label propagation, and iteratively retains high-confidence corrections. Its design addresses heterophilous graphs, where raw adjacency does not imply label similarity [2310.16560].

### Regression detection with Gaussian processes

In Gaussian-process regression, each observation receives an individual variance $\sigma_i$. With $C=K+\Sigma$, where $\Sigma=\operatorname{diag}(\sigma_1,\ldots,\sigma_N)$, the negative log marginal likelihood is

$$
\log|C|+\boldsymbol{y}^{T}C^{-1}\boldsymbol{y}.
$$

The leave-one-out error and predictive standard deviation are

$$
e_{-i}=\frac{(C^{-1}\boldsymbol{y})_i}{(C^{-1})_{ii}},
\qquad
\sigma_{-i}=\frac{1}{\sqrt{(C^{-1})_{ii}}}.
$$

Maximum-likelihood optimization links the learned sample variance to the condition that the leave-one-out error is contained by posterior uncertainty. Samples with large $\sigma_i$ are ranked as likely corrupted. A multiplicative update preserves non-negativity and showed monotonic convergence empirically, although a complete convergence proof is supplied only for diagonal kernels [2103.11352].

### Transition-matrix estimation

Transition-matrix methods estimate $T$ and use it for forward or backward loss correction. Anchor-point estimators rely on examples whose clean class posterior is nearly one; clusterability methods rely on neighborhoods sharing a clean label. Both assumptions can fail for tabular data, text embeddings, sparse features, or strongly correlated representations.

“Beyond Images” decorrelates features, estimates coordinate informativeness using noisy-label $f$-mutual information, downweights weak coordinates in a soft-cosine similarity, and then applies HOC to two-neighbor label triples. Total-variation mutual information preserves feature-informativeness ordering exactly under binary class-dependent noise satisfying $e_1+e_2<1$; KL mutual information preserves it only up to an $\epsilon$ bound [2202.01273].

Instance-dependent transition matrices are generally non-identifiable from noisy observations alone: both $P(Y\mid X)$ and $P(\widetilde{Y}\mid Y,X)$ are unknown. ILDE approximates instance-specific label distributions by estimating noisy posteriors, computing mini-batch inter-class correlations, normalizing them into a transition proxy, and transforming the noisy posterior into soft clean-label supervision [2212.08380]. ReIDVAE avoids explicit transition estimation by using a structural causal model with content and style latents and a noisy-label decoder conditioned on the input and latent representation [2305.18377].

## 4. Robust training and label correction

Robust training methods modify the supervision process rather than merely detecting errors. They may reweight examples, replace labels, impose consistency, use semi-supervised learning, or regularize the training dynamics.

### Trusted-set meta-learning

“Distilling Effective Supervision from Severe Label Noise” uses a small trusted probe set $D_p$ to meta-optimize per-example weights and relabeling decisions for a much larger noisy set $D_u$. A one-step model update is differentiated through the probe loss to obtain exemplar weights. The method also forms pseudo-labels by averaging predictions over augmented views and temperature-sharpening them with $\tau=0.5$.

For each noisy example, the target combines the observed one-hot label $y_i$ and pseudo-label $g(x_i,\Phi)$ through

$$
\mathcal{P}(\lambda_i)=\lambda_i y_i+(1-\lambda_i)g(x_i,\Phi).
$$

A rectified sign of the meta-gradient determines whether to retain the observed label or replace it with the pseudo-label. The final training objective combines weighted supervision, hard corrected labels, probe-anchored mixup, and augmentation-consistency KL loss. On CIFAR-100, the method reaches $80.2\%$ at $40\%$ uniform noise and $75.5\%$ at $80\%$ noise with ten trusted images per class [1910.00701].

### Reweighting, relabeling, and semi-supervised learning

Co-teaching trains two networks that exchange small-loss examples. ELR regularizes predictions toward early-learning behavior. SAT progressively replaces hard labels with a moving combination of annotations and predictions. JoCoR uses two networks, agreement regularization, and sample selection. In multi-label settings, these methods must operate at the label-entry level because one image can contain both clean and noisy labels [2207.13975].

Robust DivideMix addresses BadLabel by perturbing labels adversarially before estimating a Bayesian Gaussian mixture over losses. The perturbation makes BadLabel examples more sensitive and restores loss separability. The method then performs cross-network MixMatch on mostly clean and unlabeled subsets, rejecting low-quality mixture divisions. On CIFAR-10 with $40\%$ BadLabel noise, it reaches $86.70\%$, compared with $58.44\%$ for DivideMix [2305.18377].

### Noise-adaptive multi-label learning

NAR assigns every multi-label entry one of three states:

- retain the annotation as supervised information;
- deactivate it and treat it as temporarily unlabeled;
- flip it when the model is highly confident that it is incorrect.

For an observed positive with low predicted probability, NAR can correct $1\rightarrow0$; for an observed negative with high predicted probability, it can correct $0\rightarrow1$. Intermediate-confidence entries are deactivated. Confidence-weighted BCE is combined with multi-label ELR. The method is particularly effective under subtractive and mixed remote-sensing noise, where deactivation supplies most of the recoverable improvement and selective flipping provides an additional gain [2601.08446].

### Feature-noise regularization

Feature noise leaves observed labels unchanged but perturbs training inputs:

$$
\widetilde{\mathbf{x}}=\mathbf{x}+\mathbf{z},
\qquad
\mathbf{z}\sim\mathcal{N}(\mathbf{0},\sigma^2 I).
$$

The proposed interpretation is information-theoretic: the Markov chain $X\rightarrow\widetilde{X}\rightarrow W$ limits the mutual information between model weights and original training features. This constrains memorization and tightens a PAC-Bayes generalization bound. The benefit is nonmonotonic: stronger feature noise can reduce memorization but eventually destroy class-relevant information. Gaussian and impulse noise were more effective than Gaussian blur and snow under severe label corruption [2308.01609].

### When correction is unnecessary

Under the posterior-drift framework, ordinary noisy-label ERM can be optimal when the noisy Bayes classifier agrees with the clean Bayes classifier and RSS remains sufficiently large. This motivates NI-ERM: empirical risk minimization that ignores the noise mechanism. A practical implementation extracts features without noisy labels and fits a linear classifier on noisy labels. DINOv2 features plus a linear classifier achieved $95.50\%$ accuracy on CIFAR-10 with $80\%$ symmetric noise and $98.69\%$, $98.80\%$, $98.65\%$, $98.67\%$, and $95.71\%$ on several CIFAR-10N settings [2411.00079].

## 5. Soft labels, annotator disagreement, and uncertainty

A noisy hard label is not necessarily devoid of information. If an observed label is sampled from the posterior class-membership distribution,

$$
Z'\mid Y=y\sim\operatorname{Multinomial}(1,\tau(y;\Psi)),
$$

then $(Y,Z')$ and the latent ground-truth pair have the same joint distribution. The probability that the sampled manual label differs from the latent class is

$$
P(Z'\neq Z\mid Y=y)
=
1-\sum_i \tau_i(y;\Psi)^2.
$$

Disagreement is therefore concentrated near posterior decision boundaries, where classification is intrinsically difficult. Multiple independent annotations retain information about uncertainty; collapsing them to a majority label discards vote proportions. A Dirichlet–multinomial random-effects model represents overdispersion among annotators while preserving the posterior mean. Under the stated model, logistic regression using multiple noisy annotations can have asymptotic relative efficiency

$$
\operatorname{ARE}
=
\frac{m(1+\alpha_0)}{m+\alpha_0}
\geq1
$$

relative to one ground-truth label, provided annotators are comparable, nonmalicious, conditionally independent, and their disagreement reflects posterior uncertainty rather than systematic bias [2104.02872].

SYNLABEL provides a controlled way to generate clean and soft labels from real-world feature distributions. A selected ground-truth function $f^G$ is evaluated after resampling hidden features, and the resulting class frequencies form a soft label distribution. The framework distinguishes Ground Truth, Partial Ground Truth, Observed Soft Label, and Observed Hard Label datasets. Feature Hiding produces uncertainty through marginalization over unavailable features, while NCAR, NAR, and NNAR transformations inject additional noise. Total variation distance quantifies changes between label distributions, and entropy quantifies uncertainty within one distribution [2309.04318].

The distinction between uncertainty and error is essential. Human disagreement may represent genuine ambiguity rather than an objectively incorrect label. Treating every disagreement as noise can remove informative boundary examples or collapse a meaningful label distribution. This issue is particularly visible in SNLI and TweetNLP, where additional crowd disagreement produces substantial performance degradation but no evaluated method consistently improves over vanilla BERT [2305.16337].

## 6. Domain-specific effects

### Multi-label remote sensing

Remote-sensing annotations often originate from thematic products, crowdsourcing, or automatic mappings. Their errors are incomplete and directional. Subtractive noise is damaging because it suppresses positive evidence; additive noise is often less harmful; mixed noise is the most challenging. Evaluation should report both micro-mAP and macro-mAP, since micro-mAP can conceal severe degradation on rare classes. ELR and SAT are useful baselines, while label-entry-level adaptive methods such as NAR are designed to suppress ambiguous entries and selectively correct high-confidence errors [2207.13975; 2601.08446].

### Graphs

Graph neural networks are affected by label noise through neighborhood aggregation. A corrupted node can alter the neighborhood label distribution used by nearby nodes. Raw graph topology is especially problematic under heterophily, where adjacent nodes often have different labels. Reconstructing a more homophilous graph before propagation can make label aggregation useful. LP4GLN iteratively reconstructs the graph, propagates labels, and transfers high-confidence corrections into a clean set; LENS screens nodes through leave-one-out neighborhood entropy and relabels detected nodes [2310.16560; 2610.22998].

### Text classification

BERT can be robust to randomly injected label noise, but feature-dependent weak-supervision errors and annotator disagreement substantially reduce performance. Consensus-based training with CETA improved Hausa and Yoruba results, while boosting and heterogeneous ensembles offered smaller gains. Loss-based cleaning was unreliable because high loss identifies difficult examples as well as mislabeled ones. This supports using detected errors for manual review rather than automatically deleting all high-loss examples [2305.16337].

### Scientific regression

For real-valued scientific targets, heteroscedastic Gaussian-process modeling can simultaneously screen corrupted observations and reduce their influence during regression. However, large learned variance indicates model inconsistency, not definitive objective corruption; poor kernels, sparse regions, or function misspecification can produce the same signal [2103.11352].

## 7. Evaluation, limitations, and research directions

Label-noise results are highly sensitive to the corruption mechanism. Uniform synthetic flipping is insufficient as a universal proxy for real annotation processes. Evaluation should distinguish class-conditional, instance-dependent, locally concentrated, open-set, adversarial, multi-label additive/subtractive, graph-structured, and annotator-disagreement noise. Clean test labels are essential because validation labels corrupted by the same mechanism can reward the wrong decision rule [2003.10471].

Several recurring limitations shape the field:

- **Identifiability**: instance-dependent transition matrices cannot generally be recovered from noisy observations without structural assumptions, trusted labels, anchor points, or representation constraints.
- **Representation dependence**: neighbor-based, mutual-information, and causal methods depend on features that preserve relevant semantics.
- **Small-loss failure**: adaptive or feature-dependent noise can make noisy examples easy to fit, invalidating early-learning and loss-separation assumptions.
- **Trusted-set dependence**: meta-learning methods require clean labels that are representative of deployment data.
- **Threshold and rate sensitivity**: filtering and bounded-loss methods require noise-rate estimates, confidence thresholds, or selection budgets.
- **Detection versus correction**: a strong detector does not guarantee improved downstream learning; deleting clean but difficult examples can be harmful.
- **Ambiguity versus error**: disagreement may encode genuine uncertainty and should sometimes be preserved as a soft label.
- **Computational cost**: bilevel differentiation, multiple networks, Gaussian-process inference, graph reconstruction, and temporal logit storage can substantially exceed ordinary training cost.

Benchmark construction is therefore a substantive research problem. Fixed-budget detection benchmarks evaluate residual noise through FNR after perfect relabeling, separating detector quality from relabeling quality [2510.16211]. Controlled frameworks such as SYNLABEL provide clean references, soft labels, adjustable task complexity, and measurable transformations. Robust training should report not only clean accuracy but also detection error, noise recall, F1, macro-F1, calibration, class-specific performance, and computational cost.

The contemporary view is that label noise is not a single nuisance parameter. Its effect is determined by the interaction among the label-generation mechanism, feature geometry, model capacity, optimization dynamics, annotation uncertainty, and evaluation protocol. In favorable cases, noisy labels preserve the Bayes decision and ordinary ERM is close to optimal. In unfavorable cases, errors align with decision boundaries, exploit model-specific weaknesses, corrupt graph neighborhoods, remove sparse positive evidence, or defeat loss-based detection. Effective systems therefore combine mechanisms appropriate to the regime: trusted-set meta-learning, soft-label supervision, feature-aware transition estimation, causal representation learning, entropy- or graph-based detection, selective entry-level correction, semi-supervised training, or information-limiting regularization.

Source: https://www.emergentmind.com/topics/label-noise