Label Noise Gradient Descent

• Label Noise Gradient Descent is a suite of optimization techniques designed to mitigate the impact of noisy labels in deep neural network training.
• It employs methodological modifications such as limited gradient descent, stochastic label flipping, and implicit regularization to favor flatter minima and reduce overfitting.
• Empirical benchmarks on datasets like CIFAR-10 and Clothing-1M demonstrate that label noise GD improves test accuracy and controls noise memorization compared to standard GD.

Label noise gradient descent (GD) refers to a suite of optimization methodologies and theoretical analyses targeting the challenge of learning from data with noisy or corrupted labels, primarily in deep neural networks. Key advances in this domain include explicit algorithms for robust stopping criteria, implicit regularization analyses, and empirical demonstrations of noise-robust generalization. Techniques within this area not only reinterpret standard practices such as early stopping, but also introduce principled procedures (e.g., injected label noise) that can systematically enhance generalization—particularly in overparameterized or low signal-to-noise ratio (SNR) regimes.

1. Methodological Modifications for Label Noise

Multiple approaches operationalize label noise gradient descent by modifying the gradient computation, learning dynamics, or stopping rules:

• Limited Gradient Descent (LGD) modifies standard stochastic gradient descent (SGD) by introducing a small controlled subset of “reverse-labeled” samples alongside the noisy-labeled main set. At each epoch, accuracies are tracked on both unchanged (“leftover”) and reverse-labeled partitions. The stopping criterion is chosen to maximize the “leftover-over-reverse” (LoR) ratio, obviating the need for a clean validation set (Sun et al., 2018).
• Label Noise Injection employs stochastic label flipping at each epoch, creating a sequence of randomized optimization problems. In one implementation for two-layer networks, for each training sample the label is flipped with probability $p$ and the loss is evaluated accordingly. Standard gradient descent updates are applied on this randomly perturbed loss, resulting in an algorithm termed “Label Noise GD” (Huang et al., 20 Oct 2025).
• Implicit Regularization Analyses interpret noisy label GD (and its variants such as SGD with label noise or stochastic label flipping) as optimizing a regularized version of the original loss. That is, SGD with label noise is shown to be equivalent (in a certain regime) to minimizing $L(\theta) + \lambda R(\theta)$ for a data-dependent regularizer $R(\theta)$, where $\lambda$ depends on learning rate, noise variance, and batch size. This regularizer penalizes sharp minima and favors flatter ones (Damian et al., 2021).
• Early Stopping and Stability Heuristics are theoretically justified in overparameterized regimes. Empirically, the initial phase of training fits clean data while noisy labels are only memorized if training proceeds for too long or if the trajectory strays too far from initialization. Therefore, classic early stopping, distance from initialization regularization, and “oracle” control via minimal norm interpolants are all validated as noise-robust variants of GD (Li et al., 2019, Richards et al., 2021).

2. Theoretical Foundations and Generalization Properties

Label noise GD mechanisms are underpinned by several theoretical results:

• Implicit Regularization and Flatness: In depth, SGD with label noise converges (with polynomial rate) to stationary points of a regularized loss whose regularizer, $R(\theta)$, penalizes the trace and (for larger learning rates) the spectral norm of the Hessian, thereby biasing optimization toward flatter minima (Damian et al., 2021).
• Wasserstein Contraction and Generalization Bounds: For non-convex losses satisfying uniform dissipativity and smoothness, explicit bounds are established on the generalization error in terms of a polynomial of parameter dimension $d$, achieving $O(n^{-2/3})$ scaling with sample size—outperforming the $O(n^{-1/2})$ rate of SGLD (Huh et al., 2023). This analysis links algorithmic stability to exponential contractions in Wasserstein semimetrics.
• Degenerate Dynamics and Implicit Bias: In the singular-limit analysis of noisy GD, label noise (unlike Dropout-type noise) yields a degenerate quadratic perturbation of the loss, causing the post-zero-loss manifold dynamics to become deterministic constrained gradient flow along a regularizer $Reg(w)$ that measures solution flatness or curvature (Shalova et al., 18 Apr 2024). Label noise thus regularizes by deterministic, rather than stochastic, evolution, selectively favoring flatter minima within the global minimizer set.
• Benign Overfitting in Low SNR Regimes: In regimes where the input is dominated by non-signal or adversarial noise, standard GD overfits the noise patterns, resulting in non-vanishing test error. In contrast, Label Noise GD suppresses growth of “noise memorization” coefficients, permitting rapid increase of signal-aligned components while maintaining controlled generalization error. As a consequence, population zero-one loss is exponentially small in the problem dimensions, while standard GD retains high test error even at vanishing training loss (Huang et al., 20 Oct 2025).
• Algorithmic Stability and Excess Risk: Explicitly, on-average stability analyses show that for shallow networks, the generalization and excess risk of GD (with or without explicit kernelization) are controlled by the relative norm path length between the iterates and initialization. Early stopping ensures that risk converges to the noise level, and practical bounds are derived in terms of oracle minimization combining empirical risk and squared deviation from initialization (Richards et al., 2021).

3. Practical Algorithms and Evaluation

Practical label noise GD algorithms are implemented with careful attention to the proportion and structure of label corruption:

• LDG Reverse-Sample Construction: Reverse samples are created by randomly selecting a small fraction $\beta$ of samples and shifting their labels cyclically. Theoretical analysis bounds appropriate choices of $\beta$ for both symmetric and asymmetric label noise (e.g., $\beta < 0.1$ ensures the reverse pattern is sufficiently distinct to track memorization) (Sun et al., 2018).
• Empirical Benchmarks: On standard vision benchmarks such as CIFAR-10, CIFAR-100, and noisy real-world data (Clothing-1M), LGD with the LoR-based stopping criterion achieves test accuracy competitive with or superior to classical early stopping using large clean validation sets. Notably, LGD reduces outcome variance and reliably tracks the transition point before noise memorization. On Clothing-1M, LGD surpasses validation-dependent baselines by up to 2.45% (Sun et al., 2018).
• Low SNR Performance: Label Noise GD consistently outperforms standard GD in low SNR settings, both in synthetic controls and on image benchmarks with adversarially modulated noise. Standard GD achieves low training loss but generalizes poorly as noise increases, while Label Noise GD maintains high test accuracy and bounded noise memorization (Huang et al., 20 Oct 2025).
• Implementation Simplicity and Efficiency: Label Noise GD introduces minimal overhead, requiring only stochastic label flipping during loss evaluation. The method is compatible with standard architectures and can be combined with various loss functions or robust training heuristics.

4. Geometry, Dynamics, and Implicit Regularization

Analyses of label noise GD reveal deep connections between injected noise covariance and optimization geometry:

• Noise Alignment Metrics: The covariance of the stochastic gradient noise caused by label corruption is exactly aligned with the local empirical Fisher (curvature), i.e., $$\Sigma_{\text{label}}(\theta) = \varepsilon^2 G(\theta)$$. Thus, label noise induces dynamics that “see” the geometry of the loss landscape, ensuring energy in random directions is injected proportionally to curvature (Wang et al., 2023). In practice, this alignment means SGD with label noise escapes sharp minima chiefly by accumulating energy in flatter directions, consistent with the robust generalization associated with flat minima.
• Time Scale of Regularization Effects: The structure of label noise is degenerate in the quadratic sense, so once optimization enters the zero-loss manifold, noise-induced stochasticity rapidly “averages out,” and only deterministic constrained gradient flows governed by the induced regularizer persist. The deterministic flow selects among the global minimizer set according to a regularity or flatness preference (Shalova et al., 18 Apr 2024).
• Comparisons to Other Noise Types: Unlike minibatch SGD noise or Dropout, which can yield persistent stochastic evolution along the zero-loss manifold, label noise forces dynamics at a slower timescale (by $1/(a^2\sigma^2)$ rather than $1/(a\sigma^2)$) and ensures regularization operates solely through deterministic bias toward flatness. Classical minibatch SGD can even “freeze” once the solution reaches sharp minima, while label noise continues to bias toward flatter optima.

5. Extensions, Generalizations, and Open Directions

The paper of label noise gradient descent encompasses several extensions and outstanding questions:

• Complex Noise Models: While LGD and related methods are proven robust to symmetric and asymmetric label noise, structured or open-set noise remains an open challenge. Development of adaptive strategies for reverse-sample selection and integration into more complex architectures are suggested directions (Sun et al., 2018).
• Algorithmic Generalization Theory: Advances in stability-based analysis, contraction bounds in Wasserstein distances, and oracle inequalities yield generalization guarantees with polynomial dependence on model dimension—even in non-convex settings (Huh et al., 2023).
• Loss Functions and Momentum: Convergence analyses for label noise SGD extend to general loss functions (e.g., logistic, exponential) and SGD with momentum or anisotropic label noise. In these cases, flatness-seeking regularization persists, but its precise form and convergence dynamics require further characterization (Damian et al., 2021).
• Test-Time Adaptation and Pseudo-Labels: Related analyses in test-time adaptation demonstrate that use of “conjugate” pseudo-labels (which can be viewed as a type of controlled label noise) enables gradient descent to robustly adapt to distribution shift, outperforming “hard” label schemes (Wang et al., 2022).
• Benign Overfitting and Practical Adoption: The generality, computational efficiency, and direct effectiveness of Label Noise GD position it as an attractive candidate for domains with limited clean validation data or where SNR is intrinsically low, such as large-scale web annotation or biomedical imaging (Huang et al., 20 Oct 2025).

6. Implications for Deep Learning Practice

Label noise gradient descent fundamentally transforms the training dynamics of deep networks in the presence of label corruption:

• By exploiting the early learning dynamics of deep networks, label noise GD enables training algorithms to emphasize population-level patterns and suppress the memorization of spurious noise, leading to improved generalization even in highly overparameterized settings.
• The methodology provides a rigorous framework for understanding and exploiting implicit regularization, offering algorithmic simplicity (label flipping) with provable guarantees and strong empirical results on diverse benchmarks.
• The consistency of findings across theoretical analysis, numerical experiments, and practical regimes supports systematic adoption of label noise GD as a robust regularization scheme in modern supervised learning.

Algorithm/Approach	Core Idea	Key Insight/Result
Limited Gradient Descent	Reverse-labeled subset + LoR	Removes need for clean validation set for robust early stopping
Label Noise GD	Stochastic label flipping	Suppresses noise memorization, improves generalization at low SNR
SGD with Label Noise	Noise in gradient updates	Implicit regularizer favors flat/hessian-low minima
Early Stopping in GD	Halt before fitted noise grows	Maintains signal generalization, supported by parameter-distance bounds

These interdisciplinary advances establish label noise gradient descent as a theoretically sound, empirically validated, and practically deployable framework for noise-robust learning in deep neural networks.