Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

Abstract: Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.

• The paper's primary contribution is LPG, a unified class-adaptive gradient perturbation method that outperforms existing approaches in challenging regimes.
• The methodology decomposes training into forward and backward phases, enabling targeted gradient adjustments based on class-specific statistics.
• Empirical results show significant gains in long-tail classification and noise robustness, with notable improvements on CIFAR benchmarks over baseline methods.

Gradient Perturbation: A Unified and Adaptive Framework for Optimizing Deep Neural Networks

Introduction

The paper "Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training" (2605.29494) presents a systematic investigation into the role of perturbations applied during backpropagation in deep neural network (DNN) optimization. By unifying several well-known gradient manipulation methods under a single framework, the authors demonstrate that canonical approaches such as SAM, gradient clipping, and gradient noise injection can be recast as particular cases of class-level gradient perturbation. The central contribution is LPG (Learning to Perturb Gradients): an efficient, class-wise, low-dimensional adaptive gradient perturbation method operating in the logit-gradient space—designed to yield improved generalization and robustness characteristics, especially in challenging regimes such as class imbalance and label noise.

Unified Gradient Perturbation Framework

A key insight of this work is the decomposition of DNN training into forward and backward computational chains, and the realization that systematic, category-aware perturbation of the backward chain (gradients) has been relatively neglected. The authors formally connect various established strategies to the general paradigm:

• SAM introduces a curvature-dependent perturbation by evaluating gradients at adversarial parameter locations.
• Gradient clipping adaptively shrinks large gradients, preferentially affecting head classes in long-tail distributions.
• Gradient noise imparts isotropic, non-adaptive stochasticity to all class gradients.

Each of these replaces the native gradient $g$ with a perturbed variant $\tilde{g} = g + \delta_g$. The main deficiency identified is the lack of adaptability to per-class properties, motivating the development of LPG—which flexibly perturbs gradients according to class accuracy, sample size, or intra-class gradient variance.

Figure 1: Relative gradient variation $\|\tilde{g}_c - \bar{g}_c\|/\|\bar{g}_c\|$—LPG adaptively modulates gradients for classes based on their learning needs, unlike existing methods which are agnostic or only weakly class-selective.

The LPG Method: Category-Aware Gradient Perturbation

LPG operates by formulating the backward pass at the level of logit gradients, reducing the perturbation space from the parameter dimension $|W|$ to the manageable size $C$ (number of classes). The update mechanism involves adaptively amplifying or dampening the gradient norms for specific classes, based on scenario-driven criteria:

• In long-tail classification, tail classes are boosted (positive augmentation), head classes are suppressed (negative augmentation).
• In noisy label learning, high gradient variance classes are dampened to mitigate noise amplification.
• In balanced settings, gradient modifications are based on running class-wise accuracy.

The authors formalize closed-form and PGD-based solutions for optimal class-wise perturbation, with the closed-form solution scaling logit gradients and PGD providing directionally optimal adjustments under norm constraints.

Theoretical Analysis

A crucial theoretical result is the demonstration of forward-backward duality between logit perturbation (LPL) and LPG. Logit perturbations in the forward pass induce specific gradient perturbations in the backward pass via the chain rule, but LPG subsumes LPL, offering strictly greater expressive power by allowing arbitrary perturbation directions in logit-gradient space, unconstrained by the Hessian structure of the loss.

Furthermore, a PAC-Bayesian generalization bound is established, linking the class-wise perturbation budget to probabilistic generalization guarantees. This justifies the design of LPG's per-class perturbation magnitudes and reinforces the intuitive regularization provided by gradient perturbation.

Empirical Results

The experimental suite comprises balanced classification, long-tailed recognition, and noisy label learning, with LPG consistently outperforming canonical baselines and recent forward-chain perturbation methods (e.g., LPL).

Figure 2: Long-tail classification accuracy on CIFAR-10-LT and CIFAR-100-LT. LPG sharply improves tail-class performance without significant loss in head-class accuracy.

In long-tail classification tasks, LPG yields pronounced gains on tail classes, with an especially marked difference on CIFAR-100-LT, where the overall improvement over LPL and SAM is in excess of 3–6%.

Figure 3: Test accuracy under symmetric label noise. LPG demonstrates superior robustness by effectively suppressing the gradients arising from noisy, high-variance classes.

For label noise benchmarks, LPG's gradient dampening mechanism delivers distinct improvements over both vanilla and sophisticated robust training algorithms, demonstrating a substantial margin at extreme noise rates (e.g., 80% label noise on CIFAR-10).

Figure 4: (a) LPG as a plug-in module for various methods yields deterministic error reduction; (b) LPG and LPL are complementary, with additive gains evident when combined.

The complementarity of LPG with forward perturbation strategies (LPL, Logit Adjustment, SAM) is empirically validated. Joint application leads to additive improvements, supporting the proposition that LPG alters the optimization geometry along axes orthogonal to logit-level control.

Representation Analysis and Hyperparameter Robustness

Feature visualizations via t-SNE indicate LPG's effect on embedding geometry: classes, especially minority tails, develop tighter intra-class cohesion and greater separation. Ablation studies on hyperparameters (thresholds, perturbation magnitude, PGD steps) reveal minimal sensitivity, underscoring LPG's practical deployability.

Figure 5: t-SNE plots of learned features—LPG achieves denser intra-class clusters and increased inter-class margin, particularly benefitting tail-class structure.

Figure 6: LPG demonstrates stability across a range of hyperparameter choices, indicating strong practical robustness.

Implications and Future Directions

The unified framework for backward perturbation provides practitioners with a rigorous toolkit for category-adaptive control during training. LPG's modularity—as a plug-in to virtually any standard or enhanced optimizer—enables application in both traditional and challenging data regimes. The theoretical insights regarding forward-backward duality highlight the non-redundancy of perturbation at different stages of computation.

Potential future extensions include:

• Instance-level gradient perturbation, which could further improve adaptability.
• Integration with curriculum learning for dynamic perturbation scheduling.
• Transfer to domains beyond vision, such as sequence modeling, where class or token-level gradient modulation could mitigate bias and noise.

Conclusion

This work establishes explicit links between a spectrum of gradient manipulation techniques, analyzing them through the lens of class-wise adaptation. LPG provides both a generalization-theoretic and empirical basis for backward perturbation as a primary axis for model improvement, synthesizing and extending previous approaches. Its efficient, expressivity-unconstrained, and complementary design positions LPG as a robust method for optimizing neural network training dynamics under a variety of adverse dataset conditions.