Layerwise Learning Rate (LLR)

• Layerwise Learning Rate (LLR) is a strategy that assigns unique learning rates to each neural network layer, addressing differences in curvature, noise, and dynamics.
• It employs methods such as discriminative decay, cosine-annealed schedules, and noise-adaptive scaling to tailor optimization to each layer's characteristics.
• LLR enhances training efficiency and generalization, with empirical results showing faster convergence and improved performance in large-scale models.

Layerwise Learning Rate (LLR) refers to any strategy in which distinct neural network layers (or structured groups of parameters) are assigned their own independent learning rates during optimization. In deep learning, the conventional practice of using a single global learning rate has been shown to be suboptimal due to pronounced heterogeneity in curvature, noise, functional impact, and training dynamics across layers. Modern LLR methods systematically address these mismatches, enabling faster convergence, improved generalization, and more robust scaling in very deep architectures, transformers, and transfer learning scenarios (Zeng et al., 15 May 2026, He et al., 21 May 2026, Ludziejewski et al., 4 Jul 2025, Hao et al., 15 Oct 2025, Kokane et al., 2024, Yao et al., 30 Apr 2026, Milsom et al., 24 Feb 2025, Ro et al., 2020, Zhang et al., 2018).

1. Theoretical Foundations and Motivations

LLR emerged from the recognition that the role and statistical landscape of deep neural layers differ dramatically across architectures and training regimes. In transfer learning, lower-level (input-proximal) layers are generally responsible for extracting universal, task-agnostic features and should be updated conservatively, while higher-level (output-proximal) layers respond to task-specific information and require accelerated adaptation (Ro et al., 2020, Yao et al., 30 Apr 2026). In transformer-based LLMs, empirical spectral density (ESD) analysis and heavy-tailed self-regularization (HT-SR) theory reveal substantial heterogeneity in layerwise training signal, calling for targeted step-size adaptation (He et al., 21 May 2026).

LLR can enhance both optimization and generalization by resolving layerwise disparities in curvature, gradient noise, and functional impact, as established across bilevel optimization theory (Stackelberg games) (Zeng et al., 15 May 2026), geometry-aware algorithms (Hao et al., 15 Oct 2025), and function-space learning rate matching (Milsom et al., 24 Feb 2025).

2. Principal Methodologies and Update Schemes

LLR encompasses a broad range of algorithmic approaches:

• Discriminative Layer Decay: Exponential decay in learning rate with respect to depth, as in ULMFiT or DALS (“Discriminative Adaptive Layer Scaling”), used extensively for transfer learning and fine-tuning (Yao et al., 30 Apr 2026).
• Relative and Cosine-Annealed Schedules: Assigns per-layer or per-module multipliers to base learning rate schedules (cosine decay, etc.), with parameters tuned on proxy models and transferred via invariant ratios (Ludziejewski et al., 4 Jul 2025).
• Heavy-Tailed Self-Regularization (HT-SR)–Guided LLR: Dynamically sets learning rates as a function of the estimated heavy-tailedness (power-law exponent α) of each layer’s weight spectrum. Layers with weaker heavy-tailedness (higher α) get larger learning rates, and vice versa (He et al., 21 May 2026).
• Back-Matching Propagation Approximations: Reframes the gradient update per layer to match the desired change in output in a least-squares sense, resulting in a local rescaling of the effective learning rate (Zhang et al., 2018).
• Noise- or Curvature-Adaptive Rates: Estimation of local gradient noise (in the dual geometry-aware norm of the layer) to tune layerwise stepsizes in response to dynamic sharpness/stochasticity (Hao et al., 15 Oct 2025).
• Function-space Learning Rates: Measures the RMS change in network output induced by each layer’s parameter update, setting per-layer η to match desired functional step sizes across model scales (FLeRM) (Milsom et al., 24 Feb 2025).
• Distillation-driven LLR: Assigns layerwise rates as an inverse function of the divergence between student and teacher Jacobian/attention/Hessian mappings (Kokane et al., 2024).
• Auto-tuned or Monotonically Sorted Schemes: Automatically tunes per-layer rates to enforce monotonic weight variation from lower to higher layers, in harmony with feature specificity (Ro et al., 2020).

3. Mathematical Formulations and Algorithmic Implementations

Key mathematical schemes underlying LLR include:

• Stackelberg-based Two-Time-Scale Updates:

$$\theta_B^{t+1} = \theta_B^t - \alpha_B \nabla_{\theta_B}L(\theta_B^t, \theta_F^t),\quad \theta_F^{t+1} = \theta_F^t - \alpha_F \nabla_{\theta_F}L(\theta_B^{t+1}, \theta_F^t)$$

$\alpha_F \gg \alpha_B$ to align with the bilevel optimality structure (Zeng et al., 15 May 2026).

• Heavy-Tail-Guided Multipliers:

$$f_t(l) = \eta\cdot \left[1 + \frac{(\alpha_l - \alpha_{\min})(s-1)}{\alpha_{\max} - \alpha_{\min}}\right]$$

where $\alpha_l$ is the layer’s current PL exponent and $s$ is a scaling factor (He et al., 21 May 2026).

• Gradient Noise–Adaptive Scaling (LANTON):

$$H_t^{\ell} = \beta_2 H_{t-1}^{\ell} + (1-\beta_2) \|G_t^{\ell} - G_{t-1}^{\ell}\|_{(*)}^2$$

$$\eta_t^{\ell} = \eta_t \cdot \sqrt{\frac{\alpha_t^{\ell}}{\alpha_t^m}}$$

(Hao et al., 15 Oct 2025).

• Function-space LLR Estimation:

$$\text{LLR}_\ell = \frac{\|\Delta_\ell \mathbf{f}\|_\mathrm{RMS}}{\|\Delta \theta_\ell\|}$$

FLeRM sets $$\eta_\ell = \eta_0 \frac{\|\Delta_\ell \mathbf{f}\|_\text{base}}{\|\Delta_\ell \mathbf{f}\|}$$ (Milsom et al., 24 Feb 2025).

• Distillation Loss-Driven Rate Adaptation (JSD and momentum filtering):

$$\eta_j^{(t)} = \gamma\eta_j^{(t-1)} + (1-\gamma)\mathrm{JSD}_j^{(t)}$$

$\alpha_F \gg \alpha_B$0

(Kokane et al., 2024).

Tables summarizing representative approaches:

Reference	LLR Scheme	Control Signal
(He et al., 21 May 2026)	HT-SR Power Law	ESD $\alpha_F \gg \alpha_B$1 Hill estimator
(Hao et al., 15 Oct 2025)	LANTON Noise-Adapt	Dual-norm stochastic gradient
(Milsom et al., 24 Feb 2025)	Function-space (FLeRM)	RMS $\alpha_F \gg \alpha_B$2 per-layer
(Yao et al., 30 Apr 2026)	DALS Discriminative Decay	Depth, phase, trust ratio
(Ludziejewski et al., 4 Jul 2025)	Relative Cosine Schedules	Module/group LR multipliers

4. Empirical Performance and Comparative Analyses

Across a diverse suite of models and learning regimes, LLR methods have demonstrated substantial improvements in sample efficiency, downstream accuracy, and stability over conventional global or parameter-adaptive schedules:

• Heavy-Tailed LLR: Achieved up to 1.5× convergence speedup and 2% absolute gain in zero-shot accuracy (47.09% $\alpha_F \gg \alpha_B$3 49.02% for LLaMA-1B) with minimal extra tuning overhead (He et al., 21 May 2026).
• RLRS (Relative Learning Rate Schedules): Produced 10–23% faster convergence, enabling stable hyperparameter transfer across $\alpha_F \gg \alpha_B$4 model scale-up without retuning module ratios (Ludziejewski et al., 4 Jul 2025).
• Noise-adaptive LANTON: Lowered wall-clock/convergence time and outperformed state-of-the-art geometry-aware optimizers on both GPT and LLaMA variants, retaining robustness to global $\alpha_F \gg \alpha_B$5 hyperparameter (Hao et al., 15 Oct 2025).
• AutoLR: Monotonic per-layer adaptation provided Recall@1 increases of $\alpha_F \gg \alpha_B$6 (CUB-200), $\alpha_F \gg \alpha_B$7 (Cars-196), and led benchmark performance in metric learning (Ro et al., 2020).
• DALS: Demonstrated no single LLR regime excels everywhere; DALS synthesizes phase-adaptive, depth-aware, and trust-ratio normalization to offer regime-robust performance (Yao et al., 30 Apr 2026).
• FLeRM: Maintained optimal functional update magnitudes across scale changes, aligning train-loss vs. LR curves for width, depth, and initialization scaling (Milsom et al., 24 Feb 2025).

5. Best Practices and Practical Guidance

Implementation of LLR requires addressing layer grouping, update granularity, signal estimation, and schedule synchronization:

• Layer/group selection: For transformers and MoE architectures, stratify into semantically meaningful blocks (Embedding, Attention, FFN, Experts), tuning per-group multipliers (Ludziejewski et al., 4 Jul 2025).
• Update timing: For spectral or functional estimators, perform updates only during the first 20% of training ("active phase") and employ smoothing (soft-switch) to avoid instability (He et al., 21 May 2026).
• Tuning protocol: Tune base/global hyperparameters (e.g., $\alpha_F \gg \alpha_B$8, final fraction $\alpha_F \gg \alpha_B$9) and transfer multipliers (e.g., $$f_t(l) = \eta\cdot \left[1 + \frac{(\alpha_l - \alpha_{\min})(s-1)}{\alpha_{\max} - \alpha_{\min}}\right]$$0, $$f_t(l) = \eta\cdot \left[1 + \frac{(\alpha_l - \alpha_{\min})(s-1)}{\alpha_{\max} - \alpha_{\min}}\right]$$1) from proxy runs (Ludziejewski et al., 4 Jul 2025).
• Noise/spectral estimation: Use exponential moving averages for stochastic quantities (variance, JSD, spectral density) and minimize overhead via random projections or partial SVD (Hao et al., 15 Oct 2025, He et al., 21 May 2026).
• Functional matching: For architecture scaling, match function-space step sizes to those of reference models, correcting for scaling perturbations (Milsom et al., 24 Feb 2025).
• Regularization: Clamp or bound multipliers/ratios to avoid degenerate or exploding rates, particularly with LARS/LAMB/Trust Ratio–style normalizations (Yao et al., 30 Apr 2026).
• Transfer learning: Monotonically sorted weight variation from bottom to top layers preserves general features and accelerates task adaptation (Ro et al., 2020).

6. Taxonomy and Evolution of LLR in Optimization Theory

LLR sits within a five-generation systematization of learning rate engineering:

1. Gen1: Global fixed LR (η constant for all weights)
2. Gen2: Global LR scheduling (e.g., step, cosine annealing)
3. Gen3: Parameter-level adaptation (e.g., AdaGrad, Adam)
4. Gen4: Layer-level differentiation (discriminative decay, LARS, LAMB, AutoLR)
5. Gen5: Joint layer-by-time scheduling (DALS, STLR+Discriminative, RLRS)

A key insight is the regime dependence: discriminative decay-only (Gen4) is beneficial in fine-tuning but detrimental in from-scratch tasks due to under-updating of lower layers. Layerwise learning rates must therefore be modular, dynamically responsive, and sensitive to phase/state, as codified in Gen5 frameworks (Yao et al., 30 Apr 2026).

7. Convergence Guarantees and Theoretical Advances

Recent developments anchor LLR in rigorous optimization theory. Stackelberg and bilevel formulations yield provable improvement in strong convexity and convergence rates when applying non-uniform LRs: global stationarity is reached at $$f_t(l) = \eta\cdot \left[1 + \frac{(\alpha_l - \alpha_{\min})(s-1)}{\alpha_{\max} - \alpha_{\min}}\right]$$2 under strong-convexity reduction, exceeding the $$f_t(l) = \eta\cdot \left[1 + \frac{(\alpha_l - \alpha_{\min})(s-1)}{\alpha_{\max} - \alpha_{\min}}\right]$$3 of single-rate SGD (Zeng et al., 15 May 2026). Noise-adaptive layerwise scaling in geometry-aware settings sharpens convergence guarantees, especially in the presence of heterogeneous, layer-specific stochasticity (Hao et al., 15 Oct 2025). The alignment of function-space updates via FLeRM further supports scale-invariant transfer with no additional per-layer parameterization (Milsom et al., 24 Feb 2025).

---

LLR now constitutes a core paradigm in deep learning optimization, unifying diverse techniques under a common goal: harmonizing per-layer learning dynamics with the network’s geometry, data-induced noise, and task-specific adaptation requirements for maximal efficiency and robustness across architectures and problem domains.