Learning Rate Curriculum (LeRaC) Explained

• The paper introduces two distinct curriculum frameworks—model-level modulation of per-layer learning rates and task-level mastering-rate scheduling—to enhance training efficiency.
• It demonstrates robust improvements such as faster convergence, accuracy gains of up to 1.5%, and significant sample reductions in multi-task and reinforcement learning settings.
• Practical guidelines detail hyperparameter calibration and implementation strategies using frameworks like PyTorch, highlighting its data- and architecture-agnostic benefits.

Learning Rate Curriculum (LeRaC) designates a class of curriculum learning algorithms that organize either (i) the sequence of tasks or (ii) the schedule of model parameter updates so as to facilitate more efficient and robust training of deep neural networks. Under the “LeRaC” designation, two fundamentally distinct frameworks appear in the research literature: (1) the model-level, data-agnostic “Learning Rate Curriculum” of Croitoru et al., and (2) the task-level, sample-efficient “Mastering Rate” paradigm of Willems et al., which is also referred to as a learning-rate or mastering-rate curriculum. Both frameworks depart from the classic sample-difficulty-sorting approach but differ substantially in their mechanisms, mathematical formalism, and domains of application.

1. Model-Level Learning Rate Curriculum: Principles and Formalism

The model-level Learning Rate Curriculum (LeRaC) defines a curriculum on the neural network itself by modulating the learning rate assigned to each layer as a function of both depth and training time (Croitoru et al., 2022). Early (“shallower”) layers receive the largest learning rate at the onset, with learning rates for deeper layers initialized at much smaller values (typically 5–6 orders of magnitude below the base rate). During the initial $k$ epochs, all layer-wise learning rates are monotonically increased—on a log-exponential schedule—until they converge to a single value $\eta_{\mathrm{base}}$ used for the remainder of training.

Let $f$ be a $n$-layer network with parameters $\theta_1, \ldots, \theta_n$. For layer $j$, at iteration $t$ of the first $k$ curriculum epochs, the learning rate is

$$\eta_j^{(t)} = 10^{ \left( (1-\frac{t}{k}) \log_{10} \eta_j^{(0)} + \frac{t}{k} \log_{10} \eta_{\mathrm{base}} \right) },$$

where $\eta_1^{(0)} = \eta_{\mathrm{base}}$ (earliest layer) and $\eta_{\mathrm{base}}$0 (deepest layer). After epoch $\eta_{\mathrm{base}}$1, all $\eta_{\mathrm{base}}$2 are set to $\eta_{\mathrm{base}}$3, and standard optimization resumes.

The rationale is that early layers rapidly learn low-level features with high learning rates, while deeper layers are protected from high-variance gradient updates until lower-level representations are sufficiently “denoised.” This staged approach is both architecture- and data-agnostic, requiring neither problem-specific difficulty measures nor data annotation (Croitoru et al., 2022).

2. Task-Level Mastering-Rate Curriculum: Motivation and Algorithm

In contrast, the Mastering Rate (MR) curriculum is structured at the task or environment level and is particularly suited to scenarios involving multiple discrete tasks, such as reinforcement learning curricula or structured multi-task supervised learning (Willems et al., 2020). Here, one begins with a set $\eta_{\mathrm{base}}$4 of tasks, each equipped with user-supplied rough estimates $\eta_{\mathrm{base}}$5 and $\eta_{\mathrm{base}}$6. In addition, a partial order (DAG) for task prerequisites is typically available.

For each task $\eta_{\mathrm{base}}$7, the mastering rate at training step $\eta_{\mathrm{base}}$8 is: $\eta_{\mathrm{base}}$9 where $f$0 is the running mean return over the last $f$1 task samples. Tasks are sampled for training at each step via an “attention” mechanism that composes ancestor and successor mastering rates, the current mastering rate, and (optionally) learning progress, with redistribution to ensure exploration of easier predecessors.

A key innovation is that tasks are only prioritized for sampling if (i) all prerequisites are adequately mastered, (ii) the task is not already mastered, and (iii) potential for learning progress remains. This circumvents the failure modes of pure learning-progress curricula (which may oversample already mastered or intractable tasks) (Willems et al., 2020).

3. Algorithmic Procedures

3.1 Model-Level LeRaC (Layer-wise Curriculum)

The following sequence summarizes the LeRaC training protocol (Croitoru et al., 2022):

• Assign per-layer learning rates: $f$2, $f$3 to $f$4, monotonic in depth.
• For each of the first $f$5 epochs:
  • At epoch $f$6, set for each layer $f$7:

  $f$8

• For $f$9: set all $n$0.

• Update model parameters per standard optimizer rules with these learning rates.

• Resume conventional learning-rate scheduling (cosine decay, step, etc.) after $n$1 epochs.

3.2 Task-Level Mastering-Rate Curriculum

The MR curriculum learning loop (Willems et al., 2020):

• For each task $n$2, maintain:

  • Running window ($n$3 samples) mean return $n$4, minimum $n$5, and maximum $n$6.
  • Mastering rate $n$7.
  • Ancestor and successor mastering rates according to task DAG.
• At each step:
  • Compute attention scores:

  $n$8 - Redistribute attention to ancestors ($n$9) and successors ($\theta_1, \ldots, \theta_n$0). - Normalize and sample task $\theta_1, \ldots, \theta_n$1 according to the resulting distribution. - Train on the sampled task/environment and update statistics.

4. Theoretical Intuition and Empirical Dynamics

The model-level LeRaC framework draws on an analysis of the propagation of signal-to-noise ratio (SNR) in deep architectures, showing that early overfitting of noisy activations in deep layers can be avoided by delaying aggressive updates. The learning-rate schedule ensures low-level features mature in a “clean” environment, after which deeper abstractions can stably emerge (Croitoru et al., 2022).

Empirical observations confirm that LeRaC moderates entropy loss in deeper feature maps during early phases and aligns the pace of representation learning across network depth. This pacing effect is not observed for anti-curriculum schedules or standard single-LR warm-up strategies.

In task-level curricula, the MR algorithm eliminates exploration inefficiencies common to learning-progress-based “teacher-student” curricula—such as over-committing to already-mastered or unlearnable tasks. By masking attention to tasks whose prerequisites are unsatisfied or already mastered, sampling is tightly coupled to actionable skill acquisition. Comparative evaluations show substantial reductions in sample requirements on multi-stage supervised and RL curricula (Willems et al., 2020).

5. Experimental Benchmarks and Comparative Performance

Experiments on diverse classification, detection, and regression benchmarks (vision, language, audio domains), as well as multi-task and RL settings, demonstrate:

Curriculum	Domain	Main Gains over Baseline
LeRaC	All (CNN/RNN/Transf.)	+0.3–1.5% accuracy, +0.2–1.4 mAP, faster convergence, zero overhead (Croitoru et al., 2022)
MR (task-level)	Multi-task, RL	2–3× reduction in samples to mastery; full convergence where progress-based methods can stall (Willems et al., 2020)

In direct comparison with Curriculum by Smoothing (CBS), LeRaC achieves robust improvements (+0.3–1.5% accuracy; lower mean absolute error), and unlike CBS, imposes no runtime or storage overhead. For the MR algorithm, in RL KeyCorridor benchmarks, learning-progress curricula failed to converge within $\theta_1, \ldots, \theta_n$2 frames, while MR reached the median return target in $\theta_1, \ldots, \theta_n$3 frames (Willems et al., 2020).

6. Implementation: Practical Guidelines and Calibration

For model-level LeRaC, implementation leverages optimizer “parameter groups” (e.g., PyTorch) to assign per-layer learning rates. The only required hyperparameters are:

• $\theta_1, \ldots, \theta_n$4: number of curriculum epochs (2–7 recommended).
• $\theta_1, \ldots, \theta_n$5: lowest initial rate for deepest layer ($\theta_1, \ldots, \theta_n$6 or $\theta_1, \ldots, \theta_n$7 recommended).

Early stopping, weight decay, or learning-rate annealing should be deferred until after the curriculum phase. The method is robust to the schedule duration and initial LR scale.

For MR curriculum, key hyperparameters include window size $\theta_1, \ldots, \theta_n$8 for smoothing/regression, mixing parameters $\theta_1, \ldots, \theta_n$9 and $j$0, and redistribution rates $j$1. The curriculum logic requires only an acyclic task order and coarse return bounds per task. Extensions include automatic estimation of $j$2, adaptive schedule tuning, and hybridization with intrinsic motivation (Willems et al., 2020).

7. Limitations and Directions for Extension

The Learning Rate Curriculum (model-level) is broadly applicable, architecture-agnostic, and requires no data curation or auxiliary layers. However, it is not directly interpretable as curriculum over “semantic” content; it controls training dynamics at the representation level only (Croitoru et al., 2022).

Task-level Mastering Rate curriculum requires an explicit, user-defined curriculum (task DAG) and even coarse return ranges per task. In domains where problem structure is unavailable or inherently flat, MR reduces to classic learning-progress sampling. A plausible implication is that for highly structured domains (e.g., RL with compositional tasks, structured supervised curricula), mastering-rate methods provide principled sample allocation; for unstructured tasks, model-level LeRaC is preferred.

Extensions proposed include adaptive calibration of schedule and ancestor attention, automated inference of task orderings, and compositional schemes combining MR with model-level schedules or intrinsic exploration bonuses (Croitoru et al., 2022, Willems et al., 2020).