Logarithmic-Time Weight-Decay

• The topic introduces logarithmic-time weight-decay, where sublinear L2 regularization decay aligns with information-theoretic and optimization principles to improve training efficiency.
• It details reciprocal (1/t) and logarithmic decay schedules that, with a warmup phase, prevent singularities and maintain stability in gradient updates.
• Empirical studies show that this approach yields 10–15% compute savings and robust performance across scales, especially in training language models.

Logarithmic-time weight-decay refers to a class of weight-decay schedules in which the $L_2$-regularization coefficient decays sublinearly, typically either as $1/t$ or as a negative power of the logarithm of the training step, i.e., $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$. Such schedules are motivated both by statistical properties of language and optimization-theoretic considerations, and have demonstrated empirical efficiency gains in large-scale deep learning—especially in LLM training—relative to conventional constant-coefficient schemes (Ferbach et al., 5 Feb 2026, Richemond et al., 2019).

1. Standard Weight-Decay in Deep Nets

Standard practice with modern deep learning optimizers such as AdamW is to maintain an exponential moving average of past gradients ($\beta_1$), their squares ($\beta_2$), and to apply a decoupled $L_2$ weight-decay term with fixed strength $\lambda$ at every step. The canonical AdamW update, omitting bias corrections, is: $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$ where $\gamma(t)$ is a scheduled learning rate, $\gamma^*$ its peak, and $1/t$0 a numerical stabilizer. Across large-scale training tasks, $1/t$1 are almost always held constant (e.g., $1/t$2, $1/t$3, $1/t$4) (Ferbach et al., 5 Feb 2026).

2. Logarithmic-Time Weight-Decay Schedules

Logarithmic-time (log-time) weight-decay schedules replace the fixed $1/t$5 with a function that decays slowly over the course of training. The most common instantiation is the reciprocal rule: $1/t$6 where $1/t$7 is a scale-invariant hyperparameter. To avoid singularities at early steps ($1/t$8), a "warmup" of $1/t$9 is introduced ($\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$0 = total training iterations), giving: $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$1 This schedule corresponds to an exponential decay per unit of log-time, i.e., under the substitution $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$2, $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$3, and thus implements a constant attenuation in logarithmic time (Ferbach et al., 5 Feb 2026). An alternative, closely connected to statistical physics models of neural loss landscapes, is: $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$4 with $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$5 and $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$6 as hyperparameters (Richemond et al., 2019).

Schedule Type	Formula for $\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$7	Hyperparameters
Reciprocal ("1/t")	$\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$8	$\lambda(t) \propto [\log(1 + t/\tau)]^{-\gamma}$9, $\beta_1$0
Logarithmic	$\beta_1$1	$\beta_1$2, $\beta_1$3, $\beta_1$4
Power-law	$\beta_1$5	$\beta_1$6, $\beta_1$7

Both styles achieve a gradual reduction in regularization, matching the decaying information gain and improved signal-to-noise ratio as learning progresses.

3. Theoretical Motivation: Information-Theoretic and Optimization Perspectives

(a) Power-law Context Memory in Language

Information-theoretic studies (Shannon 1951, Hilberg 1990, Takahira 2016) establish that reductions in per-token uncertainty from increasing context length $\beta_1$8 follow a power law, i.e., $\beta_1$9 with $\beta_2$0. This implies each additional token yields diminishing information, with an effective context horizon that grows sublinearly. Consequently, regularization early in training should be much stronger than late, aligning with the 1/t decay (Ferbach et al., 5 Feb 2026).

(b) Complexity-Matched Weight-Decay

From a statistical physics viewpoint, the loss surface of heavily overparameterized deep nets exhibits isotropic Gaussian properties with critical-point complexity $\beta_2$1, a quadratic form in the normalized energy (loss) $\beta_2$2 and average Hessian eigenvalue $\beta_2$3. Complexity gradient descent prescribes that both energy and penalty decay in lockstep, yielding a "matched" schedule for weight-decay. This justifies closing the form of $\beta_2$4 to mirror that of the learning rate, typically via logarithmic or power-law schedules (Richemond et al., 2019).

(c) Riemann-Sum Attenuation

Log-time schedules for $\beta_2$5 ensure that each geometric window $\beta_2$6 contributes with constant attenuation to the parameter update, as opposed to the exponential attenuation of fixed $\beta_2$7, thus better matching the power-law decay in information content (Ferbach et al., 5 Feb 2026).

4. Stability Properties and Coupling to Momentum

Logarithmic-time weight-decay, when used alone, introduces no novel instabilities to AdamW or SGD. The simple reciprocal rule $\beta_2$8 requires no damping or adjustment for numerical stability. When pairing log-time decay of $\beta_2$9 with time-varying momentum hyperparameters $L_2$0—as in the "ADANA" optimizer—exact coordination and shared warmup offsets ($L_2$1) across schedules are required to maintain stability and prevent the gradient-momentum balance from being overwhelmed. However, for weight-decay alone, these constraints are relaxed (Ferbach et al., 5 Feb 2026).

5. Empirical Effects and Efficiency Gains

Implementing 1/t weight-decay within AdamW for decoder-only transformers ("Enoki" models) trained on the FineWeb corpus (Chinchilla regime: 20 tokens/parameter) yields consistent empirical gains:

• Optimal $L_2$2 is scale-invariant: A single $L_2$3 (determined by a small-scale sweep, e.g., 6-head model) is near-optimal for model sizes from $L_2$4 to $L_2$5 parameters.
• Compute-efficiency improvement: For identical validation loss, $L_2$6–$L_2$7 compute savings over fixed-weight-decay AdamW—quantified as

$L_2$8

where $L_2$9 is compute required by constant $\lambda$0, and $\lambda$1 by log-time decay.

• Efficiency gains increase slightly with model scale: $\lambda$2 at $\lambda$3 parameters to $\lambda$4 at $\lambda$5.
• Robustness: The same warmup offset $\lambda$6 suffices, no hyperparameter re-tuning required for different model sizes, and the gains persist under optimizer replacement (e.g., AdEMAMix: $\lambda$7–$\lambda$8 savings) (Ferbach et al., 5 Feb 2026).

6. Implementation and Practical Guidelines

To deploy logarithmic-time weight-decay:

• Use decoupled WD (AdamW-type), not inline "L2 regularization."
• Schedule: $\lambda$9, prefer $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$0.
• Choose $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$1 on a small model, retain this value across scales.
• Retain conventional values for $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$2, batch size, and learning-rate schedule.
• If time-varying $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$3 are employed, all schedules must share the same $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$4.
• Warmup offset $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$5 is essential to avoid instability at $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$6 (Ferbach et al., 5 Feb 2026).

For SGD-style training, matched logarithmic or power-law scheduling is easily implemented:

$\gamma(t)$0

7. Summary and Outlook

Logarithmic-time weight-decay—with representative formula $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$7—aligns the optimizer's regularization schedule with the power-law memory structure of natural language and the dynamics predicted by complexity-based loss landscape analysis. It is a computationally effective, scalable, minimally invasive modification requiring only a single transferable hyperparameter, and consistently provides $$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$8–$$\begin{aligned} m_{t+1} &= \beta_1 m_t + (1 - \beta_1) g_{t+1} \ v_{t+1} &= \beta_2 v_t + (1 - \beta_2) g_{t+1}^2 \ \theta_{t+1} &= \theta_t - \gamma(t)\left[\gamma^* \frac{m_{t+1}}{\sqrt{v_{t+1} + \epsilon}} + \lambda \theta_t\right] \end{aligned}$$9 reductions in training compute across transformer scales. This scheme requires no novel stabilization measures, can be incorporated as a drop-in to AdamW and related optimizers, and serves as a robust baseline for further advances in adaptive weight-decay scheduling (Ferbach et al., 5 Feb 2026, Richemond et al., 2019).