Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training

Abstract: Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate {\eta} and weight decay {\lambda}. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B. Recent work suggests the AdamW timescale, B/({\eta}{\lambda}D), should remain constant across training settings, and we verify the implication that optimal {\lambda} scales linearly with B, for a fixed N,D. However, as N,D scale, we show the optimal timescale obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict {\lambda}opt in advance of large-scale training. We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast with prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives.

• The paper shows that tuning weight decay via a derived power law on the AdamW timescale is key for efficient LLM pre-training.
• It finds that optimal and critical batch sizes scale as power laws in dataset size (approximately D^0.4 and D^0.5), independent of model size.
• The study offers a practical roadmap using μP and loss-data scaling laws to balance training time and compute costs in large-scale training.

Efficiently pre-training LLMs at scale requires careful tuning of hyperparameters like learning rate ($\eta$) and weight decay ($\lambda$). Traditional hyperparameter sweeping is often infeasible for the largest models due to computational costs. This paper presents scaling laws derived from hundreds of training runs to guide the selection of weight decay, batch size, model size ($N$), and dataset size ($D$) for optimal training performance, particularly focusing on practical tradeoffs between training time and total compute.

The paper investigates the scaling behavior of the AdamW timescale, defined as $\mathcal{T}_{epoch} = B / (\eta\lambda D)$. This metric represents the effective fraction of the training data over which weight updates are averaged. While previous work suggested keeping this constant for multi-epoch training, this study demonstrates that for LLM pre-training (typically one epoch), the optimal $\mathcal{T}_{epoch}$ is not constant but follows a precise power law in the tokens-per-parameter ratio ($TPP = D/N$). The fitted law shows $\mathcal{T}_{epoch} \propto TPP^{-0.5}$, meaning the optimal timescale decreases as models are trained on more data relative to their size.

A key practical implication is that, when using the AdamW optimizer and the Maximal Update Parameterization ($\mu$P) framework to set the learning rate, the weight decay ($\lambda$) should be the primary hyperparameter adjusted as batch size ($\lambda$0) and dataset size ($\lambda$1) change. The paper shows empirically that tuning $\lambda$2 to maintain the optimal $\lambda$3 is more effective than tuning $\lambda$4 as $\lambda$5 or $\lambda$6 varies. This provides a concrete recipe for practitioners: use $\lambda$7P to set $\lambda$8 based on model size $\lambda$9, and then set $N$0 using the derived scaling law for $N$1 and the formula $N$2. The linear relationship between optimal $N$3 and $N$4 for fixed $N$5 holds up to a certain batch size, further supporting adjusting $N$6 with $N$7.

The study also provides insights into optimal batch size ($N$8) and critical batch size ($N$9). $D$0 is the batch size that achieves the lowest loss for a given $D$1 and $D$2. $D$3 is defined based on an empirical model of the tradeoff between the number of tokens ($D$4) and the number of optimization steps ($D$5) required to reach a target loss $D$6. The model is expressed as $D$7, where $D$8 and $D$9 are the minimum tokens and steps, respectively, and $\mathcal{T}_{epoch} = B / (\eta\lambda D)$0. The paper introduces a novel method to estimate $\mathcal{T}_{epoch} = B / (\eta\lambda D)$1 by fitting batch-size-specific loss-data scaling laws and interpolating the data needed for a target loss, avoiding the need for dense checkpoint evaluation or constant learning rates.

Contrary to some prior work that suggested $\mathcal{T}_{epoch} = B / (\eta\lambda D)$2 and $\mathcal{T}_{epoch} = B / (\eta\lambda D)$3 scale primarily with total compute ($\mathcal{T}_{epoch} = B / (\eta\lambda D)$4) or target loss ($\mathcal{T}_{epoch} = B / (\eta\lambda D)$5), this paper finds that both $\mathcal{T}_{epoch} = B / (\eta\lambda D)$6 and $\mathcal{T}_{epoch} = B / (\eta\lambda D)$7 scale as power laws in the dataset size $\mathcal{T}_{epoch} = B / (\eta\lambda D)$8, largely independent of model size $\mathcal{T}_{epoch} = B / (\eta\lambda D)$9. Specifically, the findings suggest $\mathcal{T}_{epoch}$0 and $\mathcal{T}_{epoch}$1. This aligns with recent concurrent work (Zhang et al., 2024), reinforcing the fundamental dependence of optimal and critical batch sizes on the amount of data used. Practically, this means practitioners can estimate these values from small-scale runs and extrapolate based on the training data size. The $\mathcal{T}_{epoch}$2 tradeoff equation $\mathcal{T}_{epoch}$3 can then be used to understand the computational cost (proportional to $\mathcal{T}_{epoch}$4) and training time (proportional to $\mathcal{T}_{epoch}$5 and $\mathcal{T}_{epoch}$6) implications of choosing a particular batch size $\mathcal{T}_{epoch}$7 for a given $\mathcal{T}_{epoch}$8 and target loss $\mathcal{T}_{epoch}$9.

Using these derived scaling laws, the paper analyzes the Pareto-optimal configurations for $TPP = D/N$0 to achieve a target loss $TPP = D/N$1 while balancing training time and compute. For a fixed total compute budget, traditional advice suggests training at roughly 20 TPP to minimize loss. However, considering training time (which decreases with larger $TPP = D/N$2, but increases total $TPP = D/N$3 and compute via the $TPP = D/N$4 tradeoff), the analysis reveals that smaller, over-trained models (TPP > 20) can be Pareto-optimal. This is because overtrained models are trained on larger datasets, leading to higher $TPP = D/N$5 and thus allowing for more efficient use of larger batch sizes to reduce training time, even if the total compute is higher than the 20 TPP optimum. It is shown to be Pareto-inefficient to target 20 TPP actual training TPP when using very large batch sizes ($TPP = D/N$6).

The core practical takeaways are:

• When using AdamW and $TPP = D/N$7P, fix the learning rate based on model width and tune weight decay ($TPP = D/N$8) based on the derived $TPP = D/N$9 scaling law ($\mathcal{T}_{epoch} \propto TPP^{-0.5}$0) and batch size ($\mathcal{T}_{epoch} \propto TPP^{-0.5}$1).
• Optimal and critical batch sizes ($\mathcal{T}_{epoch} \propto TPP^{-0.5}$2) scale with dataset size $\mathcal{T}_{epoch} \propto TPP^{-0.5}$3, not total compute $\mathcal{T}_{epoch} \propto TPP^{-0.5}$4 or target loss $\mathcal{T}_{epoch} \propto TPP^{-0.5}$5. Estimate their scaling from small runs ($\mathcal{T}_{epoch} \propto TPP^{-0.5}$6 and $\mathcal{T}_{epoch} \propto TPP^{-0.5}$7) and use these laws to select $\mathcal{T}_{epoch} \propto TPP^{-0.5}$8 for large-scale training.
• Leverage the $\mathcal{T}_{epoch} \propto TPP^{-0.5}$9 tradeoff model and the $\mu$0 scaling law to plan training runs that optimally balance total compute (FLOPs) and training time, especially considering the benefits of larger batch sizes. This may favor smaller, over-trained models for faster training times at a given performance level.

Implementation requires:

1. Train a proxy model with $\mu$1P to find base hyperparameters, including $\mu$2.
2. For target model width $\mu$3, set peak $\mu$4.
3. From limited small-scale experiments across various $\mu$5 and losses, fit:
   • The optimal $\mu$6 power law with TPP [(2505.13738), Eq. 3].
   • The $\mu$7-specific loss-data power laws $\mu$8 [(2505.13738), Fig. 4].
   • The $\mu$9 tradeoff curve $\lambda$0 [(2505.13738), Eq. 6], yielding $\lambda$1 and $\lambda$2 for various losses. Calculate $\lambda$3.
   • The $\lambda$4 power law with $\lambda$5 [(2505.13738), Eq. 7].
4. For a desired target loss $\lambda$6, model size $\lambda$7, and dataset size $\lambda$8:
   • Predict the target $\lambda$9 for loss $\lambda$00 at size $\lambda$01 (e.g., using a loss scaling law like [(Ouyang et al., 2022), Eq. 1]).
   • Predict $\lambda$02 for this $\lambda$03 using the fitted power law.
   • Choose a batch size $\lambda$04. The required dataset size will be $\lambda$05. Ensure the actual dataset used is at least this size.
   • Calculate the required steps $\lambda$06. Configure the LR schedule accordingly.
   • Calculate the target $\lambda$07 for the chosen $\lambda$08 ratio.
   • Set $\lambda$09.

This systematic approach, grounded in empirical scaling laws, offers a more predictable and efficient way to navigate the complex hyperparameter space for large-scale LLM pre-training, enabling better control over training costs and time. The findings also highlight that achieving the fastest training time for a given performance level might involve training models on significantly more data than the compute-optimal minimum.

Some limitations mentioned include the focus on AdamW and a specific LR schedule shape, the need for more data/architectural variations in future work, and the empirical observation that small batches degrade performance more than theory suggests, potentially requiring tuning beyond $\lambda$10. Practical systems constraints of very large batches are also not explicitly modeled.