Generalization of the proposed scaling laws across training setups and tasks

Determine the extent to which the three-term scaling law L(N,M,K) = E + A/N^alpha + B/M^beta + C/K^gamma and the model-specific two-term laws L(M,K) = E + B/M^beta + C/K^gamma generalize to different training setups and tasks, assessing whether their fitted parameters and implied optimal batch-size scaling remain valid beyond the settings studied in this work.

Background

The paper proposes a three-term scaling law that models the final loss as a function of model size N, token batch size M, and number of training steps K, and an alternative two-term formulation fitted per model size. These laws are evaluated on two datasets (Li and OpenEuroLLM) and shown to capture optimal batch-size scaling and aspects of critical batch size behavior.

However, the authors observe quantitative inconsistencies across datasets (e.g., differing parameter estimates such as E and tau), and show that reducing the three-term law back to a Chinchilla-type law can yield substantially different compute-optimal allocations. This motivates examining whether the fitted forms and their implications remain stable across broader training configurations and tasks.