Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training (2512.08894v1)

Abstract: While scaling laws for LLMs traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure, which is prone to compounding errors. Furthermore, we introduce functional forms that predict accuracy across token-to-parameter ratios and account for inference compute under repeated sampling. We validate our findings on models with up to 17B parameters trained on up to 350B tokens across two dataset mixtures. To support reproducibility and encourage future research, we release the complete set of pretraining losses and downstream evaluation results.

• The paper introduces a direct power law scaling approach to forecast downstream LLM accuracy based on training compute, dataset composition, and token-to-parameter ratios.
• It shows that direct scaling outperforms traditional proxy-based methods by reducing compounding errors and enhancing extrapolation stability across diverse benchmarks.
• Empirical results from 130 model runs highlight the critical role of data mixture sensitivity and establish robust guidelines for planning LLM training budgets.

Scaling Laws for Downstream Metrics in LLM Training

Introduction

This paper challenges prevalent views regarding the predictability of downstream benchmark performance in LLMs. Existing practices extensively leverage scaling laws, particularly those developed for proxy metrics such as pretraining loss or negative log-likelihood, to forecast optimal LLM training resource allocation. However, direct prediction of downstream accuracy is widely perceived as unreliable due to noise, domain-specific thresholding, and emergent phenomena. This work systematically analyzes scaling behavior across twelve popular benchmarks in models up to 17B parameters, proposing a unified functional form—a power law—for direct extrapolation of downstream performance as a function of training budget, dataset composition, and token-to-parameter ratio.

Empirical Framework and Direct Scaling Law

The experimental suite includes 130 model runs on two dataset mixtures—C4 and a curated mix of general, mathematical, and code data—spanning compute budgets up to $3.77 \times 10^{22}$ FLOPs. The paper's central claim is that, for a fixed token-to-parameter ratio, normalized log accuracy on diverse downstream tasks displays a smooth, predictable dependence on training compute, well fit by the proposed power law:

$-\log(Q) = \frac{A}{C^\alpha}$

where $Q$ is normalized accuracy, $A$, $\alpha$ are benchmark-specific, and $C$ is training FLOPs.

The benchmark suite includes ARC-Easy, ARC-Challenge, SciQ, PIQA, HellaSwag, WebQS, Winogrande, LAMBADA, TriviaQA, GSM8k, HumanEval, and LBPP, spanning classification, generation, reasoning, and coding.

Downstream accuracy scaling under this law demonstrates robust fits (average MAE $<$ 3%), even for overtrained models and varying token to parameter regimes. Notably, the direct approach excels at forecasting held-out high-compute accuracy from fits at moderate scale. The S-shaped trend (convex then concave) observed in ARC-Easy and HellaSwag is effectively captured, surpassing prior piecewise linear and broken scaling law models.

Figure 1: Scaling law fits. Comparing the direct approaches: Power Law and BNSL with two-stage approaches (Linear and Logistic) for ARC Challenge accuracy extrapolation.

Comparison to Two-Stage and Proxy-Based Approaches

Traditional two-stage pipelines first regress compute to proxy metrics (e.g., NLL or validation loss), then transform this metric via non-linear mapping (e.g., logistic) to benchmark accuracy. This study demonstrates:

• Multiple proxy metrics (log-probability, validation loss, Brier Score) are highly correlated with downstream accuracy ($R^2 > 0.95$), as confirmed by logistic fits across tasks.

  Figure 2: Dependency between downstream task and proxy metric candidates shows strong prediction, i.e.~high $R^2$ and low RMSE, for all metrics.

• Despite this, the two-stage procedure is suboptimal for extrapolation: compounding errors in multi-step fitting lead to higher MAE and reduced robustness under distribution shift or scale variation.
• In all tested benchmarks, direct scaling outperforms two-stage approaches in validation error and extrapolation stability.

Figure 3: Comparing various proxy metrics for different benchmarks confirms high predictive power but does not confer extrapolation stability in multi-stage procedures.

Extension Across Token-to-Parameter Ratios and Sampling

The authors generalize the scaling law to:

$-\log Q = \frac{A}{N^\alpha} + \frac{B}{D^\beta}$

where $N$ is model size (parameters), $D$ is dataset size (tokens), $A,B,\alpha,\beta$ fit the observed empirical scaling.

Additionally, for coding benchmarks with pass@k metrics, they derive an explicit functional form connecting training compute and inference budget to pass rate, enabling tractable compute/inference tradeoff analysis.

Data Mixture Sensitivity

Using the C4 dataset in isolation demonstrates attenuation of code and math-specific benchmark scaling, confirming that scaling coefficients and extrapolability are valid only within the context of the training data mixture, which substantially influences downstream transfer.

Figure 4: ARC-Easy scaling on C4-composed versus domain-augmented datasets illustrates the dependence of downstream performance scaling on data mixture.

Robustness and Critical Scale Thresholds

• Aggregated trends are generally smooth for established benchmarks, but composite tasks (e.g., BIG-bench style mixes) may present non-monotonic or thresholded scaling, limiting extrapolability.
• Reliable extrapolation is only feasible above task-specific minimal FLOPs thresholds; fits are unstable below these scales (see logistic regression analysis in supplementary material).

Figure 5: Trend of two metrics MRE and MAE across benchmarks and scaling laws, highlighting robust extrapolation regions for the direct approach.

Implications and Future Work

The work refutes the notion that downstream scaling laws are unreliable due to noise or emergent breaks, establishing pragmatic recipes for robust model planning under fixed training and dataset regimes. Key implications include:

• Downstream accuracy can now be forecasted for budget planning without reliance on brittle proxy mapping, improving efficiency and confidence in large-scale LLM development.
• The demonstrated sensitivity to data composition underscores the necessity for standardized benchmarking and dataset transparency in scaling law studies.
• The empirical functional forms invite future work on mechanistic and theoretical grounding, such as connections to item-difficulty, error decay, or entropy measures.

Potential extensions involve calibrated prediction intervals, expanded support for post-pretraining adaptation (instruction tuning, RLHF), mixture-aware metric audits, and robust scaling under advanced architectures (MoE, retrieval-augmented, multimodal transformers).

Conclusion

This paper provides compelling evidence that downstream performance of LLMs on diverse benchmarks scales predictably with training compute, model size, and dataset size when expressed via a direct power law. Thorough empirical validation across model families and benchmark types supports the adoption of simple, direct scaling laws for downstream capabilities forecasting, conditional on data mixture and above minimal compute thresholds. This advances practical methodology for systematic, reproducible planning and evaluation of frontier LLMs.

Figure 6: Illustration of the fit quality for predicting the average across benchmarks using the direct scaling law.