Forecasting Downstream Performance of LLMs With Proxy Metrics

Abstract: Progress in LLM development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which training recipe to apply. Making these decisions well requires reliable performance forecasts, yet the two commonly used signals are fundamentally limited. Cross-entropy loss is poorly aligned with downstream capabilities, and direct downstream evaluation is expensive, sparse, and often uninformative at early training stages. Instead, we propose to construct proxy metrics by aggregating token-level statistics, such as entropy, top-k accuracy, and expert token rank, from a candidate model's next token distribution over expert-written solutions. Across three settings, our proxies consistently outperform loss- and compute-based baselines: 1) For cross-family model selection, they rank a heterogeneous population of reasoning models with mean Spearman Rho = 0.81 (vs. Rho = 0.36 for cross-entropy loss); 2) For pretraining data selection, they reliably rank 25 candidate corpora for a target model at roughly $10{,}000\times$ less compute than direct evaluation, pushing the Pareto frontier beyond existing methods; and 3) for training-time forecasting, they extrapolate downstream accuracy across an $18\times$ compute horizon with roughly half the error of existing alternatives. Together, these results suggest that expert trajectories are a broadly useful source of signal for assessing model capabilities, enabling reliable performance forecasting throughout the model development life cycle.

• The paper demonstrates that proxy metrics, derived from token-level statistics on expert reasoning traces, outperform traditional metrics like cross-entropy loss.
• It details a methodology using a library of 80 proxy metrics extracted in a single forward pass, enabling efficient ranking of LLMs and pretraining data evaluation.
• Empirical results show proxy-based extrapolation predicts final task accuracy with half the error of traditional methods, providing a robust forecasting framework.

Proxy Metrics for LLM Downstream Performance Forecasting: Methods, Results, and Implications

Introduction

Accurately forecasting downstream task performance during LLM development is critical for informed decisions about architecture, pretraining data, and training duration. Traditional signals such as pretraining cross-entropy (CE) loss are poorly aligned with hard downstream reasoning capabilities, while direct evaluation on downstream tasks is both computationally expensive and typically uninformative at early training stages. The paper "Forecasting Downstream Performance of LLMs With Proxy Metrics" (2605.18607) systematically addresses these challenges by developing proxy metrics computed from token-level statistics over model predictions on expert reasoning trajectories. These proxy metrics are empirically validated as both computationally efficient and highly predictive of downstream performance across heterogeneous model families, tasks, and training regimes.

Methodology: Proxy Metric Construction

The core methodological advance is the construction of a diverse library of proxy metrics, all derived from a model's next-token predictive distribution evaluated along expert-provided solution traces (either from high-performing models or humans). Fundamental aspects include:

• Token-level statistics: Cross-entropy, entropy, rank and reciprocal rank of the expert token, margin, and wrong-confidence, all computed at every position in the expert trace.
• Weighting schemes: Each statistic is aggregated over the trajectory using weights that emphasize high-value positions, such as those with high model uncertainty or rare tokens, leading to 80 distinct proxy metrics.
• Single forward pass efficiency: All metrics can be extracted from a single model forward pass per expert trace, yielding a dense, task-anchored signal with trivial compute cost.

  Figure 1: Overview of the proxy metric computation pipeline, aggregating token-level statistics using the model's next-token prediction over expert trajectories.

Cross-Family Model Selection: Ranking LLMs Without Downstream Evaluation

A primary application is relative ranking of candidate LLMs on challenging reasoning tasks, where either benchmark evaluation is prohibitively costly or sparsity of ordinal signal precludes direct task-based ranking.

• Benchmark suite: Six reasoning benchmarks—AIME, HMMT, GPQA, USACO, MMLU-Pro, and SuperGPQA—covering mathematical, scientific, and code tasks.
• Model set: 18 models, spanning major open-weight architectures and post-training recipes from 0.6B to 70B parameters.
• Proxy selection: Linear and nonlinear (RankSVM, sparse linear) functions over proxy metrics are fit (using held-in benchmarks and/or models) for generalization to unseen tasks or model variants.
• Comparators: Baselines include CE loss over generic text and rBridge [JSE40ljyKm], which reweights model likelihoods by expert logprobs.

Key empirical results:

• The linear RankSVM proxy achieves mean cross-validated Spearman $\rho = 0.81$ for model ranking, far surpassing CE loss ($\rho = 0.36$) or rBridge ($\rho = 0.33$).
• Even with only a small fraction of models or tasks used for proxy selection, high correlations are maintained.
• The ranking signal is highly concentrated: only a handful of the 80 proxy metrics are selected with high frequency, with the best univariate metrics upweighting rare or high-entropy tokens indicative of critical reasoning steps.

Figure 2: (Left) The linear RankSVM proxy robustly recovers downstream model rankings on held-out reasoning benchmarks. (Right) In pretraining corpus ranking, the proxy metric achieves high decision accuracy with orders-of-magnitude less compute than downstream evaluation.

Figure 3: Proxy metric selection frequency, revealing that a narrow cluster of metrics accounts for most ranking capability.

Pretraining Data Selection: Ranking Corpora at Miniscule Compute Budgets

The next application targets efficient ranking of diverse pretraining corpora, a central operational decision with multi-million dollar implications. The DataDecide testbed [magnusson2025datadecide] evaluates whether small proxy models can reliably rank the utility of corpora for downstream LLM performance.

• Experiment: 25 pretraining corpora; rankings induced by training 4M–90M proxy models and measuring their alignment with 1B-model downstream performance.
• Outcome: Frequency-weighted top-5 accuracy as a proxy metric delivers decision accuracy above 0.85 at only $10^{-5}$ of target compute, dramatically outperforming prior state-of-the-art (rBridge) and direct downstream evaluation baselines.

Interpretation: At scales where benchmark accuracy is at chance, the model’s compatibility with expert trajectories differs measurably across corpora, making proxy metrics highly discriminative.

Training-Time Forecasting: Extrapolation Across Long Horizons

A persistent open problem is accurate prediction of end-of-training downstream accuracy from early checkpoints, given highly non-linear and often unreliable scaling laws [lourie2025unreliable]. The proxy metric approach addresses this by:

• Fitting smooth (power-law) trajectories for proxy metrics across checkpoints, then extrapolating.
• Empirical validation on OLMo-3-7B pretraining/post-training trajectories and ten OLMES-style downstream benchmarks.

Quantitative findings:

• Across tasks, proxy-based extrapolation achieves a mean RMSE of 0.024 in predicting final accuracy—approximately half the error of CE loss- or compute-based sigmoidal fits.
• The proxy-to-accuracy relationship remains monotonic and stable even through compute horizons as large as $18\times$ beyond the observed window.

Figure 4: Proxy metrics evolve predictably along both pretraining and post-training trajectories, enabling accurate extrapolation to unseen computational regimes.

Figure 5: Proxy metric power-law fits track task accuracy on HellaSwag with far lower extrapolation error than CE loss-based approaches.

Analysis and Broader Implications

Contradictory finding: The traditional reliance on CE loss for downstream capability forecasting is robustly contradicted in cross-family settings and for hard reasoning tasks—scalar loss metrics are poorly monotonic and non-discriminative, especially across architectures or training regimes.

Theoretical implications:

• Token-level, task-conditioned evaluation provides a much denser and more reliable learning signal than traditional aggregate metrics.
• The critical efficacy of rare or high-entropy tokens as discriminators is consistent with RL-based findings on high-entropy “forking tokens” in reasoning (Wang et al., 2 Jun 2025).
• Proxy metrics essentially serve as a process-level diagnostic, offering a continuous relaxation of argmax-based accuracy and grounding prediction in the model’s latent reasoning competence.

Practical implications:

• Rapid, compute-light evaluation: All metrics are available from a single forward pass and require no access to expert logprobs—enabling usage of closed-weight models or human-provided traces.
• Enabling early and granular decision-making: Accurate ranking of models/corpora and training-stage extrapolation—potentially saving large-scale compute and reducing risk in LLM development.
• Flexibility: The methodology generalizes across tasks, models, and data regimes, pending further empirical work for domains beyond reasoning (e.g., long-context or agentic evaluations, mixture-of-experts, etc.).

Limitations and Open Problems:

• Oracle upper bounds indicate further potential: Cross-validation leaves a measurable gap to performance where proxy selection is done with full knowledge of target tasks.
• Generalization to broader model scales, architectures, and more diverse task regimes remains open.
• The requirement for high-quality expert trajectories could limit applicability in domains with limited human or model expertise.

Conclusion

This study establishes proxy metrics over expert reasoning traces as a robust, efficient, and highly predictive framework for downstream LLM performance forecasting. By providing dense, task-specific signals where traditional metrics fail, proxy metrics enable superior model and data selection as well as accurate, long-horizon training-time forecasting. These findings substantially improve the fidelity and efficiency of model development decisions in the LLM pipeline and suggest several avenues for future research on end-to-end aggregation learning, extension to generative/agentic settings, and the interplay between expert quality and proxy signal (2605.18607).