Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most

Abstract: We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The pattern appears on ForecastBench-Sim (FBSim), a contamination-free, simulated-world benchmark we release, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation. A per-quantile decomposition shows the failure concentrates at the upper tail, which more capable models shift upward to track aggressive extrapolations of growth, while the lower tail stays put. A within-family study of Llama-3.1 shows that both model scale and post-training independently contribute to this effect. Domain knowledge does not reliably rescue calibration. This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs. Single-threshold scoring at conventional cutoffs misses the upper-tail cost; tail-inclusive scoring reverses the sign of the capability--accuracy relationship on the same outputs. We recommend that LLM forecasting evaluations use continuous (and unbounded) measures of accuracy alongside bounded binary threshold metrics.

• The paper demonstrates that higher capability LLMs produce systematically poorer upper-tail forecasts during regime changes in risk-sensitive domains.
• Using both synthetic and real-world benchmarks, the study finds that traditional threshold metrics overlook the inverse scaling of forecast accuracy revealed by CRPS.
• The findings imply that current evaluation methods must be revised to address LLM overcommitment in tail distributions for safe deployment in high-stakes applications.

Inverse Capability Scaling in LLM Forecasting: A Systematic Liability in the Tails

Abstract and Core Contribution

The paper "Is Capability a Liability? More Capable LLMs Make Worse Forecasts When It Matters Most" (2605.22672) identifies a structurally grounded failure mode in modern LLMs: on distributional forecasting tasks characterized by superlinear growth and the risk of regime change—a signature of domains such as finance and epidemiology—LLMs with higher measured capability make systematically worse forecasts, specifically in the upper tails of the predicted distributions. This inverse scaling is robust across simulated and real-world data, persists across model lineages, and is undetectable with the binary/threshold scoring rules prevalent in the LLM forecasting literature. The findings have direct implications for the practical application of LLMs in risk-sensitive domains and for the methodology of model evaluation.

Figure 1: On a continuous proper scoring rule (red), more capable models produce worse forecasts in two, unrelated domains. This inverse scaling does not appear on a threshold-based proper scoring rule (blue). Top: FBSim, pooled. Bottom: pre-vaccine measles. Both series shown at furthest forecast horizons.

Empirical Overview: Benchmarking Inverse Scaling

The paper establishes its claims using multiple, complementary sources:

• ForecastBench-Sim (FBSim): a contamination-free synthetic strategy-game benchmark, with diverse, discontinuous shocks and varying outcome structures.
• Synthetic Epidemiological (SIR) Series: controlled procedural simulations separating the confounding effects of trend and regime change.
• Real-World Data Replications: broadening the evaluation to COVID-19 incidence, housing prices during the bubble era, hyperinflation episodes, and historic measles outbreaks, all with intrinsic superlinear growth/regime change dynamics.

Numerical results highlight the effect's strength. On FBSim, at the longest forecast horizon, the Spearman correlation between capability (as measured by the Epoch Capabilities Index, ECI) and tail-integrating forecast accuracy (CRPS) is $\rho = -0.42$ ($N=28$), with bootstrapped 95% CIs $[-0.72, -0.02]$. On real COVID-19, housing, and hyperinflation data, $\rho$ reaches $-0.67$ for housing prices and $-0.59$ in hyperinflation, both with bootstrapped CIs excluding zero.

Figure 2: Across-horizon evolution of the capability–accuracy relationship. CRPS reveals positive scaling at short horizons, crossing to inverse scaling at longer horizons in both synthetic (top) and real (bottom) domains, while Brier remains positive.

Mechanism: The Upper-Tail Overcommitment

A critical analytic advance of the paper is the decomposition of CRPS into per-quantile pinball loss, which isolates the source of inverse scaling to the upper tail (p90 quantile and above) of the predictive distributions.

• Upper-Tail Inflation: With increasing model capability, the upper quantiles (p90) of predicted distributions are driven upward, closely tracking an extrapolated continuation of the pre-crash trajectory. However, when regime change occurs (e.g., post-peak in an epidemic or market crash), these predictions systematically overshoot the realized outcome, leading to sharply increased loss.
• Stability of the Lower Tail: The lower tail (p10) remains comparatively stable, suggesting that the models' ability to capture crash scenarios does not improve or degrade with capability.

  Figure 3: Upper-tail predictions drive the inverse scaling, in both domains. The p90 quantile swings from strongly positive scaling at short horizons to strongly negative at long horizons, while the p10 remains flat.

Synthetic SIR experiments confirm that the joint presence of superlinear growth and regime-change is necessary to trigger the effect. On matched series with only linear growth (plus a crash), higher capability restores the usual positive scaling.

Figure 4: Only time series with superlinear growth plus regime change exhibit inverse scaling; purely linear data, even with a crash, does not.

Scale, Post-Training, and Prompt Interventions

The pattern persists across scale, alignment, and post-training variations. A controlled $2{\times}2$ within-family experiment on Llama-3.1 (70B/405B × base/instruct) demonstrates that both scaling up the model size and performing RLHF-style post-training independently—and compositely—worsen CRPS under the trigger conditions, compounding the liability.

Prompt-based interventions focusing on uncertainty cues or explicit domain identity yield inconsistent rescues:

• Explicit domain information (e.g., naming the disease or economic context) sometimes attenuates, but does not eliminate, inverse scaling.
• Direct knowledge probes indicate the models possess relevant priors, but these are not consistently translated into calibrated upper tails.

  Figure 5: Domain knowledge has inconsistent effects across domains; only in select cases does naming the domain rescue positive scaling.

Evaluation Metrics: A Critical Methodological Artifact

A principal finding is that single-threshold scoring rules (e.g., Brier at a salient cutoff) are blind to the inverse scaling failure. On the same models and outputs, threshold-scoring metrics report a positive or null capability–accuracy relationship, whereas CRPS, as a tail-integrating proper scoring rule, reveals the inversion.

• The sign of scaling can be reversed by switching metrics, meaning standard LLM forecasting benchmarks relying solely on threshold metrics will systematically miss or misrepresent model liabilities in risk-critical applications.

Implications and Future Research

These findings undermine the naive deployment of state-of-the-art LLMs as modular forecasters in high-variance domains. In scenarios where tail events drive real-world costs—pandemics, financial crises, systemic risks—the most capable LLMs will systematically overcommit to continuation of growth and fail to represent regime change in probabilistic forecasts. This constitutes a substantive liability for automated decision-making pipelines in policy and risk management.

Theoretically, the result foregrounds a divergence between retrieval-based knowledge (the model possesses correct priors) and calibrated representation of distributional uncertainty under out-of-domain shifts. The positive scaling under threshold metrics but inverse scaling under continuous scoring rules presents an acute methodological challenge; benchmark design must shift to include tail-integrating rules or sweeping of binary thresholds.

The open question is mechanistic: why does this upper-tail overcommitment persist with model scale and post-training, and why are prompt-level domain disclosures inconsistent in effect? Progress in mechanistic interpretability and further controlled experiments will be necessary to resolve this gap.

Conclusion

This paper rigorously demonstrates that, contrary to prevailing expectations, increased LLM capability can systematically degrade performance on time series forecasting tasks with superlinear growth and regime transitions, precisely where robust uncertainty estimation is essential. These findings demand a reorientation of LLM evaluation and suggest the need for new techniques—architectural, procedural, or metric-based—to correct tail miscalibration before safe deployment in high-stakes applications.