Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking (2511.18394v1)

Abstract: LLMs demonstrate partial forecasting competence across social, political, and economic events. Yet, their predictive ability varies sharply with domain structure and prompt framing. We investigate how forecasting performance varies with different model families on real-world questions about events that happened beyond the model cutoff date. We analyze how context, question type, and external knowledge affect accuracy and calibration, and how adding factual news context modifies belief formation and failure modes. Our results show that forecasting ability is highly variable as it depends on what, and how, we ask.

• The paper highlights the uneven forecasting performance of LLMs across domains, showing strong calibration in structured areas and notable weaknesses in volatile ones.
• It introduces a robust benchmarking pipeline that filters 10,000 market questions to 392 verifiable events, assessing models with metrics like Brier Score and Expected Calibration Error.
• The study identifies key failure modes such as recency bias, rumour overweighting, and definition drift, emphasizing the need for domain-aware evaluation and improved context handling.

Uneven Distribution of Forecasting Ability in LLMs: An Evaluation Across Domains and Contexts

Introduction

This study rigorously evaluates the forecasting competence of state-of-the-art LLMs, delineating how predictive performance varies not only by model architecture but critically by question structure, domain, and contextual input. Utilizing forecast questions from major prediction platforms with future ground truth (post-model cutoff dates), the authors construct a filtered benchmark and perform comparative analysis across Entertainment, Finance, Geopolitics, Politics, Sports, and Technology. Both raw and news-augmented conditions are assessed, exposing vital failure modes that undermine LLM reliability in practical deployment scenarios.

Benchmark Construction and Evaluation Protocol

The authors implement a multi-stage, LLM-driven curation pipeline on 10,000 raw market questions, performing volume-based and contextually aware filtering to exclude signals uninformative for causal or probabilistic reasoning, resulting in a final balanced set of 392 event-grounded, publicly verifiable questions. Each is categorized and paired with timestamp metadata to simulate temporally bounded inference. The benchmark evaluation covers GPT-5, GPT-4.1, DeepSeek-R1, and Claude 3.7 Sonnet, measuring accuracy, Brier Score, and Expected Calibration Error (ECE) under deterministic zero-temperature sampling for both direct and context-augmented forecasts.

Figure 1: Data-processing pipeline for the construction of the filtered, balanced forecasting benchmark, eliminating non-informative or non-verifiable questions.

Domain-Specific Performance Analysis

Performance metrics reveal highly variable forecasting capability by domain and model, challenging the notion of emergent, cross-domain reasoning in large-scale pretrained models.

• In structured domains such as Geopolitics and Politics, Claude-3.7 and GPT-5 exhibit strong calibration (ECE as low as 0.09 for GPT-5 in Geopolitics; accuracy up to 88%). This reflects effective alignment with publicly available priors and stable event dynamics.
• In noisy or rapidly evolving domains such as Entertainment and Technology, ECE and Brier scores degrade noticeably (e.g., GPT-4.1 ECE of 0.47 in Technology), with accuracy often sub-60% for Finance and Sports. This is compounded by prompt-induced ambiguity and lack of stable reference signals.

Augmentation with contemporaneous news context shifts performance profiles:

• Improvement is observed in domains where recent news directly informs outcomes, notably Finance and Sports (Brier decreases for GPT-5 in Finance from 0.26 to 0.23).
• Detrimental effects, including calibration degradation and directional error, appear in domains like Entertainment and Technology, reflecting increased model susceptibility to recency and salience biases inherent in retrieved snippets.

  Figure 2: Impact of news context on forecasting accuracy across categories (left: domains where news aids, right: domains where news degrades), and schematic of recurring failure modes.

Characterization of Failure Modes

The research provides clear taxonomy and concrete reasoning traces for several systematic forecast failures:

Recency Bias: LLMs reweight predictions toward the most temporally proximate headlines. In financial questions, this leads to reversal of correct conservative calls in favor of momentum-biased, outcome-incorrect forecasts as demonstrated in S&P 500 prediction tasks.

Rumour Overweighting: When exposed to speculative or unverified news, models anchor final probabilities on perceived "trend" rather than evidentiary truth, flipping answers across the threshold due to momentum narratives rather than institutional or procedural likelihood.

Definition Drift: Ambiguous acronyms or entity references cause semantic frame-switching under news augmentation. This results in model confusion between similarly named events (e.g., MATS interpreted as academic program vs. trade show), breaking logical consistency with the original prompt context.

In addition, DeepSeek-R1 is noted for its omission of explicit reasoning traces despite prompt scaffolding requirements, indicating model-specific deficiencies in interpretable probabilistic reasoning output.

Practical and Theoretical Implications

Results demonstrate that neither scale nor prompt engineering alone produces robust, generalizable LLM-based forecasters. Forecasting reliability is conditional on both model design and the specificity/structure of the queried event. The naive application of news context, while intuitively beneficial, provokes contradictory outcomes and amplifies failure modes due to semantic drift and salience misalignment. Benchmarking must therefore disentangle retrieval-augmented knowledge recall from actual inferential competence.

The study further substantiates the inadequacy of existing static benchmarks for forecasting evaluation and motivates methodological advances: dynamic, context-grounded testbeds; adversarial prompt diversification; and rigorous internal trace analysis for failure taxonomy. From an application standpoint, caution is advised in using LLMs for high-stakes prediction or market-anchored inference tasks without domain-aware filtering, calibrated context, and robust failure detection.

Future Directions

Subsequent research should consider hybrid retrieval/reasoning pipelines with counterfactual control, improved context disambiguation, and direct model supervision on failure mode minimization. The intersection of LLM knowledge, external retrieval, and robust uncertainty quantification remains an open frontier. Advancements in interpretability, meta-calibration, and automated bias compensation will be required for trustworthy deployment in forecasting, policy analysis, and market governance.

Conclusion

This investigation exposes the conditional, uneven nature of forward-looking inference in LLMs, establishing that the addition of news context and the framing of prompts can both enhance and degrade accuracy depending on domain and question features. Systematic failure modes—recency bias, rumour anchoring, definition drift—can invert correct predictions under naive augmentation, underscoring the need for targeted benchmark development and careful interface design. The study sets a foundation for domain-aware LLM evaluation methodologies in future forecasting research.