WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning

Abstract: Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether agents genuinely forecast requires more than final-answer accuracy: a model may be correct by recalling memorized training facts, citing fabricated evidence, or producing an unsupported causal story. We present WorldReasoner, an evaluation framework for temporally valid event forecasting. Each task gives an agent a resolved forecasting question, a simulated forecast date, and access only to evidence available before that date; after resolution, the framework scores the submitted probability, cited evidence, and optional causal event graph. WorldReasoner reports three complementary axes: outcome quality against resolved answers, evidence quality over cited sources, and reasoning quality against post-resolution hindsight graphs. The benchmark is built by an agentic construction pipeline that generates forecasting questions, collects time-stamped evidence, and builds hindsight reference graphs at scale, yielding 345 resolved tasks derived from 14,141 articles with graphs covering 8,087 extracted events. Across six controlled agent settings, temporally valid retrieval is the strongest driver of outcome accuracy; causal graph construction improves key-event recovery; and correct graph-enabled forecasts are more strongly grounded in key events and relevant sources, yet agents still struggle to convert grounded evidence into calibrated probabilities.

• The paper introduces a three-axis evaluation framework decomposing outcome, evidence, and reasoning quality for LLM forecast agents.
• It employs temporal gating and causal graph construction to ensure forecasts derive from valid, timestamped evidence.
• Results reveal that accurate retrieval improves predictions while aligning causal reasoning with calibrated probabilities remains challenging.

Formal Summary of "WorldReasoner: Evaluating Whether LLM Agents Forecast Events with Valid Reasoning" (2606.11816)

Problem Statement and Motivation

Reliable forecasting of real-world events by LLM agents requires reasoning under temporal uncertainty, navigating incomplete and time-bounded evidence. The core evaluation challenge is to distinguish genuine forecasting ability—reasoned, prospective judgment—from mere recall of training-set facts or post-hoc rationalization. Existing benchmarks are limited either by temporal leakage (static resolved datasets) or by operational impracticalities (slow-moving live evaluation). Furthermore, typical evaluations focus solely on outcome accuracy, neglecting critical dimensions such as evidence validity and causal reasoning.

Framework and Contributions

WorldReasoner introduces a temporally controlled and reproducible evaluation framework for assessing LLM agents on event forecasting. Each benchmark instance is formulated as a resolved, time-stamped question, with agent evidence access strictly gated by a "Temporal Gateway" that enforces historical information flow at a simulated forecast date. Submissions comprise a forecast probability, cited sources, and (optionally) a causal event graph.

Key contributions:

1. Three-axis Evaluation: Each agent submission is decomposed into outcome quality (accuracy, Brier/log scores), evidence quality (source precision on time-validity and relevance), and reasoning quality (key-event recall in causal graph construction). This avoids collapse into a monolithic leaderboard score and enables diagnosis of the sources of success and failure.
2. Agentic Benchmark Construction: An automated pipeline generates forecasting questions by ingesting prediction markets and news streams, collecting timestamped evidence, and building post-resolution "hindsight" event graphs. This supports scalable, reproducible creation and expansion of the benchmark.
3. Comprehensive Experimental Analysis: Six agent settings are evaluated, ranging from parametric LLM-only (Vanilla), causal simulation, retrieval-augmented, and tool-augmented agents, up to real-time (non-reproducible) agents with live web access. Each experimental configuration rigorously enforces or filters for knowledge cutoffs to preclude post-hoc contamination.

Experimental Results

Aggregate Findings

• Retrieval is Necessary but Not Sufficient: Temporally valid retrieval is the primary driver of outcome accuracy. Search-enabled agents increase performance (e.g., 68.8% accuracy, contamination-filtered), with accuracy further rising as forecast date approaches event resolution.
• Causal Graph Construction: Explicit causal graph building raises key-event recall (e.g., up to 20.4% KER), reflecting improved causal event identification, but does not reliably improve answer selection. This implies that correct identification of causal events is a necessary precursor but not itself sufficient for calibrated probabilistic forecasting.
• Evidence and Reasoning Decoupling: Agents frequently retrieve relevant evidence and reconstruct causal chains but fail to map grounded evidence to well-calibrated, outcome-aligned probabilities. For example, Gemini 3. Pro shows pronounced gaps in source precision between correct and incorrect forecasts, underscoring that evidence collection and reasoning are only weakly coupled to calibrated answer derivation.

Temporal and Model Heterogeneity

• Temporal Sensitivity: Forecast accuracy robustly increases with shorter forecast horizons, supporting true evaluation difficulty at earlier simulation times (e.g., from ~62% to ~80% accuracy).
• Knowledge Cutoff Filtering Is Critical: Without filtering by knowledge cutoff date, parametric LLM-only benchmarks overstate model forecasting ability. E.g., removing contamination drops GPT-5.4 Vanilla accuracy from 69.2% to 52.7%, clarifying the necessity of strict temporal boundaries.
• Model Variability: Per-model differences are observed in how correctness aligns with evidence retrieval and reasoning, motivating the reporting of decomposed axis scores.

Theoretical and Practical Implications

WorldReasoner resolves two persistent obstacles in LLM event forecasting evaluation: temporal leakage and the diagnostic opacity of scalar accuracy metrics. By making all evidence and reasoning temporally and structurally explicit and decomposing agent performance, the framework enables future research to:

• Diagnose model failure modes with respect to evidence acquisition, causal identification, and probability calibration.
• Incentivize improvements not just in answer accuracy, but in claims verification and causal validity.
• Scale reproducible benchmarks over time and across model generations, supporting robust longitudinal model comparisons.
• Support the development of agents and tools where causal reasoning and evidence grounding are as rigorously measurable as final predictive skill.

The persistent bottleneck, as highlighted by the results, is the mapping from grounded evidence and causal chains to calibrated forecast probabilities. This suggests avenues for new TTE architectures, retrieval-reasoning-calibration pipelines, and attention to probabilistic inference in LLM agents. Moreover, the modular construction and three-pronged evaluation can be adopted or extended for domains requiring high assurance and auditability, such as scientific discovery or risk assessment.

Limitations

• Benchmark Coverage: News-based artifact construction reflects media coverage bias; underrepresented domains remain.
• Resolution Granularity: Temporal granularity is controlled by day-level evidence gating, which cannot resolve intra-day leakage.
• Reference Quality: Human-validated hindsight event graphs are approximate references, not exhaustive causal ground truths.
• Generalizability: Evaluation is constrained to simulated or recent historical horizons, not "true" prospective future forecasting.

Conclusion

WorldReasoner provides a robust research infrastructure for evaluating LLM event-forecasting agents on temporally grounded, evidence- and reasoning-sensitive axes. It demonstrates that high forecast accuracy is inseparable from the careful design of evaluation protocols that enforce temporal validity and transparent reasoning. Future work grounded on this framework promises to push beyond superficial benchmarking, toward LLM agents capable of truly justified, causally sound, and well-calibrated predictive reasoning.