WorldReasoner: Temporal Forecasting

• WorldReasoner is a framework for temporally valid event forecasting that enforces historical evidence constraints to ensure genuine predictive inference.
• It evaluates forecasts along three axes—outcome quality, evidence quality, and reasoning quality—using metrics like accuracy, Brier score, and key-event recall.
• The system integrates forward and backward pipelines, combining simulated forecasting with causal graph construction to audit and calibrate predictions.

WorldReasoner is an evaluation framework for temporally valid event forecasting that tests whether language-model agents forecast resolved real-world events under historical information constraints rather than merely reproducing memorized outcomes or unsupported narratives. It presents each task with a resolved forecasting question, a simulated forecast date, and access only to evidence available before that date, then scores the submitted probability, cited evidence, and optional causal event graph along three complementary axes: outcome quality, evidence quality, and reasoning quality (Chi et al., 10 Jun 2026). In this formulation, forecasting is treated as a compound capability involving temporally bounded retrieval, source grounding, causal reconstruction, and probability calibration, not just final-answer correctness.

1. Historical replay and temporally valid forecasting

WorldReasoner defines each benchmark instance as

$q_i = (x_i, t_i^{o}, t_i^{sim}, t_i^{res}, y_i),$

where $x_i$ is the question text, $t_i^{o}$ is the earliest forecastable date, $t_i^{sim}$ is the simulated date shown to the agent, $t_i^{res}$ is the actual resolution date, and $y_i$ is the resolved outcome. This framing turns a resolved event into a reproducible prospective task: the event is already known to the evaluator, but not to the agent operating under the simulated historical state (Chi et al., 10 Jun 2026).

Temporal validity is enforced through the Temporal Gateway, which exposes only the subset of evidence dated before the simulated forecast date. If $A_i$ is the evidence repository for task $q_i$, and $\tau(a)$ is the timestamp of evidence item $a$, the accessible evidence is

$x_i$0

This mechanism is central to the benchmark’s claim that forecasting should be distinguished from retrospective answer recovery.

The agent submits a forecast

$x_i$1

where $x_i$2 is the predicted answer, $x_i$3 is the reported probability, $x_i$4 is the set of cited or accessed sources, $x_i$5 is the natural-language rationale, and $x_i$6 is the forecast-time causal graph if graph tools are enabled. The framework is explicitly designed to detect failure modes that a single accuracy number hides: memorized knowledge masquerading as forecasting, hallucinated grounding, unsupported causal stories, and grounded but uncalibrated forecasts.

2. Three-axis evaluation scheme

WorldReasoner formalizes evaluation as

$x_i$7

thereby separating correctness of the forecasted outcome from correctness of the evidence base and correctness of the reconstructed causal structure (Chi et al., 10 Jun 2026).

Axis	Object scored	Main metrics
Outcome quality	Predicted answer and probability	Accuracy, Brier score, Log score
Evidence quality	Cited or accessed sources	Source precision
Reasoning quality	Forecast graph vs hindsight graph	Key-event recall

Outcome quality uses three standard metrics. Accuracy is

$x_i$8

For binary questions, $x_i$9; for multiple-choice questions, the predicted option is the one with highest assigned confidence. Quantity questions count as correct if the predicted value is within 10% of the resolved value, and timeframe questions require an exact normalized match. Calibration is measured with Brier score,

$t_i^{o}$0

and Log score,

$t_i^{o}$1

Evidence quality is measured by source precision,

$t_i^{o}$2

with $t_i^{o}$3 if $t_i^{o}$4. The benchmark uses precision rather than recall because the hindsight set is intended to be exhaustive while the agent’s source budget is limited. The paper notes an important caveat: source overlap is computed by exact article identifier match, so equivalent evidence hosted elsewhere may be undercounted.

Reasoning quality is measured by key-event recall against hindsight graphs. Let $t_i^{o}$5 be the hindsight reference graph, and let $t_i^{o}$6 denote the top-$t_i^{o}$7 hindsight events ranked by impact score $t_i^{o}$8, with $t_i^{o}$9. Then

$t_i^{sim}$0

A match requires both text similarity and date proximity: $t_i^{sim}$1 with

$t_i^{sim}$2

and

$t_i^{sim}$3

Each hindsight event is matched to at most one forecast event by greedy assignment. This metric is intentionally strict: it credits a forecast graph only when it reconstructs the most causally important hindsight events with both textual and temporal alignment.

3. Agentic benchmark construction and hindsight graph generation

WorldReasoner is built through a forward pipeline and a backward pipeline. The forward pipeline ingests news streams and prediction markets. An LLM-based question generator proposes forecasting questions from article clusters, while a parser converts market metadata into the same schema. A quality-assurance module filters candidate questions for clarity, measurability, answerability, and temporal validity before admission to the benchmark (Chi et al., 10 Jun 2026).

The backward pipeline constructs hindsight references after question resolution. In the first stage, a HindsightAgent retrieves post-resolution articles and synthesizes a causal explanation of how the outcome came about, including key events and their temporal order. In the second stage, a GraphBuilderAgent converts that explanation into a structured causal graph using dedicated tools such as ProposeSubgraphTool, RecordOutcomeImpactTool, and correction tools like delete and revise operations for self-correction.

Hindsight graphs must satisfy explicit quality thresholds. The graph must have a minimum causal chain length of at least 3 hops, a minimum event count of at least 10 nodes, a minimum of at least 20 distinct supporting articles, at least 1 directed edge, edge confidence $t_i^{sim}$4, and edge strength $t_i^{sim}$5. If thresholds are not met, the system iteratively adds intermediate causes and retries; graphs that still fail are excluded from evaluation.

The hindsight graph is defined as

$t_i^{sim}$6

where $t_i^{sim}$7 are event nodes, $t_i^{sim}$8 are causal edges, and $t_i^{sim}$9 are outcome-impact records. Event nodes store text, date, event type, source articles, and review status. Causal edges store relation type, strength $t_i^{res}$0, confidence $t_i^{res}$1, reasoning text, and supporting evidence articles. Outcome-impact records store impact direction, impact magnitude $t_i^{res}$2, and confidence $t_i^{res}$3. Event importance is determined by the product $t_i^{res}$4, which is used to select the key events underlying KER.

4. Corpus composition and controlled agent settings

The benchmark contains 345 resolved questions, 14,141 collected articles, 8,087 extracted events, and 345 questions with hindsight graphs, spanning 10 domains. Of the questions, 248 are news-derived and 97 are Polymarket questions (Chi et al., 10 Jun 2026).

Question types are distributed as 69.0% binary (238), 13.3% multiple choice, 11.6% quantity, and 6.1% timeframe. The benchmark is concentrated in Politics, Culture, and Health, which together account for 58% of the questions. Estimated forecast horizons are 142 questions at $t_i^{res}$5 days, 32 at $t_i^{res}$6 days, and 171 at $t_i^{res}$7 days, with median horizon 86 days and mean horizon 174 days.

WorldReasoner evaluates agents under six controlled settings:

1. Vanilla LLM: parametric knowledge only, with no retrieval or graph tools.
2. Causal Simulation: internal causal graph reasoning, but no retrieval.
3. Search-Enabled: simulated-date retrieval, but no graph construction.
4. Search-Enabled Graph (SE Graph): retrieval plus causal graph construction.
5. Near-Resolution Topline: the Temporal Gateway is set to one day before resolution.
6. Real-Time Agent: live web access, used as an upper bound but not as a reproducible historical setting.

A central evaluation control is contamination filtering. A model-question pair is excluded if the question’s estimated start date precedes the model’s knowledge cutoff. This is meant to prevent parametric leakage from being mistaken for forecasting ability.

5. Empirical findings

The benchmark’s strongest aggregate pattern is that temporally valid retrieval is the strongest driver of accuracy. Weighted-average accuracies reported in the paper are 58.7% for Vanilla LLM, 56.6% for Causal Simulation, 68.8% for Search-Enabled, 64.4% for SE Graph, 74.7% for Near-Resolution, and 88.8% for Real-Time Agent (Chi et al., 10 Jun 2026). Internal causal simulation by itself does not improve over Vanilla, whereas retrieval at the correct historical date yields a large boost. Accuracy also rises as the simulated date moves later in the forecasting window.

The results also show that causal graph construction improves structured reasoning signals without necessarily improving final outcomes. SE Graph is the first setting that produces forecast graphs, and its weighted-average key-event recall is 9.8%. Yet SE Graph’s accuracy (64.4%) is lower than Search-Enabled accuracy (68.8%). The framework therefore separates a model’s ability to recover hindsight-important events from its ability to translate that recovery into a better forecast probability.

Conditioning grounding metrics on correctness produces a similarly mixed picture. In Causal Simulation, correct and incorrect forecasts have nearly the same KER: 5.1% versus 4.7%. In Search-Enabled, the SrcP gap is small: 63.7% for correct forecasts versus 61.5% for incorrect forecasts. In SE Graph, the separation is larger: correct forecasts have 10.8% KER versus 8.0% for incorrect forecasts, and 62.6% SrcP versus 48.0%. This indicates that grounding matters most when retrieval and graph reasoning are combined, but the coupling remains imperfect.

The paper identifies the principal remaining bottleneck as conversion of grounded evidence into calibrated probabilities. Even when agents recover relevant sources and key events, they may still fail to map that information into a proper probability estimate. A cited example is GPT-4o, which has relatively low SE Graph accuracy but nontrivial KER at Near-Resolution. This suggests that causal recovery and probabilistic calibration remain separable competencies.

Contamination controls substantially affect measured performance. For GPT-5.4 Vanilla, unfiltered accuracy is 69.2%, while filtered accuracy is 52.7%. The paper also states that DeepSeek V4 drops substantially after filtering. A plausible implication is that forecasting benchmarks without both run-time temporal gating and knowledge-cutoff filtering risk conflating genuine prospective inference with parametric recall.

6. Methodological significance, controversies, and limitations

WorldReasoner’s principal methodological claim is that forecasting evaluation should be multi-dimensional. Final-answer accuracy alone is insufficient because a correct forecast can still be produced by leaked parametric knowledge, fabricated or irrelevant evidence, or a causal story that does not match hindsight. Conversely, an agent may retrieve relevant sources and identify hindsight-important events yet still fail on probability calibration (Chi et al., 10 Jun 2026).

A common misconception addressed by the framework is that stronger reasoning structure should automatically increase final accuracy. The Search-Enabled versus SE Graph comparison argues against that assumption. Another is that “forecasting” can be established from a correct binary or multiple-choice outcome without constraining evidence access; the benchmark’s Temporal Gateway and contamination filtering are designed precisely to contest that interpretation.

The paper is explicit about several limitations. It notes coverage bias, since news-based benchmarks overrepresent highly covered public topics and underrepresent less-covered domains. It notes resolution ambiguity, because some questions have subjective or institution-dependent resolution criteria. It notes time granularity, since day-level gating may still permit intra-day leakage. It further states that knowledge cutoffs are imperfect proxies for actual pretraining exposure, that reference graphs are approximate rather than complete causal ground truth, that exact source matching may undercount relevant evidence expressed in different articles, and that human annotation covers only a subset, leaving graph-quality metrics approximate.

These caveats do not negate the framework’s contribution. Rather, they locate WorldReasoner within a diagnostic tradition: it is intended to decompose forecasting performance into evidence access, causal recovery, and calibration, not to claim that hindsight graphs or source-overlap metrics provide exhaustive ground truth.

7. Position within world-reasoning research

WorldReasoner belongs to a broader line of research that asks whether model behavior is grounded in valid interaction with a world-like substrate, but it addresses this question in a specifically forecasting-oriented form. Whereas SituatedThinker trains LLMs to combine internal reasoning with external interfaces such as retrieval, code execution, KB queries, table access, and text-game feedback, WorldReasoner evaluates whether temporally bounded retrieval and causal grounding actually support valid real-world forecasts rather than merely better answers (Liu et al., 25 May 2025). This suggests a complementary relation: SituatedThinker studies how to learn grounded interaction policies, while WorldReasoner studies how to audit grounded forecasting behavior.

Its scope also differs from simulation-oriented work. World Reasoning Arena evaluates world models as internal simulators along Action Simulation Fidelity, Long-horizon Forecast, and Simulative Reasoning and Planning, with an emphasis on counterfactual rollouts, physically plausible continuation, and planning utility (Team et al., 26 Mar 2026). Einstein World Models proposes an LLM-centered reasoning architecture in which a callable world-module inserts short visual-temporal rollouts into the reasoning trace as inspectable hypotheses (Nwadike et al., 25 Jun 2026). By contrast, WorldReasoner does not evaluate simulated physical rollouts; it evaluates whether an agent forecasting real events used the right evidence at the right historical time and recovered the right key events.

A broader synthesis is that recent “world reasoning” research spans at least three substrates: external information interfaces, internal or external simulation modules, and temporally bounded evidence-grounded forecasting. WorldReasoner’s distinctive contribution is to make the third substrate measurable. It separates knowing the answer, finding the right evidence, recovering the right causal events, and assigning a calibrated probability, and thereby provides an explicit test of whether a language-model agent is genuinely forecasting rather than retrospectively reconstructing the past (Chi et al., 10 Jun 2026).