ForecastBench-Sim: Simulated Forecasting Benchmark

Updated 6 July 2026

The paper demonstrates how ForecastBench-Sim enables immediate scoring of probabilistic forecasts by leveraging simulated Freeciv game rollouts.
It employs fixed world reports and diverse forecast question families—including binary, continuous, and interventional queries—to test calibration and causal inference.
Validation results reveal horizon-dependent performance degradation and effective separation of report comprehension from forward forecasting accuracy.

Searching arXiv for the benchmark paper and closely related forecasting benchmark work to ground the article with current citations. ForecastBench-Sim is a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. In each task, forecasters receive a fixed world report—a structured snapshot of the current simulated state—and answer questions about hidden future states; the benchmark then continues the simulation and scores the forecasts against the realized future. Because the world is simulated, the same setup can generate continuous or binary forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. The benchmark is intended to complement real-world forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic reasoning under dynamic world states (Lee et al., 17 Jun 2026).

1. Motivation and benchmark rationale

ForecastBench-Sim is motivated by structural limitations in real-world forecasting evaluation. Real-world benchmarks are valuable, but outcomes can take weeks or months to resolve, rare events are sparsely observed, counterfactual and conditional questions are usually unscorable because only one history actually happens, and model evaluation can be confounded by limited resolution and delayed feedback. ForecastBench-Sim addresses these constraints by moving the forecasting problem into a simulated world where the evaluator can choose the snapshot shown to forecasters, continue the world into the future, fork the world into interventions, and densely sample rare or disruptive outcomes (Lee et al., 17 Jun 2026).

The benchmark is explicitly framed as a complement rather than a replacement. That distinction is central: the simulated setting provides control, repeatability, and immediate resolution, but the paper also states that Freeciv worlds are simplified. A common misconception is therefore to treat ForecastBench-Sim as a synthetic substitute for real-world forecasting. The benchmark’s stated role is narrower and more methodological: it isolates probabilistic reasoning under dynamic world states in a setting where futures are known, controllable, and scorable.

This design also changes what can be measured. In a simulated world, unconditional forecasting, observational conditional forecasting, interventional forecasting, and rare-event evaluation can all be implemented within a single interface. A plausible implication is that the benchmark is useful not only for comparing raw forecasting quality, but also for studying calibration, causal updating, and horizon-dependent degradation under tightly controlled conditions.

2. Simulation substrate and fixed world reports

ForecastBench-Sim is built on Freeciv rollouts using the CivRealm infrastructure for Freeciv-based AI environments. The simulated world includes multiple civilizations, cities, technology progress, territory, treasuries, diplomacy, conflicts, and other evolving state variables. The forecast origin is fixed at turn 60, and future turns are hidden until resolution. Freeciv is presented as a suitable substrate because it is inspectable, multi-agent, rich in interacting dynamics, and contains a wide range of forecastable quantities (Lee et al., 17 Jun 2026).

The benchmark’s core input is the fixed world report. This is a pre-generated, immutable snapshot of the simulated game at the forecast origin. It contains the civilization roster, current state at turn 60, time series of variables sampled across prior turns, event history, and territory maps over time. The report is the only world-specific information shown to the forecaster; the forecaster sees evidence about the world, not the simulator state itself. The paper describes this as the common input surface for both humans and models, with structured text, tables, histories, and sometimes map images.

The appendix example for game seed seed5 illustrates the intended report structure: a turn-60 territory map, current-state tables for each civilization, histories of treasury, population, and technologies, and chronological events such as government changes and wonder completion. This matters because the benchmark is not framed as direct simulator-state prediction. Instead, it emulates a standard forecasting situation in which the agent must infer future states from a bounded evidential surface.

3. Question families, horizons, and benchmark composition

ForecastBench-Sim supports binary, continuous, conditional, and causal or intervention questions. Binary questions ask for probability forecasts of yes-or-no events, such as whether a civilization’s treasury will exceed a threshold at a future turn or whether an event will happen by a specified horizon. Continuous questions ask for future values of variables such as city count, technology count, or treasury value, and the benchmark elicits five quantiles: p10, p25, p50, p75, and p90. Conditional questions distinguish observational conditional forecasts, $P(Y \mid X)$ , from interventional forecasts, $P(Y \mid do(X))$ . Causal or intervention questions are enabled by mutating the saved game state before rollout and then scoring against the outcome in the altered branch (Lee et al., 17 Jun 2026).

Time is organized into multiple horizons. H1–H7 are forward-looking questions, each corresponding to 30-turn increments past the turn-60 snapshot, covering turns 90 through 270. H0 questions are different: they concern the snapshot itself and function as report-comprehension checks rather than forecasting tasks. The paper emphasizes that H0 results are reported separately from forecasting scores. This separation prevents simple report-reading ability from being conflated with forward predictive performance.

The benchmark’s question counts are summarized as follows:

Question set	Horizons	Count
Binary	H1–H7	4,655
Continuous	H1–H7	2,310
H0 binary	snapshot	562
H0 continuous	snapshot	165

The paper further notes that the H1–H7 binary questions use 10 / 11 templates/worlds, the H1–H7 continuous questions use 6 / 11, the H0 binary questions use 10 / -, and the H0 continuous questions use 3 / 11. The release artifacts include a structured world report format, forecasting templates, question generation across horizons, rollouts for hidden futures, and archival materials such as H0 comprehension checks, human pilot data, cross-benchmark correlation analyses, full validation diagnostics, conditional and intervention diagnostics, and tail-risk examples.

4. Scoring rules, normalization, and controlled causal evaluation

The operational pipeline is straightforward: show the fixed report to the forecaster, collect probability forecasts, continue the simulation forward, reveal the hidden future, and score predictions against the realized future. Binary forecasts are scored with Brier score, and continuous forecasts are scored with CRPS from elicited quantiles. In the paper’s notation, for binary outcome $y \in \{0,1\}$ and predicted probability $p$ ,

$\text{Brier}(p,y) = (p-y)^2.$

For a predictive cumulative distribution $F$ and realized outcome $x$ ,

$\mathrm{CRPS}(F,x) = \int_{-\infty}^{\infty} \left(F(z) - \mathbf{1}\{z \ge x\}\right)^2 \, dz.$

Lower scores are better (Lee et al., 17 Jun 2026).

Continuous scores are normalized by fixed family-specific ranges so that different templates can be averaged on a common scale. The paper specifies ranges of 40 for cities, 60 for technologies, and 2000 for treasury. Accordingly, the reported continuous metric is normalized CRPS. This normalization is important because it avoids letting large-scale variables dominate pooled evaluations.

A distinctive feature of ForecastBench-Sim is that it operationalizes several forecasting regimes within the same benchmark. Unconditional forecasting predicts $P(Y)$ from a fixed report without intervention. Observational conditional forecasting predicts $P(Y \mid X)$ based on naturally occurring co-occurrence patterns in the corpus of simulated worlds. Interventional forecasting predicts $P(Y \mid do(X))$ 0 after the benchmark copies the savegame, applies an intervention, rolls out the modified world, and scores the forecast against that forked-world outcome. The paper highlights this as a regime that real-world forecasting benchmarks generally cannot score, because they observe only one realized history.

The simulation setting also enables direct study of rare and disruptive outcomes. Because large numbers of rollouts can be sampled, the benchmark can expose tail outcomes densely rather than waiting for them to occur naturally. The paper gives a concrete example in which treasury outcomes fall below the model’s stated p10 much more often than expected, reaching about 50% by H6–H7 in one analysis. That result is presented as evidence that the benchmark can surface undercoverage of downside outcomes and tail-risk calibration failures.

5. Validation results and empirical behavior

The paper reports several validation slices intended to show that ForecastBench-Sim behaves like an actual forecasting benchmark rather than merely a report-reading exercise. Across curated models, binary Brier at H1–H7 ranges roughly from 0.220 to 0.313, and normalized CRPS ranges roughly from 0.283 to 0.590. Performance worsens as horizon increases: mean Brier rises from 0.205 at H1 to 0.264 at H7, while mean normalized CRPS rises from 0.134 at H1 to 0.639 at H7. This horizon structure is interpreted as evidence that the benchmark contains genuine forecastable signal alongside increasing uncertainty over time (Lee et al., 17 Jun 2026).

The H0 comprehension checks provide an important control. Curated models perform very well on H0, with binary Brier below 0.032 and continuous normalized CRPS below 0.024. The intended interpretation is that forward-looking errors are not primarily due to inability to read the report. In other words, the benchmark appears to separate comprehension of the fixed evidential surface from forecasting beyond it.

ForecastBench-Sim also shows cross-benchmark associations. The paper reports that benchmark rankings correlate with the Epoch Capabilities Index and the real-world ForecastBench Dataset Brier. For pooled H1–H7 binary Brier, the correlation with ECI is $P(Y \mid do(X))$ 1, $P(Y \mid do(X))$ 2, and the correlation with ForecastBench Dataset Brier is $P(Y \mid do(X))$ 3, $P(Y \mid do(X))$ 4. The cleanest single-horizon comparison is H1, with $P(Y \mid do(X))$ 5 with ECI and $P(Y \mid do(X))$ 6 with ForecastBench Dataset Brier. The paper presents this as suggesting that the simulation benchmark tracks general forecasting capability while remaining distinct.

The anonymized human pilot is more limited and is described as a feasibility check rather than a strong human-versus-model comparison. It includes 10 participants, 24 continuous questions, 240 individual forecasts, 2 simulated-world reports, templates for city, technology, and treasury, and horizons H1, H3, H4, and H6. The mean individual normalized CRPS is 0.171, the crowd mean is 0.154, and the uniform-bin baseline is 0.152. Aggregation helps, but the small sample does not clearly beat the uninformed baseline. This guards against overstating what the human study establishes.

The conditional and intervention pilots provide more direct evidence that intervention semantics matter. The archived runs use two interventions: switching a civilization’s government to Republic and adding +500 gold to treasury. The paper reports positive intervention gains for the curated models on both pilots. It also describes a placebo test in which a null conditional leaves Brier nearly unchanged, whereas the real Republic conditional worsens performance sharply. The stated interpretation is that the observed effect is tied to intervention content rather than merely to the presence of conditional wording.

6. Position within the forecasting-benchmark landscape

ForecastBench-Sim occupies a specific position among recent forecasting benchmarks. Relative to real-world replay and pastcasting systems, its defining feature is not a historical corpus but direct control over world evolution. Bench to the Future uses “pastcasting,” in which a model forecasts from the perspective of a past date using a hermetic, repeatable offline corpus of roughly 20,000 web pages per question (FutureSearch et al., 11 Jun 2025). FutureSim instead replays real-world events chronologically, exposing agents to a daily-growing CCNews corpus over a roughly 90-day period and measuring open-ended adaptation as questions remain active and news arrives (Goel et al., 14 May 2026). ForecastBench-Sim differs from both by generating immediately resolvable futures through simulation rather than by reconstructing a bounded historical information state.

It also differs from benchmarks whose primary challenge is not simulated dynamics. TFRBench evaluates whether forecasting systems can explain forecasts through interpretable, factually grounded reasoning over trends, seasonality, cross-channel dependencies, and external events in time-series data (Ahamed et al., 7 Apr 2026). QuantSightBench focuses on continuous quantitative forecasting via prediction intervals and evaluates empirical coverage and interval sharpness over 1,000 forecasting questions (Qin et al., 17 Apr 2026). PolyBench evaluates forecasting and trading on timestamp-locked Polymarket snapshots with aligned news and historical CLOB state, emphasizing financially grounded execution and return-based metrics (Cheng et al., 3 Apr 2026). These benchmarks target reasoning, interval calibration, or market-grounded decision-making; ForecastBench-Sim targets controllable hidden futures, intervention forks, and dense rare-event evaluation inside a simulated multi-agent world.

A recurrent misconception is to read ForecastBench-Sim as merely a game benchmark. The paper’s framing is narrower and more methodological. The benchmark is meant to study calibrated forecasting under controlled conditions, conditional and causal updating, immediate resolution, rare-event and tail-risk behavior, horizon-dependent uncertainty, and structured dynamics in multi-agent worlds. Its contribution is therefore not domain realism in the ordinary empirical sense, but a reusable evaluation substrate in which the evaluator controls the future and can score regimes that are ordinarily inaccessible in naturalistic forecasting settings (Lee et al., 17 Jun 2026).

In that sense, ForecastBench-Sim can be understood as a controlled benchmark for probabilistic reasoning under dynamic world states. Its fixed report at turn 60, hidden future rollouts, proper scoring rules for binary and continuous forecasts, intervention-enabled forked worlds, and diagnostic validation suite make it a distinct entry in the emerging literature on forecasting benchmarks. The benchmark’s significance lies less in replacing real-world tests than in making previously hard-to-evaluate forecasting problems—especially causal and tail-risk questions—tractable, repeatable, and immediately resolvable.