ForecastBench-Sim (FBSim) Benchmark

• ForecastBench-Sim is a simulated forecasting benchmark that uses structured Freeciv world reports to create immediately resolvable prediction tasks.
• It enables repeated rollouts and intervention analysis to study probabilistic reasoning and sample rare, disruptive outcomes in controlled environments.
• Empirical evaluations reveal its utility in diagnosing inverse scaling, highlighting differences between threshold-based and distributional forecast scoring.

ForecastBench-Sim (FBSim) is a simulated-world forecasting benchmark built on rollouts from the turn-based strategy game Freeciv. A forecasting task is defined by a fixed game snapshot, a structured “world report” describing the observable state at that snapshot, and questions about hidden future states that are resolved by continuing the simulation. Its central design aim is to complement real-world forecasting benchmarks by providing immediately resolvable probabilistic tasks, repeated rollouts from the same initial state, paired intervention worlds for conditional or causal questions, and dense samples of rare or disruptive outcomes in a controlled dynamic environment (Lee et al., 17 Jun 2026). A later study also used FBSim as a contamination-free setting for diagnosing metric-dependent inverse scaling in long-horizon distributional forecasting, indicating that the benchmark is useful not only for leaderboard construction but also for mechanistic analysis of forecasting behavior (Merrill et al., 21 May 2026).

1. Definition and motivating rationale

FBSim evaluates probabilistic reasoning in environments where the future is hidden but generable on demand. Forecasters receive a world report derived from a Freeciv state at a fixed snapshot turn, answer questions about later turns, and are scored after the simulator is rolled forward. This contrasts with real-world forecasting benchmarks, where outcomes resolve slowly, counterfactuals are difficult to score, and only one realized history is observed (Lee et al., 17 Jun 2026).

The simulation-backed design provides four capabilities that are difficult to obtain simultaneously in real-world settings. First, forecasts can be resolved as soon as the simulation advances, rather than waiting for external events. Second, the same starting world can be rerun many times or forked into matched intervention branches. Third, rare and disruptive outcomes can be sampled in bulk rather than awaited opportunistically. Fourth, evaluators control rules, horizons, and interventions, allowing systematic studies of horizon dependence, difficulty, calibration, and causal conditionals (Lee et al., 17 Jun 2026).

This suggests that FBSim occupies a distinctive methodological niche. It does not replace real-world forecasting benchmarks; rather, it supplies a substrate in which probabilistic forecasts can be evaluated under repeated, immediately resolvable, and experimentally manipulable conditions.

2. Simulated world, state representation, and world reports

The underlying world is Freeciv, an open-source strategy game modeled on the Civilization series. A world contains multiple civilizations with cities, population, technologies, treasury, territory, government type, units, production, and military strength, together with a map and a stream of events such as government changes, tech discoveries, city founding, and wonder construction. The benchmark formalizes this as a Markov decision process with state $s_t \in \mathcal{S}$, joint actions $a_t \in \mathcal{A}$, and transition $s_{t+1} = f(s_t, a_t, \varepsilon_t)$, where $\varepsilon_t$ represents stochasticity such as combat randomness and exploration (Lee et al., 17 Jun 2026).

FBSim conditions forecasts on snapshots around turn 60. Forecasters do not see the raw simulator state. Instead, they receive a world report designed to be interpretable by both humans and LLMs. The report includes territory maps as PNG images every 10 turns from turn 10 to turn 60; a civilization roster; per-civilization current-state fields at turn 60 such as score, government type, treasury, technologies known, number of cities, territory size, population, production, and units; time series sampled every 5 turns from turns 1 to 60 for treasury, population, tech count, city count, territory size, production, units, military units, scores, and rank; and a chronological event history listing key changes. Released reports are approximately 300 lines of structured text plus 6 maps per world (Lee et al., 17 Jun 2026).

The world report is the observable evidence on which forecasts are conditioned. The future simulator trajectory is hidden until scoring. This separation between observable report and latent continuation is fundamental to FBSim’s design: it creates forecasting tasks that are rich in state detail and path dependence without exposing the rollout used for evaluation.

3. Question families, horizons, and conditioning regimes

FBSim organizes questions into an $H0$ comprehension layer and $H1$–$H7$ forward-looking forecasting layers. $H0$ asks about the snapshot state itself and exists to verify that a forecaster can parse the report. It contains 562 binary questions and 165 continuous questions. These are scored with the same metrics as forecasting questions but are excluded from main forecast performance, because they diagnose report reading rather than future-state inference (Lee et al., 17 Jun 2026).

The forward-looking benchmark spans seven horizons, where $H_i$ corresponds to turn $t_0 + 30i$ for snapshot turn $a_t \in \mathcal{A}$0, yielding target turns 90, 120, 150, 180, 210, 240, and 270. Across these horizons, FBSim defines 4,655 binary outcome questions and 2,310 continuous quantity questions. Binary questions ask whether an event occurs by a future turn or whether a quantity exceeds a threshold. Continuous questions ask for future values such as city count, technology count, or treasury, and the forecast output is elicited as five quantiles,

$a_t \in \mathcal{A}$1

corresponding to p10, p25, p50, p75, and p90 (Lee et al., 17 Jun 2026).

FBSim distinguishes three conditioning regimes. The default release uses unconditional forecasting $a_t \in \mathcal{A}$2 given the report. The benchmark also describes an observational conditional regime $a_t \in \mathcal{A}$3, populated conceptually by co-occurrence frequencies across a corpus of simulated worlds, but this regime is not part of the current released artifact. Most distinctively, FBSim implements interventional or causal questions $a_t \in \mathcal{A}$4 by copying a snapshot state, mutating the savegame to impose a treatment, and rolling out both baseline and intervention worlds under the same AI policy. Pilot interventions include switching a civilization’s government to Republic and adding $a_t \in \mathcal{A}$5 gold to its treasury (Lee et al., 17 Jun 2026).

These design choices make FBSim more than a static question set. It is a family of forecasting regimes over a shared simulated world model, ranging from straightforward future-state prediction to explicitly forked counterfactual evaluation.

4. Scoring protocol and benchmark affordances

Binary questions are scored with the Brier score,

$a_t \in \mathcal{A}$6

where $a_t \in \mathcal{A}$7 is the realized outcome and $a_t \in \mathcal{A}$8 is the forecast probability. Continuous questions are scored with the Continuous Ranked Probability Score,

$a_t \in \mathcal{A}$9

where the predictive distribution $s_{t+1} = f(s_t, a_t, \varepsilon_t)$0 is approximated from the elicited five quantiles. To make continuous targets comparable across templates, FBSim normalizes CRPS by a fixed per-family range $s_{t+1} = f(s_t, a_t, \varepsilon_t)$1, using $s_{t+1} = f(s_t, a_t, \varepsilon_t)$2 for cities, $s_{t+1} = f(s_t, a_t, \varepsilon_t)$3 for technologies, and $s_{t+1} = f(s_t, a_t, \varepsilon_t)$4 for treasury (Lee et al., 17 Jun 2026).

Evaluation is aggregated at several levels. Per-question scores are averaged by horizon, pooled across $s_{t+1} = f(s_t, a_t, \varepsilon_t)$5–$s_{t+1} = f(s_t, a_t, \varepsilon_t)$6, and sliced by template family such as cities, technologies, or treasury. Calibration analysis uses reliability diagrams by binning predicted probabilities and comparing them with observed event frequencies. Because all future states are generated by simulation, every question is resolvable once the rollout is complete (Lee et al., 17 Jun 2026).

The scoring protocol is tightly coupled to the affordances of a simulated environment. Since the same initial world can be rerun, FBSim can generate repeated resolutions, intervention branches, and dense sets of extreme outcomes under controlled horizon schedules. Since binary and continuous questions are posed over the same world dynamics, the benchmark can compare threshold-based and distributional evaluation on closely matched forecasting problems. This later proved consequential: one study reported that single-threshold and tail-inclusive scoring can imply opposite capability rankings on FBSim outputs (Merrill et al., 21 May 2026).

5. Empirical results, validation, and diagnostic use

In its main unconditional release, FBSim reports evaluations over 4,655 binary questions with 30+ models and 2,310 continuous questions with 31 models. On a curated subset of nine models with complete coverage, mean pooled Brier over $s_{t+1} = f(s_t, a_t, \varepsilon_t)$7–$s_{t+1} = f(s_t, a_t, \varepsilon_t)$8 ranged from 0.220 for GPT‑5.1 to 0.313 for Gemini 2.5 Flash, while mean pooled normalized CRPS ranged from 0.283 for o3 to 0.590 for Gemini 2.5 Pro. Horizon length is a strong difficulty axis: mean Brier increases from 0.205 at $s_{t+1} = f(s_t, a_t, \varepsilon_t)$9 to 0.264 at $\varepsilon_t$0, peaking at approximately 0.287 at $\varepsilon_t$1, and normalized CRPS rises monotonically from 0.134 at $\varepsilon_t$2 to 0.639 at $\varepsilon_t$3, a roughly $\varepsilon_t$4 increase. Model rankings on FBSim also correlate with external capability and forecasting measures: pooled binary performance has Spearman correlation $\varepsilon_t$5 with ForecastBench Dataset Brier $\varepsilon_t$6 and $\varepsilon_t$7 with the Epoch Capabilities Index $\varepsilon_t$8; at $\varepsilon_t$9, the corresponding correlations are 0.630 with ForecastBench Dataset Brier and $H0$0 with the Epoch Capabilities Index (Lee et al., 17 Jun 2026).

The release includes an anonymized human pilot with 10 general-population participants answering 24 continuous questions on 2 worlds at horizons $H0$1, $H0$2, $H0$3, and $H0$4. Mean individual normalized CRPS was 0.171, crowd-mean normalized CRPS was 0.154, and a uniform-bin baseline achieved 0.152. Two intervention pilots, based on the Republic and $H0$5 gold treatments, showed positive average intervention gain across curated models. A placebo control for Opus 4.5 left Brier nearly unchanged under a null conditional, whereas a real Republic switch substantially changed difficulty, indicating that the intervention machinery is sensitive to causal content rather than merely to conditional phrasing (Lee et al., 17 Jun 2026).

A separate study used FBSim to investigate inverse scaling in long-horizon forecasting. On FBSim binary questions, capability measured by the Epoch Capabilities Index showed positive scaling under Brier, with reported $H0$6. On continuous questions scored by CRPS, the capability–performance relation was positive at short horizons but reversed by the longest horizon: reported correlation moved from $H0$7 at $H0$8 to $H0$9 at $H1$0 with 95% CI $H1$1 and $H1$2. The effect concentrated in “disruptable” templates such as treasury, territory, population, and city count, and a per-quantile decomposition indicated that failure was concentrated in the upper tail: p90 scaling shifted from $H1$3 at $H1$4 to $H1$5 at $H1$6, while lower-tail behavior remained comparatively stable (Merrill et al., 21 May 2026). This later use of FBSim underscores its diagnostic value: the benchmark can expose failure modes that are invisible under single-threshold evaluation.

6. Position in the benchmark landscape, limitations, and extension paths

FBSim sits at the intersection of real-world forecasting benchmarks, hermetic pastcasting environments, and simulation-backed reasoning tasks. ForecastBench evaluates live unresolved real-world questions and is explicitly designed as a dynamic, leakage-resistant benchmark of future events (Karger et al., 2024). Bench to the Future and BTF‑2 instead use frozen offline corpora to create reproducible pastcasting tasks with already known resolutions, emphasizing hermetic research environments and, in BTF‑2, decomposition of research versus judgment through full reasoning traces (FutureSearch et al., 11 Jun 2025, Liptay et al., 28 Apr 2026). PolyBench moves in a different direction, coupling point-in-time prediction-market snapshots with order-book state and news to score trading-oriented outputs using execution-based financial metrics such as Confidence-Weighted Return, APY, and Sharpe ratio (Cheng et al., 3 Apr 2026). Relative to these systems, FBSim’s distinctive contribution is the use of a rich simulated world in which hidden future states, paired interventions, and rare outcomes can all be generated from the same underlying substrate (Lee et al., 17 Jun 2026).

The benchmark also has explicit limitations. Freeciv is a stylized world, so transfer to real geopolitical or economic forecasting is incomplete. The structured report format may interact with model-specific context handling. The current human baseline is small and not strongly powered. The intervention artifacts are pilots rather than a fully scaled causal benchmark. Finally, the ruleset and all civilizations’ policies are fixed and rule-based rather than adaptive or human-like, which underrepresents strategic adaptation by opponents (Lee et al., 17 Jun 2026).

Several extension paths are already identified. These include additional simulated substrates such as social simulators or epidemiological models; richer intervention sets; multiple-rollout resolution in which probabilities are scored against empirical frequencies across many continuations; sequential forecasting with updates from multiple snapshot turns; counterfactual coherence checks across alternative conditionals; and larger human baselines with more varied expertise and incentives (Lee et al., 17 Jun 2026). A plausible implication is that FBSim is best understood not as a single benchmark instance but as a general experimental framework for forecasting under dynamic world states: its present Freeciv implementation is one substrate, and its broader significance lies in demonstrating how immediate resolution, repeated rollouts, and intervention-aware scoring can be unified in a single forecasting benchmark.