Prophet Arena: LLM Forecasting Benchmark

• Prophet Arena is a live, modular benchmark that evaluates LLM forecasting by decomposing real-world events into multi-stage, probabilistic predictions.
• It separates the tasks of retrieval, context construction, and forecasting to isolate predictive reasoning, calibration, and evidence aggregation.
• The framework uses metrics like Brier score, ECE, and average returns to assess performance, highlighting strengths in calibration and challenges in rapid information updating.

Searching arXiv for the benchmark and related naming clarifications. Prophet Arena is a live, modular benchmark and experimental framework for studying LLM-as-a-Prophet: the use of LLMs to forecast real-world future events. It is designed to move beyond static, accuracy-only evaluations by centering probabilistic forecasting, repeated predictions over time, stage-wise diagnosis, and market-grounded evaluation. In the benchmark’s own framing, Prophet Arena emphasizes predictive intelligence across reasoning, calibration, evidence aggregation, and economic value, while using a Market Baseline from prediction markets to contextualize both difficulty and skill (Yang et al., 20 Oct 2025).

1. Definition and conceptual scope

Prophet Arena was introduced in "LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena" as a benchmark that continuously collects live forecasting tasks and decomposes each task into distinct pipeline stages. Its purpose is to support controlled and large-scale experimentation on predictive intelligence, rather than only leaderboard-style comparison of end metrics (Yang et al., 20 Oct 2025).

Several properties distinguish it from earlier forecasting benchmarks. The framework is live, because it collects unresolved future events; probabilistic, because models output probabilities rather than only categorical labels; multi-horizon, because each task is forecast multiple times before resolution; modular, because retrieval, context construction, forecasting, and evaluation are explicitly separated; and market-grounded, because market-implied probabilities serve as a built-in baseline and market return is part of evaluation (Yang et al., 20 Oct 2025).

This design reflects a particular conception of forecasting as a testbed for intelligence. The benchmark’s rationale is that live future events are contamination-resistant, reducing the risk that a model has already seen the answers during training. The associated paper argues that forecasting can probe reasoning, calibration, and evidence use in a setting where outcomes are not yet known at inference time (Yang et al., 20 Oct 2025).

A recurring theme is that Prophet Arena is not only a scoreboard. It is also an instrument for diagnosing failure modes. Retrieval is held fixed across models, forecast times are scheduled systematically, and the evaluation includes calibration, logical consistency, source sensitivity, and economic return. This suggests a deliberate separation between forecasting competence and pipeline confounds such as variable search quality or inconsistent timing (Yang et al., 20 Oct 2025).

2. Task collection and benchmark construction

Prophet Arena continuously sources unresolved events from Kalshi, filtering by popularity, diversity, and recurrence formats. The reported domains include politics, economics/finance, sports, entertainment, science, and climate/weather. Operationally, it retrieves 20 unresolved events daily at 12 AM UTC (Yang et al., 20 Oct 2025).

For the evaluation reported in the paper, the authors apply a cutoff date of October 11, 2025 and exclude predictions made within three hours of resolution. The resulting dataset contains 1,367 resolved events and 72,136 markets (Yang et al., 20 Oct 2025). Events are high-level questions that may contain one or more binary markets. These markets may be mutually exclusive, as in championship outcomes, or non-exclusive, as in multiple meetings involving the same political figure (Yang et al., 20 Oct 2025).

Each event $E_i$ resolves at time $\tau_i$, and each market $M_{ij}$ resolves to a binary outcome $o_{ij} \in \{0,1\}$. Prophet Arena defines a Market Baseline forecaster by setting $p_{mb} = q_{ij}$, where $q_{ij}$ is the market-implied probability obtained from normalized Yes/No prices. The normalization is necessary because fees may prevent raw prices from summing to one (Yang et al., 20 Oct 2025).

The benchmark also includes a separate subset of 100 past events from before model knowledge cutoffs. That subset is used to probe knowledge internalization and recall, rather than prospective forecasting alone. This creates an auxiliary axis of analysis: whether a model’s internal memory of past events is accurate enough to support future-event reasoning (Yang et al., 20 Oct 2025).

3. Pipeline architecture and forecasting protocol

Prophet Arena organizes forecasting into three main stages: event and market extraction, prediction context construction, and probabilistic forecasting and evaluation (Yang et al., 20 Oct 2025). The extraction stage supplies live events and the Market Baseline. The context-construction stage fixes what information the model sees at each forecasting time. The final stage records probabilities and evaluates them after market resolution.

A central design choice is the use of a halving schedule for forecast times. For event $E_i$, Prophet Arena defines multiple times $T_i = \{t^{(1)}, t^{(2)}, \dots\}$ before resolution using

$t^{(k+1)} = \frac{t^{(k)} + \tau_i}{2},$

with a minimum gap $\delta_{min}$ enforced (Yang et al., 20 Oct 2025). This schedule concentrates forecasts as resolution approaches while preserving temporal spacing. It is one reason the framework is described as multi-horizon rather than single-shot.

At each forecast time, all models receive the same curated context $\tau_i$0. This includes news sources retrieved by an LLM searcher and market snapshots with last_price, yes_ask, no_ask, and implied probabilities $\tau_i$1. In the reported experiments, the search component is GPT-4o with web access. The searcher is described as searcher-agnostic and pluggable, so retrieval can in principle be swapped without changing the benchmark protocol (Yang et al., 20 Oct 2025).

Given $\tau_i$2, each model outputs a probability $\tau_i$3 that the market resolves Yes, along with a brief rationale. Quantitative scoring uses only the probabilities. The rationale is used in separate qualitative and judge-based analyses (Yang et al., 20 Oct 2025).

This modular decomposition has methodological significance. Because retrieval is fixed across models, measured differences are intended to reflect forecasting, reasoning, and calibration rather than variation in search skill. A plausible implication is that Prophet Arena treats retrieval as an experimental control variable, not as the target capability under primary comparison (Yang et al., 20 Oct 2025).

4. Evaluation metrics and reported empirical results

Prophet Arena evaluates forecasting along three dimensions: proper scoring rules, calibration, and economic value (Yang et al., 20 Oct 2025). The main event-level scoring rule is the Brier score

$\tau_i$4

with overall score

$\tau_i$5

A pooled binary Brier formulation is also reported in the appendix (Yang et al., 20 Oct 2025).

Calibration is reported using empirical ECE with $\tau_i$6 bins. With $\tau_i$7 the set of predictions in bin $\tau_i$8, $\tau_i$9,

$M_{ij}$0

and

$M_{ij}$1

Economic value is measured through Average Return and a more general CRRA utility formulation. For risk-neutral evaluation, the protocol allocates \$1 budget per market and buys Yes or No depending on whether the model’s forecast dominates the market-implied alternative (Yang et al., 20 Oct 2025).

The benchmark also defines edges relative to the market:

$M_{ij}$2

These quantify the model’s implied advantage over market probabilities. A Sharpe-style risk-adjusted comparison is additionally reported in the appendix (Yang et al., 20 Oct 2025).

The principal quantitative findings on 1,367 events place most Brier scores in $M_{ij}$3, with random guess $M_{ij}$4. The Market Baseline achieves 0.187 ± 0.006. Strong models often achieve ECE $M_{ij}$5, while the Market Baseline has ECE = 0.069. Average returns are reported as generally below break-even (<1) for most models, with wide confidence intervals due to high variance (Yang et al., 20 Oct 2025).

Representative results include GPT-5$M_{ij}$6 with Brier 0.184 (±0.006), ECE 0.042, and Avg Return 0.943 (±0.042); Grok-4$M_{ij}$7 with Brier 0.189 (±0.005), ECE 0.043, and Avg Return 0.864 (±0.052); Claude Sonnet 4$M_{ij}$8 with Brier 0.194 (±0.006), ECE 0.041, and Avg Return 0.909 (±0.101); and Llama-4-Scout with Brier 0.219 (±0.008), ECE 0.060, and Avg Return 0.805 (±0.040) (Yang et al., 20 Oct 2025).

Temporal analysis adds an important qualification. The paper reports that LLMs sometimes outperform the market baseline at long horizons, but markets surpass LLMs near resolution because they incorporate breaking information faster. This is the stated reason for excluding predictions within three hours of market close (Yang et al., 20 Oct 2025).

5. Diagnostic findings and observed bottlenecks

The benchmark’s central substantive conclusion is not that forecasting performance is uniformly strong, but that performance is unevenly distributed across sub-capabilities. One reported strength is calibration: strong models show small ECE, and reliability diagrams indicate that top models are especially well calibrated in extreme bins such as 0–0.1 and 0.9–1.0 (Yang et al., 20 Oct 2025).

At the same time, the paper identifies several bottlenecks. One is conservative probability assignment relative to the market. Scatter analyses show that most LLMs produce more conservative probabilities than market-implied values, especially in high-confidence regions. GPT-5 and Grok-4 track market probabilities more closely than weaker models, but remain comparatively cautious (Yang et al., 20 Oct 2025).

A second bottleneck is temporal latency in information aggregation. Near event resolution, markets absorb new information more quickly than the models in the benchmark pipeline. This suggests that Prophet Arena is not only testing abstract reasoning but also the ability of systems to update under rapidly changing evidence streams (Yang et al., 20 Oct 2025).

A third bottleneck is knowledge internalization. The paper reports that recall quality varies by category: entertainment is recalled relatively reliably, whereas climate/weather and politics are often mis-recalled or date-misaligned. The example of Olivia Rodrigo’s “Vampire” is used to illustrate correct content with incorrect timing. This suggests that latent world knowledge may be semantically plausible while still being temporally unreliable for forecasting use (Yang et al., 20 Oct 2025).

A fourth bottleneck is retrieval quality. The case studies report that adding sources can help or hurt. In Bitcoin-related examples, poor-quality forecast bots and fragmented methodologies degraded forecast quality, whereas coherent crypto-specialized sources improved it. This indicates that even when retrieval is fixed across models, the benchmark can still reveal the extent to which downstream reasoning depends on source quality (Yang et al., 20 Oct 2025).

Finally, reasoning synthesis and mapping to probabilities are reported as the main differentiators among models. LLM-as-a-judge evaluations show near-parity on source use, evidence extraction, and uncertainty analysis, but much larger differences in synthesis and conversion of evidence into calibrated probabilities. This suggests that Prophet Arena measures something more specific than raw retrieval breadth: it measures how evidence is operationalized into probabilistic judgment (Yang et al., 20 Oct 2025).

6. Naming, related benchmarks, limitations, and future directions

The name Prophet Arena is used explicitly in (Yang et al., 20 Oct 2025), but adjacent literature introduces a potential source of confusion. "Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets" describes a different benchmark in which autonomous models trade on live prediction markets with real capital. That paper states that “Prophet Arena” appears to be an informal alias for the same initiative only in that context, and that the official name there is Prediction Arena (Zhang et al., 28 Mar 2026). Accordingly, Prophet Arena in the forecasting-benchmark sense and Prediction Arena in the autonomous-trading sense should be treated as distinct unless a source explicitly equates them.

Prophet Arena also differs from earlier forecasting benchmarks such as MIRAI, ForecastBench, FutureBench, and FutureX by combining live events, probabilistic forecasts, repeated predictions over time, a modular pipeline, and return metrics within a single framework (Yang et al., 20 Oct 2025). A plausible implication is that it occupies a hybrid position between classical forecast evaluation and systems evaluation for LLM-based decision pipelines.

Several limitations are stated directly. Near-resolution predictions mostly measure retrieval latency rather than core reasoning. Fixed retrieval isolates forecasting skill but does not evaluate retrieval capability itself. Source quality can confound outcomes, especially for fine-grained, date-stamped events. Market-implied probabilities require fee normalization. Returns are simulated, not executed, so transaction costs and slippage are not modeled (Yang et al., 20 Oct 2025).

The paper’s future directions follow directly from these limitations: improve retrieval and source curation, strengthen knowledge internalization fidelity, accelerate information aggregation near resolution, and focus on the reasoning-synthesis step that maps evidence into probabilities. Prophet Arena is therefore presented not as a closed benchmark, but as an extensible platform for controlled experimentation on the predictive intelligence of LLMs (Yang et al., 20 Oct 2025).