Prediction Arena Evaluation Framework

• Prediction Arena is a modular benchmark framework for predictive agents, integrating live market data, incentive-compatible scoring, and on-chain protocols.
• It decomposes forecasting tasks into distinct stages—event extraction, context construction, probabilistic prediction, and multi-dimensional evaluation.
• The framework ensures contamination-free, prospective evaluation through real-time data sourcing, rigorous uncertainty quantification, and immutable, transparent scoring.

A Prediction Arena is a comprehensive, modular evaluation framework designed to benchmark predictive agents—whether humans, LLMs, or other AI systems—on real world or simulated forecasting challenges using proper, incentive-compatible scoring. Prediction Arena architectures extend beyond static datasets and simple prediction tournaments, integrating live market data, on-chain protocols, rigorous uncertainty quantification, and multi-agent dynamics to objectively assess forecasting skill, calibration, reasoning, and market utility under controlled and adversarial conditions.

1. Frameworks and Core Design Principles

Prediction Arena frameworks are structured around several invariant design goals:

• Contamination-Free, Prospective Evaluation: Events and markets are sourced in real time from live prediction markets (e.g., Kalshi, Polymarket), guaranteeing that test questions are genuinely forward-looking and unsullied by training-data leakage (Yang et al., 20 Oct 2025, Zhang et al., 28 Mar 2026, Nechepurenko et al., 1 May 2026).
• Unified Prediction Contexts: All agents receive identical, time-stamped context, typically including a curated set of external news sources (e.g., LLM-retrieved URLs and summaries) and a market snapshot (last trade prices or market-implied probabilities) (Yang et al., 20 Oct 2025).
• Autonomous, Modular Pipeline: Forecasting tasks are decomposed into discrete stages: event harvesting, context construction, probabilistic prediction, and multi-dimensional evaluation. This modularity isolates reasoning, calibration, and memory effects for model-level diagnosis (Yang et al., 20 Oct 2025).
• Live and On-Chain Protocols: Leading implementations (notably Foresight Arena) leverage on-chain smart contracts and trustless oracles, using commit–reveal schemes to ensure prediction independence, verifiability, and agent pseudonymity (Nechepurenko et al., 1 May 2026). Accounting, scoring, and reputation accrual occur immutably on blockchain infrastructure.

2. Pipeline Stages and Mathematical Evaluation

The canonical pipeline, instantiated in frameworks like Prophet Arena and Foresight Arena, consists of:

A. Event and Market Extraction: At scheduled intervals, batches of unresolved events are pulled from primary-market APIs via liquidity, diversity, and recurrence filters. Each event may map to one or more binary markets (Yang et al., 20 Oct 2025, Nechepurenko et al., 1 May 2026).

B. Prediction Context Construction: For each event $E_i$:

• A sequence of forecast times $\{t_i^{(0)}, t_i^{(1)},\dots\}$ is created by iterative bisection up to the event deadline, enforcing a minimum gap (e.g., no forecasts within three hours of market close) (Yang et al., 20 Oct 2025).
• For each $t$, the system delivers: (i) market snapshot $\{q_{ij,t}\}$ normalized to implied probabilities, (ii) document set of $k$ recent, relevant external sources with structured metadata, and (iii) standard prompts or APIs for agent response entry.

C. Probabilistic Forecasting: Agents return probabilities $p_{ij,t}\in[0,1]$ for each market, optionally with free-text rationale, often under temperature-0 hard sampling to suppress output variance (Yang et al., 20 Oct 2025).

D. Post-Resolution Evaluation: After ground-truth realization $o_{ij}\in\{0,1\}$, scoring is performed along three axes:

• Brier Score (strictly proper):

$$BS = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{m_i}\sum_{j=1}^{m_i}(p_{ij}-o_{ij})^2$$

• Expected Calibration Error (ECE):

$$\widehat{ECE}=\frac{1}{m}\sum_{b=1}^B m_b|\hat o_b-\hat p_b|$$

with $m_b$ predictions per bin $\{t_i^{(0)}, t_i^{(1)},\dots\}$0 (Yang et al., 20 Oct 2025).

• Economic Value/Market Return:
  • Buy Yes shares if $\{t_i^{(0)}, t_i^{(1)},\dots\}$1; return is $\{t_i^{(0)}, t_i^{(1)},\dots\}$2.
  • Else, buy No; return is $\{t_i^{(0)}, t_i^{(1)},\dots\}$3.
  • The empirical average across markets produces the “market return” metric (Yang et al., 20 Oct 2025).
• Sharpe Ratio is used for volatility-normalized returns:

$\{t_i^{(0)}, t_i^{(1)},\dots\}$4

• On-chain settings supplement with Alpha Score:

$\{t_i^{(0)}, t_i^{(1)},\dots\}$5

with $\{t_i^{(0)}, t_i^{(1)},\dots\}$6 the baseline Brier of the market-implied prices (Nechepurenko et al., 1 May 2026).

• Murphy Decomposition:

$\{t_i^{(0)}, t_i^{(1)},\dots\}$7

$\{t_i^{(0)}, t_i^{(1)},\dots\}$8 is further decomposed into "resolution gain" and "reliability gap" (Nechepurenko et al., 1 May 2026).

All metric computations, except for agent free-text rationales, are strictly based on the probabilistic vector $\{t_i^{(0)}, t_i^{(1)},\dots\}$9 and known outcomes.

3. Experimental Protocols and Cohort Results

Prediction Arenas are deployed over live, longitudinal evaluations, covering:

• Multi-horizon Forecasting: Models are prompted for estimates at decreasing intervals to resolution to diagnose information lag and updating conservatism (Yang et al., 20 Oct 2025).
• Real-money Trading Simulation: In benchmarks like Prediction Arena, AI agents operate as autonomous traders on live markets, executing buys/sells and accumulating profit/loss as a function of prediction accuracy and market timing (Zhang et al., 28 Mar 2026).

Model/Cohort	Brier Score	ECE	Market Return	Platform	ROI (%)	Settlement Win Rate
GPT-5 (R) High	0.184	0.042	0.943	Kalshi	−20.5	n/a
Claude Sonnet 4	0.194	0.041	0.909	Polymarket	−2.68	33.3
Grok-4 (R)	0.189	0.043	0.864	...	−20.0	71.4 (PM)
Market Baseline	0.187	0.069	0.899	...	--	--
Gemini-3.1-pro-preview	--	--	--	Polymarket	+6.02	--

• Key empirical patterns: Brier scores for SOTA LLMs cluster in [0.18, 0.22]. Markets outperform models at short horizons; model calibration (ECE) is often superior at long range (Yang et al., 20 Oct 2025, Zhang et al., 28 Mar 2026). Average agent returns remain subzero in most settings. Sharpe ratios are uniformly negative over the bulk of agents but reflect relative ranking (Yang et al., 20 Oct 2025, Nechepurenko et al., 1 May 2026).
• Platform effects: Cohorts display stark differences across market infrastructure (e.g., models fare systematically worse on Kalshi than on Polymarket), underlining the role of market design (curated vs discovery-based, exit rules, contract types) in agent evaluation (Zhang et al., 28 Mar 2026).
• Power analyses: Statistical detection of agent superiority (e.g., $t$0 at 80% power) requires approximately 350 resolved binaries (50 rounds at 7 markets/round); halving that edge increases required sample count by a factor of four (Nechepurenko et al., 1 May 2026).

4. Bottlenecks, Pathologies, and Failure Modes

Extensive experimentation reveals persistent limits in both agent and system performance:

• Memory and Internalization: LLMs recognize only 60–80% of prior events; temporal misalignment and coarse recall degrade forecast updating, especially in fine-grained domains (weather, politics) (Yang et al., 20 Oct 2025).
• Context Understanding: Market data alone often matches mean accuracy, but variance is reduced when external sources are integrated. Conversely, low-quality or noisy sources can degrade accuracy (notably in volatile markets like cryptocurrencies) (Yang et al., 20 Oct 2025).
• Information Aggregation and Timing: LLMs systematically underweight high-probability outcomes (“conservatism”), and markets display superior reactivity near event resolution, highlighting lag in model-side information ingestion (Yang et al., 20 Oct 2025).
• Reasoning Synthesis: While models are near-saturated in direct evidence extraction and citation, large qualitative gaps remain in integrating reasoning steps into probability mapping. Chain-of-thought training with explicit numerical mapping is a recommended target (Yang et al., 20 Oct 2025).
• Tournament Pathologies: Large Prediction Arenas (tournaments) exhibit the “tournament paradox,” where moderately accurate but higher-variance predictors can, by luck, outscore more accurate ones. As the field widens, the probability that the winner is among the very best forecasters decreases unless the number of events grows or the skill range is narrowed (Aldous, 2019). This mean–variance tradeoff is a fundamental limit to high-stakes agent competitions.
• Agent Activity Control: Models often lack dynamic exit criteria, leading them to over-trade during periods of diminished edge (or under-trade when edge exists), a pathology also observed in real time leaderboards (Zhang et al., 28 Mar 2026).

5. Extensions: Beyond Simple Forecasting

Recent Prediction Arena variants generalize the paradigm:

• Multi-Model Fusion Arenas: In domains like NCAA bracket prediction, combinatorial fusion analysis (CFA) leverages diversity-weighted score and rank aggregations across multiple ML predictors using rank–score characteristic (RSC) curves and cognitive diversity (CD) metrics. Ensemble methods achieve superior forecast accuracy relative to any single system, e.g., 74.60% team-ranking accuracy, outstripping all public baselines for 2024 NCAA matchups (Wu et al., 11 Mar 2026).
• World Model Arenas: WR-Arena benchmarks world models not just on next-state prediction but on action simulation fidelity (e.g., following multi-step language instructions), long-horizon forecast coherence, and simulative reasoning/planning (i.e., the ability to serve a vision-language or RL planner as a causal simulator). Evaluation protocols cover vision-language judgment, optical-flow–based smoothness, AP penalties for consistency decay, and closed-loop action selection; results expose severe gaps between model generation fidelity and robust action-conditioned planning capability (Team et al., 26 Mar 2026).
• Automated Post-Training “Arenas”: Arena Learning and WizardArena establish closed-loop, AI-annotated systems for continual benchmarking and data flywheel construction. Automated model-vs-model battle outcomes yield near-human-consistent ELO ratings, supporting both evaluation and iterative, difficulty-aware post-training with fine-grained self-critique (Luo et al., 2024).
• On-Chain Agent Registry and Reputation: The Foresight Arena (Prediction Arena) utilizes an ERC-8004 registry, allowing agents to accumulate robust, verifiable forecasting credentials over time, supporting credentialing and ranking in open, trustless competitions (Nechepurenko et al., 1 May 2026).

6. Implications for Benchmark and System Design

The Prediction Arena paradigm yields several concrete design recommendations:

• Statistical Rigor: Longitudinal evaluation with properly powered sample sizes is required to reliably separate genuine agent edge from chance, especially in competitive cohort settings (Nechepurenko et al., 1 May 2026, Aldous, 2019).
• Skill vs. Luck Tradeoff: Arena architects should increase event count or restrict entrant skill ranges to avoid overweighting luck in determining winners. Alternatively, scoring rules can be modified to penalize variance or reward calibration consistency (Aldous, 2019).
• Transparent, Tamper-Resistant Records: Commitment to on-chain, immutable scoring and reputational records supports open, contestable benchmarks and reduces opportunities for gaming (Nechepurenko et al., 1 May 2026).
• Dynamic, Modular Contexting: Explicit control over market data, source feeds, prompt templates, and their timing enables isolation of specific deficiencies (e.g., lag, misalignment, or source contamination) (Yang et al., 20 Oct 2025).
• Open Extensibility: Architectures that permit new agent classes, novel fusion strategies, and expansion to new market universes or outcome types can track advances in agent reasoning, world modeling, and strategic play (Team et al., 26 Mar 2026, Wu et al., 11 Mar 2026).

7. Outlook and Open Directions

The Prediction Arena concept is a living framework, continuously evolving to reflect advances in AI, finance, and statistical methodology. Prospective developments include:

• Augmented Memory and Fact Retrieval: Retrieval-augmented architectures, temporal grounding enhancements, and date-alignment modules to improve agent recall and event tracking (Yang et al., 20 Oct 2025).
• Source-Quality Filtering: Automated signal/noise curation and weighting of external feeds to stabilize probabilistic inference, particularly in highly volatile or adversarial markets (Yang et al., 20 Oct 2025).
• Self-Adaptive Activity: Meta-control agents that regulate exposure based on dynamically inferred edge, optimizing for return variance and risk management (Zhang et al., 28 Mar 2026).
• Beyond Binary Markets: Integration of categorical and open-ended prediction protocols, with generalized scoring (e.g., multi-category Brier or logarithmic scores) (Nechepurenko et al., 1 May 2026).
• World Model–Agent Feedback Loops: Structured, iterative planning arenas where generative world models and LLM/RL planners jointly simulate, act, and are evaluated under physically and semantically complex objectives (Team et al., 26 Mar 2026).
• Automated Data Flywheels: Full unification of evaluation, benchmarking, and model self-improvement cycles, with battle-based selection and preference optimization forming the basis of continuous learning pipelines (Luo et al., 2024).

Prediction Arenas thus provide a rigorous, extensible testbed for quantifying, diagnosing, and ultimately advancing predictive and decision-making intelligence across domains, agent classes, and real-world forecasting tasks.