FutureSim: Replaying World Events to Evaluate Adaptive Agents

Abstract: AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory, and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world.

• The paper introduces a simulation framework that replays real-world events to test LLM agents’ adaptive forecasting through evolving evidence and free-form predictions.
• Empirical results highlight that GPT 5.5 uniquely achieves a positive Brier Skill Score and 25% top-1 accuracy, while other models remain marred by anchoring and miscalibration.
• The study reveals core deficits in test-time adaptation, calibration, and long-horizon memory, emphasizing the need for enhanced online learning and search augmentation strategies.

FutureSim: Evaluating Long-Horizon Adaptation in Forecasting Agents

Introduction

"FutureSim: Replaying World Events to Evaluate Adaptive Agents" (2605.15188) introduces a comprehensive simulation framework for benchmarking the long-horizon, adaptive forecasting capabilities of LLM agents. The evaluation paradigm is instantiated through an environment that chronologically replays real news events occurring after the agents’ knowledge cutoffs, requiring forecasts on unseen, future-resolved events. This setup is intentionally designed to stress adaptive reasoning, memory, search, and continual learning with evolving evidence, addressing limitations of both synthetic and live-market agentic benchmarks.

Environment Design and Benchmark Structure

FutureSim implements a discrete-time simulation, where each time step corresponds to a real-world date. At each step, agents can:

• Search a growing news corpus, with strict access to only contemporaneous documents;
• Update probabilistic forecasts over free-form outcome sets (not restricted to fixed choices);
• Optionally revise their prior predictions based on both fresh evidence and feedback from resolved questions.

The minimal action space (submit_forecast, next_day) maximizes experimenter flexibility and reproducibility while eliminating confounds from environment-manipulating actions. The event curation pipeline synthesizes high-quality forecasting questions (e.g., elections, geopolitical outcomes, sports winners) from timestamped news, aligning resolution dates strictly post-model cutoff and ensuring the answer cannot be leaked from stale evidence. Accurate Brier Skill Score (BSS) and top-1 accuracy are defined over free-form, agent-generated outcome distributions, leveraging LLM-based semantic answer matching.

Figure 1: Schematic of the FutureSim environment mechanics and interface for agent interaction, including data gating and permissible actions.

Agent Evaluation and Key Results

An extensive experiment set evaluates both closed- and open-weight foundation models (GPT 5.5, Claude Opus 4.6, DeepSeek V4 Pro, Qwen 3.6 Plus, GLM 5.1). Agents interact with the environment for a full simulation window (January–March 2026), consuming 10M+ unique tokens and generating hundreds to thousands of tool calls per run.

Figure 2: Agents must repeatedly update forecasts over a chronologically-evolving news corpus, with strong separation in test accuracy and calibration across models.

Key findings:

• GPT 5.5 is the only model consistently achieving a positive Brier Skill Score and the highest top-1 accuracy (25%), robustly improving its predictions over time.
• All open-weight models, despite improvements under optimized harnesses, display negative Brier Skill Scores when left to their default orchestration—i.e., being worse than abstention, with overconfident or miscalibrated mass on wrong answers.
• Most agents display marked inertia, showing limited adaptation to evidence, anchoring strongly to initial predictions even with explicit feedback and access to new data.

  Figure 3: Under improved harnesses, open-weight models improve accuracy and, in some cases, BSS, but only GPT 5.5 is consistently well-calibrated; default harnesses yield negative BSS over long horizons.

Extensively, the framework exposes large gaps between current LLM test-time adaptation and the theoretical optimum (i.e., single-question inference with maximal context), confirming that real-world sequential event forecasting is an unsolved challenge for current agents.

Analysis of Adaptation, Search, Memory, and Compute

Adaptation and Anchoring

Agents remain anchored to poor initialization; when initialized to bad baselines, nearly all models—including the best—fail to substantially improve Brier Skill Score, often not even reaching the abstain baseline (BSS = 0) despite feedback from resolved questions. This highlights severe test-time adaptation deficits.

Figure 4: Models exhibit strong anchoring; starting from poor predictions, they fail to adapt away despite accumulating evidence and feedback (left). Memory ablations indicate inference-time memory is critical for improvement (right).

Role of Memory

Ablation studies demonstrate a consistent degradation of agent performance when the ability to write and query persistent memory is removed. Structured memory is necessary for long-horizon context integration owing to context window limitations of transformer architectures.

Search and Sequential Reasoning

Updating the search corpus with newly-arrived articles each day drives large increases in accuracy and calibration, compared to freezing context or limiting agents to pre-simulation evidence. Full agentic search (multi-step, creative queries) outperforms single-query or title-based retrieval strategies, confirming that effective information-seeking under uncertainty and continual Bayesian updating are unresolved problems.

Figure 5: Ablations reveal substantial improvements from daily context updates and agentic search, both in accuracy and Brier skill score.

Test-Time Compute Scaling

Scaling GPT 5.5’s allowed stepwise reasoning strongly increases Brier skill score and accuracy, though with diminishing returns at maximal levels. Notably, low reasoning effort yields catastrophic calibration failures—accuracy mismatches calibration because correct top predictions are assigned low probability mass.

Figure 6: Increasing test-time compute (i.e., reasoning effort) yields monotonic improvements in Brier skill score, with diminishing returns after the high setting.

Multi-Agent Effects

Running simultaneous agents and exposing a peer-aggregate (mean crowd prediction) nudges individual agents toward the collective, reducing divergence and anchoring predictions. Nonetheless, absolute accuracy remains similar, indicating that endogenous teacher-forcing does not trivially overcome core adaptation deficits.

Figure 7: In multi-agent simulation, agent predictions show evidence of convergence toward the aggregate, despite a stated incentive to outperform the group.

Comparisons to Human Aggregates and Other Benchmarks

On a subset of questions with live prediction market analogs (e.g., Polymarket), GPT 5.5 often matches or leads the real-money human crowd in prediction updates for major outcomes (e.g., Super Bowl), though lags or is outperformed on questions where human commonsense or non-news evidence is critical. This demonstrates the economic relevance and challenge of the environment relative to live, unreproducible market data.

Figure 8: GPT 5.5's prediction trajectory can anticipate or closely track large human market aggregates on real-world event questions, but lags on other domains.

Implications, Limitations, and Future Directions

The FutureSim environment exposes that, despite strong performance in static test regimes, current foundation models—both closed and open-weight—struggle at adaptive, calibrated, and memory-augmented sequential world modeling. Key limitations surfaced include:

• Anchoring and Inertia: State-of-the-art LLMs exhibit insufficient test-time adaptation, being unable to reliquefy their posterior beliefs in light of substantial new evidence and resolved-feedback, especially when initialization is suboptimal.
• Calibration and Overconfidence: Many agents produce poorly distributed outcome probabilities, underlining inadequacies in structured uncertainty representation.
• Long-Horizon Memory: Even with context window extension strategies, the inability to effectively offload, retrieve, and integrate long-horizon evidence without prompt engineering or external scaffolding remains a bottleneck.

Practical implications involve the measured deployment of LLM agents for real-world decision support in contexts where new information is continually arriving and adaptation over long time horizons is mandatory. Harness engineering, improved memory and context compaction, advanced search/retrieval, and online learning (e.g., via test-time training or finetuning) are all indicated as vital research directions.

Theoretically, FutureSim quantifies a lower bound for genuinely adaptive world modeling in open-ended domains, providing a reproducible testbed for studying lifelong learning, continual Bayesian updating, and robust sequential aggregation at scale.

Conclusion

FutureSim offers the first scalable, open-ended, and reproducible framework—grounded in real world event sequences post LLM knowledge cutoff—for quantifying and dissecting agentic adaptation, search, memory, and calibration over multi-month horizons. Empirical results uncover unambiguous deficits in current agent architectures and orchestration procedures, while the environment itself constitutes a robust target for future research in adaptive reasoning, context management, search augmentation, and meta-learning for AI agents operating in dynamically evolving environments.