OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking

Abstract: LLMs are moving from static text generators toward real-world decision-support systems, where forecasting is a composite capability that links information gathering, evidence integration, situational judgment, and action-oriented decision making. This capability is in broad demand across finance, policy, industry, and scientific research, yet its evaluation remains difficult: live benchmarks evaluate forecasts before answers exist, making them the cleanest way to measure forecasting ability, but they expire once events resolve; retrospective benchmarks are reproducible, but they cannot reliably distinguish genuine forecasting from facts a model may have already learned during pretraining. Prompting models to "pretend not to know" cannot replace a genuine knowledge boundary. We propose OracleProto, a reproducible framework for evaluating LLM native forecasting capability. OracleProto reconstructs resolved events into time-bounded forecasting samples by combining model-cutoff-aligned sample admission, tool-level temporal masking, content-level leakage detection, discrete answer normalization, and hierarchical scoring. Instantiated on a FutureX-Past-derived dataset with six contemporary LLMs, OracleProto distinguishes forecasting quality, sampling stability, and cost efficiency under controlled information boundaries, while reducing residual leakage to the $1\%$ level, an order of magnitude below tool-only temporal filtering. OracleProto turns LLM forecasting from one-off evaluation into an auditable, reusable, and trainable dataset-level capability, providing a unified interface for fair cross-model comparison and a controlled signal source for downstream SFT and RL. Code and data are available at https://github.com/MaYiding/OracleProto and https://huggingface.co/datasets/MaYiding/OracleProto.

• The paper introduces OracleProto, a framework enabling reproducible LLM forecasting evaluation by enforcing knowledge cutoffs and temporal masking.
• It details a methodology that integrates dataset filtering, multi-layer content screening, and reproducible evaluation units to minimize outcome leakage.
• Experimental results reveal a narrow performance band among LLMs while highlighting critical cost efficiency and answer consistency trade-offs.

OracleProto: Framework for Reproducible and Auditable LLM Forecasting Evaluation

Motivation and Problem Statement

Evaluating the forecasting capability of LLMs—defined as their ability to reason under uncertainty about events whose outcomes are not (parametrically or externally) observable—remains a nontrivial challenge. Prospective/live-event evaluations are gold standard from a contamination perspective, but they suffer from impermanence and non-reproducibility. In contrast, retrospective evaluations are reproducible and support longitudinal audit but are marred by outcome leakage, as answers to historical events may be directly or indirectly encoded in model parameters or accessible via external tools. Ad hoc attempts to simulate ignorance via prompt-time instructions are shown to be inadequate, since models are not genuinely unaware of outcomes encountered in pretraining or retrieval contexts.

OracleProto offers a principled, dataset-level framework to mitigate such leakage by enforcing model-aligned knowledge cutoffs, strict temporal masking of retrieval channels, and multi-layer content filtering, allowing previously resolved events to be reliably re-purposed as benchmarking and training assets for evaluating the native forecasting ability of LLMs.

Figure 1: OracleProto overview, centered on the reproducible run unit $\mathcal{R}$ specifying all dataset, model, cutoff, masking, and evaluation parameters for auditable LLM-native forecasting.

Framework and Methodology

OracleProto's core novelty is constructing a reproducible unit of evaluation $\mathcal{R}$, which binds together:

• Dataset $\mathcal{D}$: Pooled from resolved, discrete-outcome forecasting events (FutureX-Past instantiation) filtered to eliminate items contaminated by knowledge-cutoff ambiguity, post-resolution cues, or event framing errors.
• Model $M$ and Knowledge Cutoff $\kappa_M$: For each LLM, only those question instances with event resolution after the knowledge cutoff are admissible.
• Temporal Masking $\chi_i$: Retrieval channels are strictly limited to pre-resolution evidence and filtered at both the metadata (date) and semantic (content) level using an auxiliary LLM-based detector.
• Discrete Output and Parsing: All model responses must be normalized to a canonical finite answer set, supporting exact, audit-friendly evaluation and reducing ambiguity.
• Evaluation Protocol $\Gamma$: Hierarchical metrics spanning item-level correctness, question-level sampling stability, and model-level aggregate scores, with chance correction and false-positive-aware partial credit schemes.

This architecture makes every component in the evaluation stack individually auditable and byte-level replayable, facilitating rigorous cross-model comparison, temporal roll-forward, and forensics.

Controlled Information Boundaries

The framework’s technical advancement lies in the enforcement of information boundaries along three dimensions:

1. Parametric Knowledge: Strict admission filtering ensures all evaluated questions resolved after the model’s disclosed pretraining cutoff $\kappa_M$.
2. Tool-mediated Retrieval: All web searches (Tavily backend) are hard-masked to exclude content postdating a configurable offset before event resolution.
3. Content-level Semantic Filtering: An auxiliary LLM inspects all search results for leaking future or resolved outcomes, achieving residual leakage rates near 1%, an order of magnitude below single-layer masking and close to manual annotation precision.

All leakage-prevention logic is decorrelated from prompt framing, which is shown in prior work to be insufficient (Li et al., 20 Jan 2026), and implemented in the execution harness itself, logged per-sample for reproducibility and traceability.

Experimental Results

Benchmarking Six Contemporary LLMs

OracleProto is instantiated on 80 resolved forecasting questions picked from the FutureX-Past corpus, covering diverse types (Yes/No, binary, and multiclass, both single and multi-answer), with six high-profile frontier LLMs evaluated in three independent runs per item:

• DeepSeek-V3.2-Exp
• GLM 5
• Qwen3.5-Flash
• MiniMax M2.5
• Kimi K2.5
• Doubao Seed 2.0 Lite

All models passed strict cutoff admission; no question fell within training windows.

Metric Synthesis

Composite accuracy, partial-credit scores, and various chance-corrected kappa indices were computed. Notably:

• Composite accuracy for all models falls within a tight 5.2pp band (0.549–0.601), revealing a current performance ceiling for native forecasting under strict contamination control.
• Cost efficiency varied sharply, with Qwen3.5-Flash achieving the top cost/per-correct decision ($\sim$\$0.003) and Kimi/GLM5 lagging at over 16× higher cost without corresponding accuracy lift.
• Sampling stability and answer consistency are quantified with pass@1, pass∧, and Fleiss' $\kappa$; Qwen demonstrates the most consistent multi-sample answers, while Kimi exhibits high output variance.
• Residual ground-truth leakage in the retrieval path is controlled to $\mathcal{R}$01.1% (95% CI: $\mathcal{R}$1) via the combined date and semantic filtering layers, significantly exceeding simple metadata-based masking and approaching manual curation fidelity.

Practical and Theoretical Implications

By reframing resolved events as reproducible, dataset-driven forecasting tasks with auditable information boundaries, OracleProto provides a systematic foundation for:

• Benchmark Evolution: Expired forecast benchmarks (e.g., FutureX-Past, Metaculus, ForecastBench) are no longer discarded once events resolve but become ever-growing audit-ready pools, facilitating continuous evaluation and RL/SFT-driven retraining under clean, contamination-controlled constraints.
• Trainable Forecasting: Rows in OracleProto's datasets carry not just evaluation signals but also dense training traces (retrieval context, deliberation trajectories, normalized answers) for direct SFT/RL application.
• Deployment in High-Stakes Domains: The run-level audit and hard information boundary make OracleProto a candidate compliance mechanism for finance, policy, and safety-critical use, allowing backward inspection of the evidentiary basis for every model judgment.
• Decoupling Evaluation from Single-Run Live Events: By design, OracleProto transitions forecasting evaluation from ephemeral, non-replayable live contest settings into a formally specified, continuously extensible dataset regime.

Speculation on Field Trajectory

OracleProto's contribution points toward a new standard: forecasting evaluation and training become longitudinal, accumulative, and strictly auditable, in parallel to the ascendancy of dataset-driven natural language understanding benchmarks. Live-event, point-in-time leaderboards yield primacy to dataset-level curation and meta-evaluation; field-wide comparison becomes possible across time, architectures, and retrieval protocols. With sufficiently scaled retroactive datasets, model-native forecasting may become an optimizable axis of LLM development rather than an emergent epiphenomenon.

Conclusion

OracleProto systematically addresses the contamination and reproducibility crisis in LLM-native forecasting evaluation. By enforcing model-aligned knowledge cutoffs, multi-layer temporal and semantic masking, and reproducible parsing/norming over a structured discrete answer space, the framework supports fully auditable, contamination-minimized, and extensible benchmarking. Results across multiple SOTA LLMs reveal both persistent challenges (tight clustering, cost-quality tradeoffs) and opportunities for more efficient and stable forecasting. Critically, OracleProto transforms resolved-event data from a wasted byproduct into an ever-growing resource for both rigorous evaluation and direct model optimization. This positions it as a foundational asset for future research in agentic forecasting, dataset curation, and trustworthy AI deployment.

---

Reference: "OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking" (2605.03762)