OracleProto: LLM Forecasting Evaluation

• OracleProto is a reproducible framework for evaluating LLM forecasting capabilities under strict, controlled information boundaries.
• It employs model-cutoff aligned sample admission, tool-level temporal masking, and content-level leakage detection to prevent data contamination.
• Empirical evaluations reveal stable composite accuracy and significant cost efficiency across LLMs, supporting its use as an auditable benchmark.

to=arxiv_search.search ฝ่ายขายข่าว_entire_response=false dev_ui=false code='{"query":"all:OracleProto", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}' to=arxiv_search.search 亿贝િતire_response=false dev_ui=false code='{"query":"all:\"Dodona Protocol\" OR ti:Dodona", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}' to=arxiv_search.search иҭ_entire_response=false dev_ui=false code='{"query":"id:(Ma et al., 5 May 2026) OR id:(Caldarelli, 6 Jun 2026) OR id:(Kakarla et al., 2023)", "max_results": 5, "sort_by": "relevance", "sort_order": "descending"}' OracleProto is a reproducible framework for evaluating LLM native forecasting capability under controlled information boundaries. It reconstructs resolved events into time-bounded forecasting samples by combining model-cutoff-aligned sample admission, tool-level temporal masking, content-level leakage detection, discrete answer normalization, and hierarchical scoring, thereby turning archival forecasting datasets into contamination-controlled, auditable, and trainable evaluation and training corpora (Ma et al., 5 May 2026). In recent arXiv usage, the name is associated primarily with forecasting evaluation, although related literature also uses closely related “oracle” terminology in distinct senses, including blockchain oracle design and oracle-based protocol testing (Caldarelli, 6 Jun 2026, Kakarla et al., 2023).

1. Motivating paradox and problem formulation

OracleProto was proposed in response to a central paradox in LLM forecasting evaluation: live forecasting benchmarks are uncontaminated but ephemeral, while retrospective benchmarks are reusable but often confound genuine forecasting with factual recall (Ma et al., 5 May 2026). The framework treats resolved events beyond a model’s disclosed knowledge cutoff as fresh forecasting samples, with identical information boundaries and scoring rules for every model.

The underlying motivation is that LLM “forecasting” differs qualitatively from simple Q&A: it must combine incomplete parametric knowledge, evidence gathering, situational judgment, and uncertainty-aware decision-making. Retrospective evaluation is therefore problematic if a model can answer from pretraining rather than from forecasting competence. OracleProto addresses this by enforcing three layered information boundaries—model-cutoff-aligned sample admission, tool-level temporal masking, and content-level leakage detection—then coupling these with discrete answer normalization and a multi-stage, hierarchical scoring protocol (Ma et al., 5 May 2026).

This design makes a single dataset rerunnable, auditable, and extensible across cutoffs. A further implication stated explicitly in the source is that the same dataset can be used as a controlled training signal for supervised fine-tuning and reinforcement learning workflows. The framework therefore recasts forecasting from a one-off evaluation setting into what the paper describes as an auditable, reusable, and trainable dataset-level capability (Ma et al., 5 May 2026).

2. Core architecture and information boundaries

OracleProto formalizes each candidate forecasting question as

$q_i = (x_i, \mathcal{A}_i, Y_i, \tau_i, \rho_i),$

where $x_i$ is the stem, $\mathcal{A}_i$ the finite candidate set, $Y_i$ the gold answer subset, $\tau_i$ the resolution time, and $\rho_i$ the answer-structure, either single-answer or multi-answer (Ma et al., 5 May 2026). Given a model $M$ with disclosed parametric knowledge cutoff $\kappa_M$ and a temporal masking offset $\delta$—typically one day—the prediction cutoff is defined as $\chi_i \coloneqq \tau_i - \delta$.

Sample admission is then restricted to

$x_i$0

Instances outside this boundary are logged as “outside-boundary” and never scored. The intent is to ensure that every scored question was genuinely unknown at inference time for the model under evaluation (Ma et al., 5 May 2026).

The second layer is tool-level temporal masking. At reasoning step $x_i$1, the model may issue a retrieval query $x_i$2, but the backend filters out any document whose published date is later than $x_i$3:

$x_i$4

This guarantees that no surface-level content younger than the prediction cutoff reaches the model (Ma et al., 5 May 2026).

The third layer is content-level leakage detection. A lightweight auxiliary LLM-based detector $x_i$5 examines each retrieved page together with the cutoff date and emits a binary keep/drop verdict:

$x_i$6

The detector prompt is restricted to title, URL, publication date, snippet, and cutoff date, and it is not allowed to read the question or correct answer. It fails closed: any transient or structural error causes drop (Ma et al., 5 May 2026).

3. Answer normalization and hierarchical scoring

OracleProto requires that every raw model answer contain a final boxed prediction in one of three families: yes/no, binary-named, or multiple-choice. A deterministic parser $x_i$7 extracts the last \boxed{...} payload and maps it into a subset of labels in $x_i$8, or returns $x_i$9 for unevaluable outputs. A normalization map $\mathcal{A}_i$0 then sends both gold answers and predictions into a finite canonical label set, so that scoring compares $\mathcal{A}_i$1 with $\mathcal{A}_i$2 (Ma et al., 5 May 2026).

The evaluation chain proceeds in four levels. The first is validity, defined by whether parsing succeeds. The second is item correctness, where valid outputs are scored by exact match between normalized predicted and gold sets. For multi-answer questions, the framework also records $\mathcal{A}_i$3, $\mathcal{A}_i$4, and $\mathcal{A}_i$5 (Ma et al., 5 May 2026).

The third level is question-level behavior over repeated runs. OracleProto defines an exam-style partial-credit function

$\mathcal{A}_i$6

It also reports pass@1, pass, pass, and a Format Skill Score based on a chance-corrected Tversky similarity with $\mathcal{A}_i$7 and $\mathcal{A}_i$8. Consistency is measured with Cohen’s $\mathcal{A}_i$9 and Fleiss’ $Y_i$0, using baselines $Y_i$1 for single-pick or $Y_i$2 per label for multi-pick (Ma et al., 5 May 2026).

At the model level, questions are partitioned into buckets $Y_i$3 and aggregated with weights $Y_i$4 for composite accuracy:

$Y_i$5

The framework also defines per-correct cost as

$Y_i$6

Taken together, these measures are designed to characterize correctness, partial correctness, stability across repeated trials, chance-corrected format discipline, and cost efficiency within a single evaluation protocol (Ma et al., 5 May 2026).

4. Dataset instantiation and reproducibility properties

The reported instantiation draws 80 closed-choice questions from the FutureX-Past archive with resolution dates between 2026-03-11 and 2026-04-14 (Ma et al., 5 May 2026). Questions were filtered for finite candidate sets with $Y_i$7, ISO-8601-parseable resolution times, and a manual “zero-leakage” stem-and-option audit by two annotators that targeted post-hoc phrasing cues, answer-encoded options, and empty multi-answer sets.

After filtering, each row is collapsed into the canonical tuple $Y_i$8. The paper reports the following question-type distribution:

Type	Single-Answer	Multi-Answer
Yes/No	37	0
Binary Choice	3	0
Multiple Choice	32	8

The dataset construction pipeline is described as fully deterministic, producing a byte-identical SQLite file given the same FutureX-Past snapshot, prompt templates, and audit decisions (Ma et al., 5 May 2026). Code and data are reported as available at https://github.com/MaYiding/OracleProto and https://huggingface.co/datasets/MaYiding/OracleProto.

This reproducibility claim is central to the framework’s purpose. Rather than relying on one-time benchmark snapshots, OracleProto defines an evaluation asset that can be replayed and audited under cutoff-aware conditions. The paper further states that each run records the entire dialogue, retrieval trace, filtered page set, reasoning chain, and final normalized answer, thereby furnishing complete training examples for supervised fine-tuning and reward signals for reinforcement learning (Ma et al., 5 May 2026).

5. Empirical results, leakage audit, and cost structure

OracleProto was evaluated on six contemporary LLMs, all accessed through an OpenAI-compatible POST /chat/completions endpoint with browsing disabled: DeepSeek-V3.2-Exp, GLM 5, Qwen3.5-Flash, MiniMax M2.5, Kimi K2.5, and Doubao Seed 2.0 Lite (Ma et al., 5 May 2026). Their disclosed cutoffs ranged from 2025-09-29 to 2026-03-10, and with $Y_i$9 day every question fell strictly after every model’s cutoff, so the effective evaluation set was the full dataset for each model.

Composite accuracy and cost were reported as follows:

Model	Composite Accuracy	Cost_per_correct (USD)
DeepSeek-V3.2-Exp	0.6016	0.025
GLM 5	0.6002	0.048
Qwen3.5-Flash	0.5896	0.003
MiniMax M2.5	0.5494	0.024
Kimi K2.5	0.5800	0.049
Doubao Seed 2.0 Lite	0.5858	0.006

The paper characterizes the spread in composite accuracy as narrow, at 5.2 percentage points, while per-correct cost varied 16×. It identifies Qwen3.5-Flash and DeepSeek-V3.2-Exp as defining the cost-quality Pareto frontier (Ma et al., 5 May 2026).

Per-bucket performance shows that all models struggled most on MC-Multi. For example, DeepSeek-V3.2-Exp obtained 0.6261 on Yes/No, 0.8889 on Binary, 0.5938 on MC-Single, and 0.2986 on MC-Multi, while Doubao Seed 2.0 Lite obtained 0.4828, 1.0000, 0.6061, and 0.2460, respectively (Ma et al., 5 May 2026). Consistency and stability metrics further differentiated the models: Qwen3.5-Flash led on FSS and Fleiss’s $\tau_i$0, which the paper interprets as signalling relative parsimony in false positives and consistency across trials.

A central empirical claim is the leakage audit. In a manual audit of $\tau_i$1 sampled pages, the content-level detector missed 3 leaks, yielding

$\tau_i$2

The paper reports a Wilson 95\% confidence-interval upper bound of approximately 3.2\%, together with recall $\tau_i$3 and specificity $\tau_i$4 (Ma et al., 5 May 2026). The abstract summarizes this as reducing residual leakage to the $\tau_i$5 level, an order of magnitude below tool-only temporal filtering.

6. Terminological context and distinct uses of “oracle”

The term “oracle” is used differently across adjacent research areas, and OracleProto should not be conflated with those other meanings. In the Eywa framework for oracle-based protocol testing, an oracle is a behavioral reference mechanism for black-box testing of network protocol implementations. There, the framework is formalized as $\tau_i$6, where an executable reference implementation synthesized automatically from natural-language sources via an LLM provides the behavioral oracle, symbolic execution with KLEE derives exhaustive test suites, and differential execution against black-box implementations flags deviations (Kakarla et al., 2023). Eywa’s DNS case study reports 26 unique bugs across ten widely used DNS implementations, including 11 new bugs (Kakarla et al., 2023).

A second distinct usage appears in the Dodona Protocol, a modular, chain-agnostic oracle service inspired by procedural patterns identified in ancient and modern oracle systems. Its first module implements a binding query-and-dispute mechanism in which a named expert resolver provides binding answers to structured questions submitted by petitioners. The system emphasizes structured consultation, access control, attributable resolution, constrained query formats, reputational accountability, and tiered service availability, and it explicitly states that the oracle does not claim to reveal objective truth but instead produces outcomes that parties have agreed in advance to accept (Caldarelli, 6 Jun 2026).

These usages share the word “oracle” but denote different technical objects. In Eywa, the oracle is an executable behavioral specification for testing. In Dodona, it is a coordination and dispute-resolution mechanism for blockchain-based systems. In OracleProto proper, the “oracle” idea is methodological rather than institutional: the framework constructs contamination-controlled forecasting tasks under explicit temporal and informational boundaries, then evaluates LLM outputs through deterministic parsing and hierarchical scoring (Ma et al., 5 May 2026). This suggests that the common term marks a family resemblance around adjudication or reference behavior, but not a single research program.