Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments

Abstract: LLMs are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers' agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.

• The paper introduces a time-partitioned framework using pre-cutoff evidence and post-cutoff outcomes to benchmark scientific idea judgments.
• It employs evidence freezing and offline sandboxing to eliminate contamination, enabling scalable, semi-verifiable future impact predictions.
• Empirical results reveal task- and model-dependent agentic benefits with significant gains in compute scaling and diagnostic execution trace insights.

Proof of Time: Semi-Verifiable Benchmarking for Scientific Idea Judgment in AI

Introduction and Motivation

The evaluation of scientific ideas—especially their future impact—remains a central challenge in metascience and AI for Science initiatives. Existing infrastructures rely predominantly on immediate peer judgments or static benchmarks, which are both time-constrained and subject to knowledge contamination. The "Proof of Time" (PoT) framework addresses gaps in the verifiable, scalable, and future-oriented assessment of scientific idea judgments by LLMs and agents. The framework operationalizes a time-partitioned benchmarking paradigm: solvers are given only pre-cutoff evidence and must forecast signals (citations, awards, SOTA shifts, and research agendas) observable only at a later point.

Figure 1: Workflow overview for the PoT benchmark, including dataset creation, task family construction, and the agentic offline sandbox protocol.

Benchmark Design and Task Formalization

PoT instances are formalized as tuples $({\mathcal{E}}_{\leq t_0}, q, y_{t_1})$, comprising pre-cutoff evidence $\mathcal{E}_{\leq t_0}$, a query $q$ over candidates (e.g., papers, researchers), and a post-cutoff label $y_{t_1}$. Key methodological innovations include:

• Evidence freezing and offline sandboxing, eliminating web-based contamination and making gains from tool use interpretable.
• Use of naturally emerging, post-cutoff signals rather than static, human-annotated targets, achieving semi-verifiability at scale.

PoT encompasses four primary task families, each targeting distinct facets of scientific impact and reasoning:

1. Impact Prediction: Citation forecasting on papers in top conferences, isolating parametric forecasting from evidence-based judgments.
2. Peer-Review Award Prediction: Anticipation of award tiers as operationalized by official conference outcomes.
3. Research Evolution: Forecasting researcher's field focus and publication attribution, requiring longitudinal reasoning.
4. Technological Frontier (SOTA): Extrapolation on benchmark progression and SOTA scores, emphasizing the robustness to prompt and metric variations.

Experimental Setup

The evaluation suite covers 30,000+ instances distributed over these task families. Solvers include:

• Zero-shot models with only prompt-based inference.
• Agentic models leveraging an offline toolset (including Python, shell, text editors) under strict evidence constraints.
• Agentic models with a further-structured prompt, designed to maximize evidence extraction and protocol adherence.

Test-time budgets are modulated via message limits (15, 30, 50), quantifying the relationship between agentic interaction and performance gains. The model pool comprises recent foundation models spanning the Anthropic Claude 4.5 series, Google Gemini lineage, and OpenAI GPT-5 family.

Results and Analysis

Scaling with Test-Time Compute

The analysis demonstrates that increased message limits yield marked accuracy improvements across most agentic configurations, with family-specific differences in scaling efficiency. Notably, Claude models benefit substantially from greater test-time compute, with accuracy gains exceeding 25 percentage points from lowest to highest budget. Gemini models display high initial performance but reduced marginal scaling, while GPT variants present moderate but saturating improvements.

Figure 2: (A) Compute scaling across models; (B) Task-family agentic vs. zero-shot performance; (C) Effect of structured prompting.

Figure 3: Model-wise scaling gains from increased compute budgets (Acc@50 – Acc@15).

Figure 4: Family-level scaling trends highlighting systematic differences in compute utility.

Agentic Versus Zero-Shot Performance

Agentic solvers outperform zero-shot solvers when tasks require evidence exploration and aggregation, with the Faculty task family exhibiting the largest boost—from near random to ~66% accuracy. Conversely, Peer-Review Award prediction shows negligible aggregate improvement, reflecting the weak signal in the available pre-cutoff evidence and human label subjectivity.

Figure 5: Direct comparison of zero-shot vs. agentic performance across task-model pairs.

Prompt Engineering and Robustness

The structured prompt, designed to enforce explicit agentic protocols and local-only operation, yields variable improvements. For some model families (notably Claude), such structuring amplifies agentic gains, but for others, especially GPT, it can be detrimental or neutral—indicating model-dependent optimal prompting regimes and suggesting risks associated with over-prescription at the policy level.

Post-Cutoff versus Pre-Cutoff Evaluation

PoT's value is underscored by its capacity to reveal shifts in model ranking when evaluation is anchored on post-cutoff, contamination-resistant signals. For example, OpenAI and Gemini models display pronounced changes—gain or drop up to ~25 percentage points—when shifting from pre-cutoff to post-cutoff signals. This highlights the practical necessity of time-partitioned benchmarking for robust evaluation.

Failure Modes and Agentic Diagnostics

Analysis of 1,759 agentic execution traces reveals three primary routing outcomes: complete-correct, complete-wrong, and incomplete (budget-exhausted). Completion errors are largely attributed to reasoning mistakes (37.7%) and retrieval errors (36.3%), while incompleteness is dominated by looping or thrashing behaviors (up to 36%), demonstrating that agentic architecture and protocol discipline are critical determinants of success. Even with correct outcomes, many “lucky” paths and redundant explorations were observed, underscoring the need for trace-based evaluation and process audits beyond aggregate scores.

Figure 6: Distribution and analysis of execution traces that exhaust the allowed interaction/message budget.

Cost, Efficiency, and Practical Trade-Offs

Agentic operation entails substantial compute and token overhead—one to two orders of magnitude higher than zero-shot inference. Efficiency frontiers plotted as accuracy gain versus token overhead reveal that such expenses are only justified in evidence-intensive, information-sparse settings (Faculty, Citations). Routine use of high-budget agents is not recommended unless required by the prospective cost of downstream errors or task importance.

Figure 7: Model-level efficiency frontier depicting the non-linear trade-off between agentic accuracy gains and token overhead.

Figure 8: Task-family efficiency comparison, highlighting greatest agentic payoffs for evidence-rich domains.

Theoretical and Practical Implications

The PoT benchmark presents a scalable, refreshable framework for time-indexed model evaluation, uniquely supporting:

• Semi-verifiable, future-facing evaluation through external post-cutoff signals.
• Isolation of contamination effects, essential in the context of ever-expanding LLM pre-training corpora.
• Detailed agentic diagnostic auditing, relevant for agent alignment research and robust tool-use protocol design.

Empirically, the findings challenge a uniform “agent always helps” assumption. Agentic gains are sharply task and model dependent. The negligible improvement on Peer-Review Award tasks raises questions about the adequacy of current automated proxies for subjective scientific value and highlights the inherent limitations imposed by proxy noise and label design.

Future Directions

Key targets for future development include:

• Expanding benchmark coverage beyond NLP and AI research venues, integrating multi-domain metascientific signals as they become available.
• Exploring architectural and interface improvements to agentic solvers that mitigate looping and budget exhaustion.
• Investigating adaptive budget allocation strategies, dynamic prompt calibration, and more expressive interaction policies.
• Integrating process-oriented diagnostics in evaluation protocols, moving beyond outcome-only metrics.
• Formalizing protocols for minimizing benchmark data contamination as model and data scale increases.

Conclusion

The "Proof of Time" framework provides a principled, semi-verifiable, and contamination-resilient foundation for the evaluation of scientific idea judgments by LLMs and agentic systems (2601.07606). Through time-partitioned design and post-cutoff outcome linkage, PoT enables direct measurement of future-predictive capability—a critical requirement for AI systems deployed within scientific workflows. The observed empirical heterogeneity across tasks, models, and prompting strategies underscores the necessity of granular, task-specific evaluation for agentic systems and paves the way for more comprehensive, evidence-grounded assessment protocols in metascience and AI for Science.