Ever-Evolving Science Exam Benchmark

• EESE is a dynamic benchmark designed with a large curated pool and a periodically refreshed public subset to assess scientific understanding in AI models.
• It emphasizes range, reach, and rigor by using expert-curated question–answer pairs and stratified sampling across 5 disciplines and 500+ subfields.
• The benchmark mitigates data leakage and supports forward-compatible evaluations by coupling diagnostic performance with resource-efficiency metrics.

Searching arXiv for the specified paper to ground the article and confirm bibliographic details. The Ever-Evolving Science Exam (EESE) is a dynamic benchmark for evaluating scientific understanding in foundation models. It was introduced to address two recurring problems in science benchmarking: data leakage risks that compromise validity and evaluation inefficiency caused by large-scale testing. EESE combines a non-public EESE-Pool containing over 100K expertly constructed science question–answer pairs across 5 disciplines and 500+ subfields with a periodically refreshed public EESE subset of 500 instances. The design is explicitly organized around Range, Reach, and Rigor, and is intended to provide a leakage-resilient, low-overhead, and forward-compatible measure of how well foundation models handle science questions (Wang et al., 22 Jul 2025).

1. Definition and benchmark design

EESE consists of two linked components. The first is EESE-Pool, a non-public reservoir with more than 100,000 science instances. The second is EESE, a 500-instance subset sampled from the pool and periodically updated. The pool is the source of breadth and long-term freshness; the subset is the operational benchmark for routine model evaluation (Wang et al., 22 Jul 2025).

The benchmark is motivated by the observation that existing science benchmarks have made progress toward broad Range, wide Reach, and high Rigor, but often remain vulnerable to contamination from public exposure and costly to administer at scale. EESE addresses these issues by separating a large, privately maintained source pool from a small public evaluation surface. This suggests a benchmark architecture in which statistical coverage and practical deployability are decoupled rather than treated as competing objectives.

The benchmark’s scientific scope is divided into five parent disciplines: Natural Sciences, Agricultural Sciences, Medical Sciences, Engineering & Technological Sciences, and Humanities & Social Sciences. In evaluation, these are reported as NS, AS, MS, ETS, and SSH, together with an overall score (Wang et al., 22 Jul 2025).

2. EESE-Pool construction and coverage

The pool is described as having scale beyond 100,000 question–answer pairs and coverage over 500+ subfields. Its disciplinary composition is explicitly enumerated in the source description.

Parent discipline	Subfields	Approximate instances
Natural Sciences (A₁–A₉₁)	91	~25,000
Agricultural Sciences (B₁–B₂₄)	24	~8,000
Medical Sciences (C₁–C₄₁)	41	~20,000
Engineering & Technological Sciences (D₁–D₁₉₀)	~190	~30,000
Humanities & Social Sciences (E₁–E₁₆₀)	~160	~17,000

Pool construction follows a multi-stage pipeline.

Stage 1: Transcription collects raw questions from textbooks, question banks, and the web. Experts and LLM filters flag formatting and factual errors, after which manual fixes are applied. Each instance is assigned to one of 163 initial subfields.

Stage 2: Expansion identifies under-covered subfields and commissions specialists to write additional high-quality items. Coarse quality control is performed through peer review and LLM checks. The reported result is coverage expanded to 500+ subfields.

Stage 3: Categorization runs $K$ top-tier LLMs, such as GPT-4 and Claude, in zero-shot mode on every instance. For instance $j$, the benchmark defines

$$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$

Difficulty is then assigned by pre-set thresholds $\tau_{\text{easy}}$ and $\tau_{\text{hard}}$, with an example setting of $0.8$ and $0.3$: $$\text{difficulty}(j)= \begin{cases} \text{easy}, & S_j \ge \tau_{\text{easy}},\ \text{medium}, & \tau_{\text{hard}} \le S_j < \tau_{\text{easy}},\ \text{hard}, & S_j < \tau_{\text{hard}}. \end{cases}$$

A further Data Refinement stage implements the benchmark’s notion of Rigor. Items identified as “too-easy” or ambiguous are routed into three parallel branches: Enhancement by Distraction, which auto-generates plausible distractors followed by expert vetting; Enrichment by Cross-Disciplinary, which injects context from other fields using an LLM draft followed by expert polishing; and Expert-Driven Refinement, which fully rewrites questions for multi-step complexity (Wang et al., 22 Jul 2025).

The stated architecture is notable because it does not treat coverage as merely topical. Difficulty, ambiguity reduction, and cross-disciplinary restructuring are part of the pool-building process itself. A plausible implication is that EESE operationalizes scientific evaluation as a joint problem of subject breadth and cognitive demand.

3. Public EESE subset: sampling, validation, and refresh

The public benchmark consists of 500 instances sampled from the pool. Sampling is stratified random sampling by discipline, difficulty tier, question format, and subfield in order to preserve the pool’s distribution. If $N_{d,f,t}$ denotes the number of pool instances in discipline $d$, format $j$0, and tier $j$1, the benchmark defines

$j$2

After automatic sampling, each candidate item undergoes expert review for redundancy and clarity. Redundancy refers to near-duplicates, while clarity concerns format and unambiguous phrasing (Wang et al., 22 Jul 2025).

The subset is not static. The reported resampling cadence is every 6 months (or on each new major model release). The rationale is explicit: by never reusing the same subset of 500, the chance that any static test question appears in training data is minimized. The benchmark also specifies a hold-out protocol in which, once deployed, an EESE subset is removed from public sources and only refreshed versions circulate.

This periodic refresh is central to the meaning of “ever-evolving.” The benchmark is therefore not only a dataset but also a release protocol. A common misconception in benchmark design is that contamination can be solved entirely by keeping one test set hidden; EESE instead treats leakage mitigation as an ongoing sampling and retirement process.

4. Evaluation methodology and reported metrics

The benchmark reports experiments on 32 open- and closed-source models, including “thinking” series and standard models. The listed families include OpenAI models such as O3, O3-mini, GPT-4o, GPT-4.1; Google models such as Gemini-2.5-pro and Gemini-1.5-pro; Anthropic models such as Claude-3-7-sonnet and Claude-3-5-sonnet; xAI models such as Grok-4, Grok-3, Grok-2; DeepSeek models such as R1 and V3; and additional families including Alibaba Qwen, Meta Llama-3, InternLM, GLM-4, Gemma, Phi-4, and Mistral-large (Wang et al., 22 Jul 2025).

Scientific performance is reported along two axes. First, there are discipline-specific scores across SSH, AS, MS, NS, and ETS, plus an overall score. Second, the benchmark treats question formats as proxies for cognitive or functional dimensions. True/false and single-choice formats target knowledge retrieval, while fill-in, open-ended, distractors, and cross-disciplinary formats target multi-step reasoning.

The accuracy metric for discipline $j$3 is

$j$4

The overall score is the uniform average across the five disciplines: $j$5

To evaluate whether the 500-instance EESE subset preserves model ordering from the full pool, the benchmark uses Spearman Rank Correlation (SRCC): $j$6 where $j$7 and $j$8 are ranks under two settings and $j$9 is the number of models.

This methodology makes the benchmark simultaneously diagnostic and compressive: diagnostic because it resolves performance by discipline and question format, and compressive because it reduces the large hidden reservoir to a small public test while retaining ranking fidelity.

5. Empirical findings

The reported overall EESE accuracies identify O3 as the top performer at approximately 40.3%, followed by Gemini-2.5-pro at approximately 38.1%, and GPT-4o at approximately 23.9%. The benchmark also reports a human expert baseline of approximately 84.7%, indicating a large gap between current model performance and expert-level scientific answering (Wang et al., 22 Jul 2025).

Performance varies substantially by discipline. Models reportedly do best on SSH and ETS, at approximately 35–50%, and worst on MS and NS, at approximately 20–36%. The benchmark interprets these differences as evidence that EESE can differentiate model strengths and weaknesses across scientific fields.

A further reported result concerns the trade-off between “thinking” and standard models: thinking models achieve about +1.3× performance but incur about +4.2× latency and +4.5× cost. The benchmark also reports that SRCC(EESE vs. Pool) diagonals are approximately 0.9–0.95, which is presented as evidence that the 500-question subset faithfully mirrors the full-pool rankings.

Model	Speed (s/q)	Cost (USD/10 Q)
Best w/o Thinking	14.2	0.049
O3 (w/ Thinking)	15.1	0.125
Gemini-2.5-pro	19.6	0.442
Grok-4	41.5	0.440
DeepSeek-R1	107.5	0.039

The same table also reports Overall Acc, rel. Speed, rel. Cost, and rel. Acc: Best w/o Thinking has $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$0, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$1, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$2, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$3; O3 has $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$4, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$5, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$6, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$7; Gemini-2.5-pro has $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$8, $$S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.$$9, $\tau_{\text{easy}}$0, $\tau_{\text{easy}}$1; Grok-4 has $\tau_{\text{easy}}$2, $\tau_{\text{easy}}$3, $\tau_{\text{easy}}$4, $\tau_{\text{easy}}$5; and DeepSeek-R1 has $\tau_{\text{easy}}$6, $\tau_{\text{easy}}$7, $\tau_{\text{easy}}$8, $\tau_{\text{easy}}$9 (Wang et al., 22 Jul 2025).

These findings situate EESE as both a capability benchmark and a resource-efficiency benchmark. A plausible implication is that benchmark choice may depend not only on validity and discriminative power but also on whether the evaluation regime must support repeated, budget-constrained model comparisons.

6. Leakage mitigation and robustness

Leakage mitigation is a primary design objective. EESE specifies embedding-based near-duplicate removal over release history. For each new instance $\tau_{\text{hard}}$0 and release history $\tau_{\text{hard}}$1, the benchmark requires

$\tau_{\text{hard}}$2

with an example threshold $\tau_{\text{hard}}$3. The rejection rule is stated explicitly: $\tau_{\text{hard}}$4

This is paired with a hard hold-out rule: once an instance is drawn into any public EESE release, it is never returned to the pool for future sampling. In addition, the benchmark applies expert verification through a random 10% audit of each batch to catch inadvertently recycled or web-scraped items (Wang et al., 22 Jul 2025).

The benchmark’s robustness claims extend beyond leakage prevention. The source description emphasizes that a very large, expertly-curated reservoir (>100K) allows repeated draws of non-overlapping 500-question evaluations. It also states that the Modular Data Engine + Refinement design permits adding new question types, such as multimodal items, or new subfields without redesigning the entire pipeline.

Taken together, these design choices define robustness in two senses: contamination resistance and extensibility. This suggests a view of benchmark maintenance as continuous systems engineering rather than one-time dataset publication.

7. Significance, interpretation, and limitations

EESE is presented as a robust, scalable, and forward-compatible solution for science benchmark design. Its central claim is that dynamic resampling offers a more realistic measure of generalization than a static public science test, especially under conditions where foundation models may absorb benchmark content through training or post-training exposure (Wang et al., 22 Jul 2025).

The benchmark also provides a framework for fine-grained discipline- and difficulty-level analysis, which is described as useful for identifying precise weaknesses such as cross-disciplinary reasoning. In practical terms, the inclusion of cost and latency measurements means that benchmark outcomes are not limited to raw capability comparisons; they also inform decisions about whether “thinking” modules deliver sufficient gains under resource constraints.

Several misconceptions are addressed by the benchmark’s design. One is the assumption that a large hidden pool alone is enough to ensure validity; EESE instead combines hidden data, resampling, duplicate filtering, retirement rules, and expert audits. Another is the assumption that a compact benchmark must sacrifice representativeness; the reported SRCC values near 0.9–0.95 are used to argue that the 500-instance subset preserves full-pool rank order effectively.

At the same time, the reported empirical results underscore that benchmark resilience does not imply solved scientific competence. The approximately 84.7% human expert baseline compared with model scores led by approximately 40.3% indicates that substantial capability gaps remain. Within the terms of the benchmark, EESE therefore functions less as a certificate of mastery than as a controlled instrument for tracking uneven progress across disciplines, formats, and evaluation regimes.