---
title: Ever-Evolving Science Exam Benchmark
url: https://www.emergentmind.com/topics/ever-evolving-science-exam-eese
type: topic
---

# Ever-Evolving Science Exam Benchmark

Searching arXiv for the specified paper to ground the article and confirm bibliographic details.
The **Ever-Evolving Science Exam (EESE)** is a dynamic benchmark for evaluating scientific understanding in foundation models. It was introduced to address two recurring problems in science benchmarking: **data leakage risks** that compromise validity and **evaluation inefficiency** caused by large-scale testing. EESE combines a non-public **EESE-Pool** containing over 100K expertly constructed science question–answer pairs across 5 disciplines and 500+ subfields with a periodically refreshed public **EESE** subset of 500 instances. The design is explicitly organized around **Range**, **Reach**, and **Rigor**, and is intended to provide a leakage-resilient, low-overhead, and forward-compatible measure of how well foundation models handle science questions [2507.16514].

## 1. Definition and benchmark design

EESE consists of two linked components. The first is **EESE-Pool**, a non-public reservoir with more than 100,000 science instances. The second is **EESE**, a 500-instance subset sampled from the pool and periodically updated. The pool is the source of breadth and long-term freshness; the subset is the operational benchmark for routine model evaluation [2507.16514].

The benchmark is motivated by the observation that existing science benchmarks have made progress toward broad **Range**, wide **Reach**, and high **Rigor**, but often remain vulnerable to contamination from public exposure and costly to administer at scale. EESE addresses these issues by separating a large, privately maintained source pool from a small public evaluation surface. This suggests a benchmark architecture in which statistical coverage and practical deployability are decoupled rather than treated as competing objectives.

The benchmark’s scientific scope is divided into five parent disciplines: **Natural Sciences**, **Agricultural Sciences**, **Medical Sciences**, **Engineering & Technological Sciences**, and **Humanities & Social Sciences**. In evaluation, these are reported as **NS**, **AS**, **MS**, **ETS**, and **SSH**, together with an overall score [2507.16514].

## 2. EESE-Pool construction and coverage

The pool is described as having scale beyond 100,000 question–answer pairs and coverage over 500+ subfields. Its disciplinary composition is explicitly enumerated in the source description.

| Parent discipline | Subfields | Approximate instances |
|---|---:|---:|
| Natural Sciences (A₁–A₉₁) | 91 | ~25,000 |
| Agricultural Sciences (B₁–B₂₄) | 24 | ~8,000 |
| Medical Sciences (C₁–C₄₁) | 41 | ~20,000 |
| Engineering & Technological Sciences (D₁–D₁₉₀) | ~190 | ~30,000 |
| Humanities & Social Sciences (E₁–E₁₆₀) | ~160 | ~17,000 |

Pool construction follows a **multi-stage pipeline**.

**Stage 1: Transcription** collects raw questions from textbooks, question banks, and the web. Experts and LLM filters flag formatting and factual errors, after which manual fixes are applied. Each instance is assigned to one of **163 initial subfields**.

**Stage 2: Expansion** identifies under-covered subfields and commissions specialists to write additional high-quality items. Coarse quality control is performed through peer review and LLM checks. The reported result is coverage expanded to **500+ subfields**.

**Stage 3: Categorization** runs \(K\) top-tier LLMs, such as GPT-4 and Claude, in zero-shot mode on every instance. For instance \(j\), the benchmark defines
\[
S_j = \frac{1}{K}\sum_{m=1}^K \mathbf{1}\{\text{model}_m \text{ answers } j \text{ correctly}\}.
\]
Difficulty is then assigned by pre-set thresholds \(\tau_{\text{easy}}\) and \(\tau_{\text{hard}}\), with an example setting of \(0.8\) and \(0.3\):
\[
\text{difficulty}(j)=
\begin{cases}
\text{easy}, & S_j \ge \tau_{\text{easy}},\\
\text{medium}, & \tau_{\text{hard}} \le S_j < \tau_{\text{easy}},\\
\text{hard}, & S_j < \tau_{\text{hard}}.
\end{cases}
\]

A further **Data Refinement** stage implements the benchmark’s notion of **Rigor**. Items identified as “too-easy” or ambiguous are routed into three parallel branches: **Enhancement by Distraction**, which auto-generates plausible distractors followed by expert vetting; **Enrichment by Cross-Disciplinary**, which injects context from other fields using an LLM draft followed by expert polishing; and **Expert-Driven Refinement**, which fully rewrites questions for multi-step complexity [2507.16514].

The stated architecture is notable because it does not treat coverage as merely topical. Difficulty, ambiguity reduction, and cross-disciplinary restructuring are part of the pool-building process itself. A plausible implication is that EESE operationalizes scientific evaluation as a joint problem of subject breadth and cognitive demand.

## 3. Public EESE subset: sampling, validation, and refresh

The public benchmark consists of **500 instances** sampled from the pool. Sampling is **stratified random sampling** by **discipline, difficulty tier, question format, and subfield** in order to preserve the pool’s distribution. If \(N_{d,f,t}\) denotes the number of pool instances in discipline \(d\), format \(f\), and tier \(t\), the benchmark defines
\[
n_{d,f,t} = \left\lfloor 500 \times \frac{N_{d,f,t}}{|\text{Pool}|} \right\rfloor.
\]

After automatic sampling, each candidate item undergoes expert review for **redundancy** and **clarity**. Redundancy refers to near-duplicates, while clarity concerns format and unambiguous phrasing [2507.16514].

The subset is not static. The reported **resampling cadence** is **every 6 months (or on each new major model release)**. The rationale is explicit: by never reusing the same subset of 500, the chance that any static test question appears in training data is minimized. The benchmark also specifies a **hold-out protocol** in which, once deployed, an EESE subset is removed from public sources and only refreshed versions circulate.

This periodic refresh is central to the meaning of “ever-evolving.” The benchmark is therefore not only a dataset but also a release protocol. A common misconception in benchmark design is that contamination can be solved entirely by keeping one test set hidden; EESE instead treats leakage mitigation as an ongoing sampling and retirement process.

## 4. Evaluation methodology and reported metrics

The benchmark reports experiments on **32 open- and closed-source models**, including “thinking” series and standard models. The listed families include OpenAI models such as **O3, O3-mini, GPT-4o, GPT-4.1**; Google models such as **Gemini-2.5-pro** and **Gemini-1.5-pro**; Anthropic models such as **Claude-3-7-sonnet** and **Claude-3-5-sonnet**; xAI models such as **Grok-4, Grok-3, Grok-2**; DeepSeek models such as **R1** and **V3**; and additional families including **Alibaba Qwen**, **Meta Llama-3**, **InternLM**, **GLM-4**, **Gemma**, **Phi-4**, and **Mistral-large** [2507.16514].

Scientific performance is reported along two axes. First, there are discipline-specific scores across **SSH, AS, MS, NS, and ETS**, plus an overall score. Second, the benchmark treats question formats as proxies for cognitive or functional dimensions. **True/false** and **single-choice** formats target **knowledge retrieval**, while **fill-in**, **open-ended**, **distractors**, and **cross-disciplinary** formats target **multi-step reasoning**.

The accuracy metric for discipline \(d\) is
\[
\mathrm{Acc}_d = \frac{\#\mathrm{correct}_d}{\#\mathrm{questions}_d}.
\]
The overall score is the **uniform average** across the five disciplines:
\[
\mathrm{Acc}_{\mathrm{overall}} = \frac{1}{5}\sum_{d\in\{\mathrm{SSH,AS,MS,NS,ETS}\}} \mathrm{Acc}_d.
\]

To evaluate whether the 500-instance EESE subset preserves model ordering from the full pool, the benchmark uses **Spearman Rank Correlation (SRCC)**:
\[
\rho = 1 - \frac{6\sum_i (r_i - s_i)^2}{M(M^2-1)},
\]
where \(r_i\) and \(s_i\) are ranks under two settings and \(M\) is the number of models.

This methodology makes the benchmark simultaneously diagnostic and compressive: diagnostic because it resolves performance by discipline and question format, and compressive because it reduces the large hidden reservoir to a small public test while retaining ranking fidelity.

## 5. Empirical findings

The reported overall EESE accuracies identify **O3** as the top performer at approximately **40.3%**, followed by **Gemini-2.5-pro** at approximately **38.1%**, and **GPT-4o** at approximately **23.9%**. The benchmark also reports a **human expert baseline** of approximately **84.7%**, indicating a large gap between current model performance and expert-level scientific answering [2507.16514].

Performance varies substantially by discipline. Models reportedly do best on **SSH** and **ETS**, at approximately **35–50%**, and worst on **MS** and **NS**, at approximately **20–36%**. The benchmark interprets these differences as evidence that EESE can differentiate model strengths and weaknesses across scientific fields.

A further reported result concerns the trade-off between “thinking” and standard models: **thinking models** achieve about **+1.3× performance** but incur about **+4.2× latency** and **+4.5× cost**. The benchmark also reports that **SRCC(EESE vs. Pool) diagonals** are approximately **0.9–0.95**, which is presented as evidence that the 500-question subset faithfully mirrors the full-pool rankings.

| Model | Speed (s/q) | Cost (USD/10 Q) |
|---|---:|---:|
| Best w/o Thinking | 14.2 | 0.049 |
| O3 (w/ Thinking) | 15.1 | 0.125 |
| Gemini-2.5-pro | 19.6 | 0.442 |
| Grok-4 | 41.5 | 0.440 |
| DeepSeek-R1 | 107.5 | 0.039 |

The same table also reports **Overall Acc**, **rel. Speed**, **rel. Cost**, and **rel. Acc**: Best w/o Thinking has \(0.2578\), \(1.0\), \(1.0\), \(1.0\); O3 has \(0.4025\), \(1.06\), \(2.55\), \(1.56\); Gemini-2.5-pro has \(0.3813\), \(1.38\), \(9.00\), \(1.48\); Grok-4 has \(0.3442\), \(2.92\), \(8.94\), \(1.33\); and DeepSeek-R1 has \(0.3251\), \(7.57\), \(0.79\), \(1.26\) [2507.16514].

These findings situate EESE as both a capability benchmark and a resource-efficiency benchmark. A plausible implication is that benchmark choice may depend not only on validity and discriminative power but also on whether the evaluation regime must support repeated, budget-constrained model comparisons.

## 6. Leakage mitigation and robustness

Leakage mitigation is a primary design objective. EESE specifies **embedding-based near-duplicate removal** over release history. For each new instance \(j\) and release history \(H\), the benchmark requires
\[
\max_{h\in H}\cos(\mathrm{emb}_j,\mathrm{emb}_h)\le \tau_{\mathrm{dup}},
\]
with an example threshold \(\tau_{\mathrm{dup}} = 0.85\). The rejection rule is stated explicitly:
\[
\text{reject if } \max_{h\in H}\cos(\mathrm{emb}_j,\mathrm{emb}_h) > \tau_{\mathrm{dup}}.
\]

This is paired with a **hard hold-out** rule: once an instance is drawn into any public EESE release, it is never returned to the pool for future sampling. In addition, the benchmark applies **expert verification** through a **random 10% audit of each batch** to catch inadvertently recycled or web-scraped items [2507.16514].

The benchmark’s robustness claims extend beyond leakage prevention. The source description emphasizes that a **very large, expertly-curated reservoir (>100K)** allows repeated draws of non-overlapping 500-question evaluations. It also states that the **Modular Data Engine + Refinement** design permits adding new question types, such as multimodal items, or new subfields without redesigning the entire pipeline.

Taken together, these design choices define robustness in two senses: contamination resistance and extensibility. This suggests a view of benchmark maintenance as continuous systems engineering rather than one-time dataset publication.

## 7. Significance, interpretation, and limitations

EESE is presented as a **robust, scalable, and forward-compatible solution for science benchmark design**. Its central claim is that **dynamic resampling** offers a more realistic measure of generalization than a static public science test, especially under conditions where foundation models may absorb benchmark content through training or post-training exposure [2507.16514].

The benchmark also provides a framework for **fine-grained discipline- and difficulty-level analysis**, which is described as useful for identifying precise weaknesses such as **cross-disciplinary reasoning**. In practical terms, the inclusion of cost and latency measurements means that benchmark outcomes are not limited to raw capability comparisons; they also inform decisions about whether “thinking” modules deliver sufficient gains under resource constraints.

Several misconceptions are addressed by the benchmark’s design. One is the assumption that a large hidden pool alone is enough to ensure validity; EESE instead combines hidden data, resampling, duplicate filtering, retirement rules, and expert audits. Another is the assumption that a compact benchmark must sacrifice representativeness; the reported **SRCC** values near **0.9–0.95** are used to argue that the 500-instance subset preserves full-pool rank order effectively.

At the same time, the reported empirical results underscore that benchmark resilience does not imply solved scientific competence. The approximately **84.7%** human expert baseline compared with model scores led by approximately **40.3%** indicates that substantial capability gaps remain. Within the terms of the benchmark, EESE therefore functions less as a certificate of mastery than as a controlled instrument for tracking uneven progress across disciplines, formats, and evaluation regimes.

Source: https://www.emergentmind.com/topics/ever-evolving-science-exam-eese