---
title: 'CORE-Bench OOD: Field Shift in Reproducibility'
url: https://www.emergentmind.com/topics/core-bench-ood
type: topic
---

# CORE-Bench OOD: Field Shift in Reproducibility

CORE-Bench OOD is the out-of-distribution extension of CORE-Bench v1.1 for evaluating computational reproducibility agents under a **field distribution shift** rather than merely under the original benchmark mix. It was introduced as part of a broader argument that benchmark saturation should not automatically trigger retirement and replacement: once accuracy approaches ceiling, evaluation can still probe construct validity, out-of-distribution generalizability, efficiency, reliability, scaffold effects, and human-agent collaboration. Within that framework, CORE-Bench OOD preserves the underlying task structure of CORE-Bench v1.1 while changing the disciplinary composition, thereby testing whether apparently saturated performance transfers beyond the original field mix [2606.26158].

## 1. Origins and benchmark lineage

CORE-Bench was originally introduced as a benchmark for **computational reproducibility agents**: systems that must take a published paper’s code repository and reproduce results well enough to answer questions about the reproduced outputs. The 2024 benchmark contains **270 tasks** derived from **90 papers** across **computer science**, **social science**, and **medicine**, with three difficulty levels that range from output extraction to full reproduction from a README alone [2409.11363].

The later post-saturation study re-examines this benchmark family after agent performance improved substantially. In that setting, the authors introduce **CORE-Bench v1.1** and its out-of-distribution extension, **CORE-Bench OOD**, as part of a broader evaluation program that studies what remains measurable after headline accuracy saturates [2606.26158].

The name should be distinguished from the unrelated code-retrieval benchmark also called **CORE-Bench**, whose full title is “CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding” [2606.11864]. In the reproducibility-benchmark line, “CORE” refers to computational reproducibility; in CORE-Bench OOD, “OOD” refers to a **shift in benchmark task distribution across scientific fields**, not to post-hoc sample scoring in the usual OOD-detection sense.

## 2. Motivation: benchmark saturation and the role of OOD evaluation

The central motivation for CORE-Bench OOD is a critique of the standard **retire-and-replace** response to benchmark saturation. The paper argues that when a benchmark’s accuracy saturates, replacing it with a harder version privileges accuracy alone and neglects several other dimensions of agent performance, including **construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration** [2606.26158].

Within that argument, CORE-Bench OOD is introduced to test a specific question: whether the near-ceiling performance observed on CORE-Bench v1.1 survives a meaningful distribution shift. The paper frames this as a defense against **benchmark-specific adaptation**. An agent or scaffold may perform well because it has adapted to the peculiarities of the original benchmark composition rather than because it has acquired a more general reproducibility capability. CORE-Bench OOD therefore asks whether performance on CORE-Bench v1.1 **transfers under a field distribution shift** [2606.26158].

The shift is treated as especially consequential because disciplines differ in **repository organization, software ecosystems, manuscript conventions, and computational workflows**. A plausible implication is that a benchmark can remain informative after saturation if it is embedded in a validity-oriented evaluation program rather than treated as a one-dimensional accuracy leaderboard.

## 3. Construction and disciplinary composition

CORE-Bench Hard and CORE-Bench v1.1 contain tasks only from **computer science**, **medical science**, and **social science**. CORE-Bench OOD becomes out-of-distribution by changing that field mix to **physics**, **engineering**, **economics**, and **computer science**, while preserving the underlying task structure of v1.1 [2606.26158].

The final benchmark contains **19 tasks**. Its field distribution is reported as follows:

| Field | Tasks |
|---|---:|
| Economics | 2 |
| Engineering | 10 |
| Physics | 5 |
| Computer science | 2 |

The paper notes that **one of the computer science tasks has a runtime of around 50 minutes** [2606.26158].

Its construction follows the same log-analysis methodology used for CORE-Bench v1.1. The reported pipeline is:

1. an initial pool of **30 OOD tasks** was written at the same time as CORE-Bench Hard;
2. these tasks were evaluated using **CORE-Agent (Opus 4.5 and 4.6)** and **OpenCode (GPT-5.2)**;
3. automated and manual log analysis then **removed 12 tasks**, **edited 8 tasks**, and **added 6 tasks**, producing a **24-task subset**;
4. additional log analysis of incorrect tasks across **12 Codex CLI runs** led to the removal of **5 additional tasks** and the **regrading of 1 task**;
5. the result was the final **19-task** CORE-Bench OOD suite [2606.26158].

The curation logic is explicitly validity-oriented rather than difficulty-oriented. The same kinds of issues sought in v1.1 were also sought here: **incorrect ground truths, grading issues, task underspecification, unsolvable tasks, and shortcut opportunities**. The benchmark is described as an **active benchmark**, because log analysis is **not exhaustive** and additional threats may surface under new agent runs [2606.26158].

## 4. Evaluation protocol and saturation criterion

For OOD evaluation, the paper reports running **12 Codex CLI agents** while varying the **model**, **reasoning effort**, and **number of subagents (`max_thr`)**. Evaluation was conducted on **Azure VMs** using the **HAL evaluation harness**, with **full filesystem and web access** [2606.26158].

The operational settings reported for this evaluation stack are:

- **per-task timeout for Codex CLI / Claude Code / OpenCode:** **45 minutes**
- **per-task timeout for CORE-Agent:** **5 hours**
- **max retries for Codex CLI / Claude Code / OpenCode:** **3**
- **CORE-Agent:** **max steps 200**, **max retries 1** [2606.26158]

To determine whether performance is saturated, the paper uses a statistical criterion based on the gap between the top agent and the \(k\)-th ranked agent. The top \(k\) agents are treated as statistically indistinguishable in accuracy when

$$
s_1 - s_k \leq z \cdot \mathrm{SE}_\Delta
$$

and the reported setting uses \(\alpha = 0.5\), \(z = 1.96\), and \(k = 5\) [2606.26158].

The paper’s reported saturation summary is:

| Benchmark | \(s_1\) | \(s_5\) | \(\Delta = s_1 - s_5\) | \(z \cdot \mathrm{SE}_\Delta\) | Saturated |
|---|---:|---:|---:|---:|---:|
| CORE-Bench v1.1 | 1 | 0.9744 | 0.0256 | 0.1240 | True |
| CORE-Bench OOD | 1 | 0.8947 | 0.1053 | 0.2881 | True |

Under this operational definition, the paper concludes that the **top five agents are statistically indistinguishable on both CORE-Bench v1.1 and CORE-Bench OOD** [2606.26158].

## 5. Empirical results on out-of-distribution generalizability

The central empirical finding is that **accuracy saturation transfers across the field shift**. The paper states that “the top five agents obtain statistically indistinguishable accuracies on CORE-Bench OOD,” and interprets this as evidence that “accuracy saturation on CORE-Bench v1.1 translates across a discipline distribution shift” [2606.26158].

The reported CORE-Bench OOD accuracies for the 12 Codex CLI variants are:

| Scaffold / setting | Model configuration | Accuracy |
|---|---|---:|
| Codex CLI (default) | GPT-5 (medium) | 89.5% |
| Codex CLI (default) | GPT-5.1 (medium) | 94.7% |
| Codex CLI (default) | GPT-5.2 (medium) | 100.0% |
| Codex CLI (default) | GPT-5.3-Codex (medium) | 89.5% |
| Codex CLI (default) | GPT-5.4 (low) | 84.2% |
| Codex CLI (default) | GPT-5.4 (medium) | 89.5% |
| Codex CLI (default) | GPT-5.4 (high) | 89.5% |
| Codex CLI (default) | GPT-5.4 (xhigh) | 100.0% |
| Codex CLI (\( \text{max\_thr}=1 \)) | GPT-5.4 (medium) | 94.7% |
| Codex CLI (\( \text{max\_thr}=3 \)) | GPT-5.4 (medium) | 89.5% |
| Codex CLI (\( \text{max\_thr}=6 \)) | GPT-5.4 (medium) | 84.2% |
| Codex CLI (\( \text{max\_thr}=9 \)) | GPT-5.4 (medium) | 84.2% |

Two settings reached **100.0%**: **GPT-5.2 (medium)** and **GPT-5.4 (xhigh)**. Several other settings clustered at **89.5%** or **94.7%**, and even the lowest reported accuracies remained at **84.2%** [2606.26158].

In the paper’s interpretation, these results indicate that the benchmark’s saturation is not merely an artifact of the original field mix. More cautiously stated, the reported evidence suggests that a disciplinary composition shift from the original benchmark domains to physics, engineering, and economics is insufficient, by itself, to restore useful separation among top-performing agents on headline accuracy.

## 6. Interpretation, scope, and limitations

CORE-Bench OOD occupies a specific place in the evaluation ecology of agent benchmarks. It is not introduced as a generic OOD-detection benchmark and does not define a conventional train/test novelty protocol over samples. Instead, it is an **out-of-distribution task suite** for **computational reproducibility agents**, where “OOD” denotes a **field distribution shift** in the benchmark task population [2606.26158].

This framing matters for construct validity. CORE-Bench OOD is designed to test whether success on CORE-Bench v1.1 reflects capability that transfers across disciplines, rather than capability overfit to the benchmark’s original scientific mix. Within the same paper, however, OOD evaluation is only one part of a broader post-saturation program. Efficiency, reliability, scaffold effects, and human-agent collaboration are treated as additional axes of measurement rather than as consequences derivable from OOD accuracy alone [2606.26158].

The benchmark’s limitations are explicit. The authors state that **log analysis is not exhaustive**, that CORE-Bench OOD is an **active and evolving benchmark**, and that the suite is still **small**, containing only **19 tasks**. They also emphasize that it tests primarily **field shift**, not every possible form of distribution shift, and that OOD results support conclusions about **accuracy saturation** rather than serving as a complete measure of real-world utility [2606.26158].

A further interpretive caution is nomenclatural. Because another 2026 benchmark also uses the name CORE-Bench for an agentic code-retrieval setting, references to “CORE-Bench OOD” require disambiguation by topic and citation context [2606.11864]. In the reproducibility-benchmark literature, CORE-Bench OOD specifically denotes the post-saturation field-shift suite attached to CORE-Bench v1.1.

Taken together, CORE-Bench OOD functions less as a harder successor benchmark than as an instrument for testing a particular failure mode of saturated evaluation: the possibility that ceiling-level performance is benchmark-specific rather than transferable. Its main reported result is that, under the paper’s statistical criterion, saturation persists even after the field distribution shifts [2606.26158].

Source: https://www.emergentmind.com/topics/core-bench-ood