CORE-Bench OOD: Field Shift in Reproducibility

Updated 4 July 2026

CORE-Bench OOD is an out-of-distribution extension of CORE-Bench v1.1 designed to evaluate computational reproducibility agents under a field distribution shift.
The benchmark preserves the core task structure while changing its disciplinary mix to include physics, engineering, and economics alongside computer science.
Empirical results show that accuracy saturation persists across field shifts, indicating that high accuracy may mask underlying construct validity issues.

CORE-Bench OOD is the out-of-distribution extension of CORE-Bench v1.1 for evaluating computational reproducibility agents under a field distribution shift rather than merely under the original benchmark mix. It was introduced as part of a broader argument that benchmark saturation should not automatically trigger retirement and replacement: once accuracy approaches ceiling, evaluation can still probe construct validity, out-of-distribution generalizability, efficiency, reliability, scaffold effects, and human-agent collaboration. Within that framework, CORE-Bench OOD preserves the underlying task structure of CORE-Bench v1.1 while changing the disciplinary composition, thereby testing whether apparently saturated performance transfers beyond the original field mix (Nadgir et al., 23 Jun 2026).

1. Origins and benchmark lineage

CORE-Bench was originally introduced as a benchmark for computational reproducibility agents: systems that must take a published paper’s code repository and reproduce results well enough to answer questions about the reproduced outputs. The 2024 benchmark contains 270 tasks derived from 90 papers across computer science, social science, and medicine, with three difficulty levels that range from output extraction to full reproduction from a README alone (Siegel et al., 2024).

The later post-saturation study re-examines this benchmark family after agent performance improved substantially. In that setting, the authors introduce CORE-Bench v1.1 and its out-of-distribution extension, CORE-Bench OOD, as part of a broader evaluation program that studies what remains measurable after headline accuracy saturates (Nadgir et al., 23 Jun 2026).

The name should be distinguished from the unrelated code-retrieval benchmark also called CORE-Bench, whose full title is “CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding” (Zhang et al., 10 Jun 2026). In the reproducibility-benchmark line, “CORE” refers to computational reproducibility; in CORE-Bench OOD, “OOD” refers to a shift in benchmark task distribution across scientific fields, not to post-hoc sample scoring in the usual OOD-detection sense.

2. Motivation: benchmark saturation and the role of OOD evaluation

The central motivation for CORE-Bench OOD is a critique of the standard retire-and-replace response to benchmark saturation. The paper argues that when a benchmark’s accuracy saturates, replacing it with a harder version privileges accuracy alone and neglects several other dimensions of agent performance, including construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration (Nadgir et al., 23 Jun 2026).

Within that argument, CORE-Bench OOD is introduced to test a specific question: whether the near-ceiling performance observed on CORE-Bench v1.1 survives a meaningful distribution shift. The paper frames this as a defense against benchmark-specific adaptation. An agent or scaffold may perform well because it has adapted to the peculiarities of the original benchmark composition rather than because it has acquired a more general reproducibility capability. CORE-Bench OOD therefore asks whether performance on CORE-Bench v1.1 transfers under a field distribution shift (Nadgir et al., 23 Jun 2026).

The shift is treated as especially consequential because disciplines differ in repository organization, software ecosystems, manuscript conventions, and computational workflows. A plausible implication is that a benchmark can remain informative after saturation if it is embedded in a validity-oriented evaluation program rather than treated as a one-dimensional accuracy leaderboard.

3. Construction and disciplinary composition

CORE-Bench Hard and CORE-Bench v1.1 contain tasks only from computer science, medical science, and social science. CORE-Bench OOD becomes out-of-distribution by changing that field mix to physics, engineering, economics, and computer science, while preserving the underlying task structure of v1.1 (Nadgir et al., 23 Jun 2026).

The final benchmark contains 19 tasks. Its field distribution is reported as follows:

Field	Tasks
Economics	2
Engineering	10
Physics	5
Computer science	2

The paper notes that one of the computer science tasks has a runtime of around 50 minutes (Nadgir et al., 23 Jun 2026).

Its construction follows the same log-analysis methodology used for CORE-Bench v1.1. The reported pipeline is:

an initial pool of 30 OOD tasks was written at the same time as CORE-Bench Hard;
these tasks were evaluated using CORE-Agent (Opus 4.5 and 4.6) and OpenCode (GPT-5.2);
automated and manual log analysis then removed 12 tasks, edited 8 tasks, and added 6 tasks, producing a 24-task subset;
additional log analysis of incorrect tasks across 12 Codex CLI runs led to the removal of 5 additional tasks and the regrading of 1 task;
the result was the final 19-task CORE-Bench OOD suite (Nadgir et al., 23 Jun 2026).

The curation logic is explicitly validity-oriented rather than difficulty-oriented. The same kinds of issues sought in v1.1 were also sought here: incorrect ground truths, grading issues, task underspecification, unsolvable tasks, and shortcut opportunities. The benchmark is described as an active benchmark, because log analysis is not exhaustive and additional threats may surface under new agent runs (Nadgir et al., 23 Jun 2026).

4. Evaluation protocol and saturation criterion

For OOD evaluation, the paper reports running 12 Codex CLI agents while varying the model, reasoning effort, and number of subagents (max_thr). Evaluation was conducted on Azure VMs using the HAL evaluation harness, with full filesystem and web access (Nadgir et al., 23 Jun 2026).

The operational settings reported for this evaluation stack are:

per-task timeout for Codex CLI / Claude Code / OpenCode: 45 minutes
per-task timeout for CORE-Agent: 5 hours
max retries for Codex CLI / Claude Code / OpenCode: 3
CORE-Agent: max steps 200, max retries 1 (Nadgir et al., 23 Jun 2026)

To determine whether performance is saturated, the paper uses a statistical criterion based on the gap between the top agent and the $k$ -th ranked agent. The top $k$ agents are treated as statistically indistinguishable in accuracy when

$s_1 - s_k \leq z \cdot \mathrm{SE}_\Delta$

and the reported setting uses $\alpha = 0.5$ , $z = 1.96$ , and $k = 5$ (Nadgir et al., 23 Jun 2026).

The paper’s reported saturation summary is:

Benchmark	$s_1$	$s_5$	$\Delta = s_1 - s_5$	$z \cdot \mathrm{SE}_\Delta$	Saturated
CORE-Bench v1.1	1	0.9744	0.0256	0.1240	True
CORE-Bench OOD	1	0.8947	0.1053	0.2881	True

Under this operational definition, the paper concludes that the top five agents are statistically indistinguishable on both CORE-Bench v1.1 and CORE-Bench OOD (Nadgir et al., 23 Jun 2026).

5. Empirical results on out-of-distribution generalizability

The central empirical finding is that accuracy saturation transfers across the field shift. The paper states that “the top five agents obtain statistically indistinguishable accuracies on CORE-Bench OOD,” and interprets this as evidence that “accuracy saturation on CORE-Bench v1.1 translates across a discipline distribution shift” (Nadgir et al., 23 Jun 2026).

The reported CORE-Bench OOD accuracies for the 12 Codex CLI variants are:

Scaffold / setting	Model configuration	Accuracy
Codex CLI (default)	GPT-5 (medium)	89.5%
Codex CLI (default)	GPT-5.1 (medium)	94.7%
Codex CLI (default)	GPT-5.2 (medium)	100.0%
Codex CLI (default)	GPT-5.3-Codex (medium)	89.5%
Codex CLI (default)	GPT-5.4 (low)	84.2%
Codex CLI (default)	GPT-5.4 (medium)	89.5%
Codex CLI (default)	GPT-5.4 (high)	89.5%
Codex CLI (default)	GPT-5.4 (xhigh)	100.0%
Codex CLI ( $k$ 0)	GPT-5.4 (medium)	94.7%
Codex CLI ( $k$ 1)	GPT-5.4 (medium)	89.5%
Codex CLI ( $k$ 2)	GPT-5.4 (medium)	84.2%
Codex CLI ( $k$ 3)	GPT-5.4 (medium)	84.2%

Two settings reached 100.0%: GPT-5.2 (medium) and GPT-5.4 (xhigh). Several other settings clustered at 89.5% or 94.7%, and even the lowest reported accuracies remained at 84.2% (Nadgir et al., 23 Jun 2026).

In the paper’s interpretation, these results indicate that the benchmark’s saturation is not merely an artifact of the original field mix. More cautiously stated, the reported evidence suggests that a disciplinary composition shift from the original benchmark domains to physics, engineering, and economics is insufficient, by itself, to restore useful separation among top-performing agents on headline accuracy.

6. Interpretation, scope, and limitations

CORE-Bench OOD occupies a specific place in the evaluation ecology of agent benchmarks. It is not introduced as a generic OOD-detection benchmark and does not define a conventional train/test novelty protocol over samples. Instead, it is an out-of-distribution task suite for computational reproducibility agents, where “OOD” denotes a field distribution shift in the benchmark task population (Nadgir et al., 23 Jun 2026).

This framing matters for construct validity. CORE-Bench OOD is designed to test whether success on CORE-Bench v1.1 reflects capability that transfers across disciplines, rather than capability overfit to the benchmark’s original scientific mix. Within the same paper, however, OOD evaluation is only one part of a broader post-saturation program. Efficiency, reliability, scaffold effects, and human-agent collaboration are treated as additional axes of measurement rather than as consequences derivable from OOD accuracy alone (Nadgir et al., 23 Jun 2026).

The benchmark’s limitations are explicit. The authors state that log analysis is not exhaustive, that CORE-Bench OOD is an active and evolving benchmark, and that the suite is still small, containing only 19 tasks. They also emphasize that it tests primarily field shift, not every possible form of distribution shift, and that OOD results support conclusions about accuracy saturation rather than serving as a complete measure of real-world utility (Nadgir et al., 23 Jun 2026).

A further interpretive caution is nomenclatural. Because another 2026 benchmark also uses the name CORE-Bench for an agentic code-retrieval setting, references to “CORE-Bench OOD” require disambiguation by topic and citation context (Zhang et al., 10 Jun 2026). In the reproducibility-benchmark literature, CORE-Bench OOD specifically denotes the post-saturation field-shift suite attached to CORE-Bench v1.1.

Taken together, CORE-Bench OOD functions less as a harder successor benchmark than as an instrument for testing a particular failure mode of saturated evaluation: the possibility that ceiling-level performance is benchmark-specific rather than transferable. Its main reported result is that, under the paper’s statistical criterion, saturation persists even after the field distribution shifts (Nadgir et al., 23 Jun 2026).