CORE-Bench v1.1: Reproducibility Benchmark

• The paper demonstrates that CORE-Bench v1.1 refines its predecessor by correcting task-level errors and removing shortcut-based solutions to bolster computational reproducibility.
• CORE-Bench v1.1 is a reproducibility benchmark that measures an agent’s ability to run provided paper code using Code Ocean capsules across diverse disciplines.
• Empirical results reveal persistent accuracy saturation while emphasizing enhanced efficiency, reliability, and notable benefits from human–agent collaboration.

Searching arXiv for the relevant CORE-Bench papers and closely related versions. arXiv search query: all:"CORE-Bench" CORE-Bench v1.1 is a benchmark for evaluating AI agents on computational reproducibility: “the ability to reproduce the results of a scientific study using the data and code provided by its authors.” In the v1.1 line, the benchmark measures whether an agent can run a paper’s code to produce a target output and report a value accepted by the grading script, using Code Ocean capsules that are locally reproducible in principle. It is a corrected successor to CORE-Bench Hard from the earlier computational reproducibility benchmark, and it was introduced after the original Hard split became accuracy-saturated for strong agents (Nadgir et al., 23 Jun 2026). The name “CORE-Bench” is also used by an unrelated repository-state–aware code retrieval benchmark for agentic coding; that benchmark does not document a v1.1 release, so “CORE-Bench v1.1” refers specifically to the computational reproducibility benchmark lineage (Zhang et al., 10 Jun 2026).

1. Historical lineage and scope

CORE-Bench originated as a benchmark for scientific computational reproducibility, introduced with 270 tasks derived from 90 reproducible CodeOcean capsules across computer science, social science, and medicine (Siegel et al., 2024). That original release defined three difficulty tiers. CORE-Bench-Easy provides complete code output and asks the agent to extract information. CORE-Bench-Medium provides a Dockerfile and README and requires the agent to run the Docker command and then extract the required information. CORE-Bench-Hard provides only a README and requires the agent to determine and install dependencies, infer and run the correct commands, and extract the required information. The benchmark’s modeled workload includes reading repositories and READMEs, setting up environments, running code, extracting results from text and images, and producing a structured report.json.

CORE-Bench v1.1 was introduced only later, in response to benchmark saturation on CORE-Bench Hard. The follow-up work argues that a “retire-and-replace” response to saturation over-privileges accuracy and neglects six other dimensions of agent performance: construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of model versus scaffold, and uplift from human–agent collaboration (Nadgir et al., 23 Jun 2026). In that sense, v1.1 is not merely a harder split; it is a benchmark revision coupled to a broader evaluation program.

The benchmark’s task semantics remain those of computational reproducibility rather than claim verification. The measured capability is the ability to reproduce a study’s outputs from provided code and data, not to determine whether the paper’s claims are scientifically correct. This preserves continuity with the original benchmark while narrowing the interpretation of success.

2. Construction of v1.1 and the correction of validity threats

CORE-Bench v1.1 is a corrected successor to CORE-Bench Hard that preserves the original disciplines, languages, and construction pipeline based on Code Ocean capsules that are locally reproducible. The released v1.1 suite contains 39 tasks total: 13 computer science, 10 social science, and 16 medical science; the languages are 18 Python and 21 R (Nadgir et al., 23 Jun 2026).

The revision process began from the original 45 Hard tasks together with 27 new candidates. These were audited using automated and manual log analysis with Docent, after which tasks were removed, edited, or added. The reported audit surfaced 15 task-level errors and 20 tasks with exploitable shortcuts. The corrective actions were to remove 16 tasks and edit 15, with edits including question fixes, ground-truth fixes, grading fixes, and removal of shortcut-only questions. Ten new tasks were then added.

The documented threats to construct validity are concrete. Some tasks could be solved without genuine reproduction, for example by inferring a label from default matplotlib color order or by obtaining an answer from trivial CSV row counts. Other problems were grading-related or environment-related: non-deterministic results when only parts of a pipeline were run in isolation, and version drift such as Python 3.6 versus 3.12 changing answers. These examples are important because they identify failure modes that only became visible once agents reached deeper points in the benchmark pipeline.

Scoring and guardrails were also updated. The revised grading expands acceptance around 95% prediction intervals to include np.isclose tolerances at bounds and to accept unrounded versus rounded value discrepancies; it normalizes boolean-as-string mismatches such as "True" versus True; and it accepts multiple correct answers for specific tasks where figures and outputs differ, covering six questions across five capsules. A plausible implication is that v1.1 treats reproducibility benchmarking as an “active” measurement construct rather than a frozen artifact: the benchmark is explicitly maintained against newly discovered shortcuts and grading defects.

3. Execution environment, success criterion, and formal metrics

In v1.1, each task is successful if the agent runs the paper’s code to produce the target output and reports a value accepted by the grading script (Nadgir et al., 23 Jun 2026). The benchmark uses a common evaluation harness, HAL, with Weave logging, and agents have full filesystem and web access. Runs execute on Azure VMs, with GPU-required tasks on Standard_NC4as_T4_v3 and others on Standard_D4s_v3. For Codex CLI, Claude Code, and OpenCode, the per-task timeout is 45 minutes with a maximum of 3 retries. For CORE-Agent, the per-task timeout is 5 hours with a maximum of 200 steps and 1 retry.

The principal accuracy metric is task success rate,

$$A = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\text{task } i \text{ succeeded}],$$

reported with 95% Wilson confidence intervals. This is stricter than per-question scoring: the task is counted as solved only if the required reproduction succeeds and the reported answer satisfies the acceptance logic.

The saturation analysis follows the Akhtar et al. criterion. The standard error of the difference between the top and $k$-th agent is

$$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$

where $n_{\mathrm{eff}} = n^{\alpha}$, $\alpha \in [0,1]$, and the reported default is $\alpha = 0.5$. The top $k$ are considered statistically indistinguishable if

$s_1 - s_k \le z \cdot SE_{\Delta},$

with $z = 1.96$. For v1.1, the paper reports $s_1 = 1.0$, $k$0, and $k$1, establishing saturation under that criterion.

The benchmark also retains explicit treatment of non-determinism through a Student-$k$2 prediction interval. Using sample size $k$3, mean $k$4, and standard deviation $k$5, the margin is

$k$6

with interval

$k$7

This formula appears in the default prompt for non-determinism checks and connects the revised grading tolerances to the underlying statistical treatment of stochastic outputs.

Efficiency and reliability are elevated to first-class metrics. Efficiency is measured via total tokens and dollar cost. Reliability is decomposed into outcome consistency, resource consistency, calibration, and discrimination. The benchmark therefore evaluates not only whether an agent succeeds, but whether it does so repeatably, economically, and with confidence estimates that bear any relation to empirical success.

4. Empirical results on saturation, efficiency, reliability, and scaffold effects

The headline empirical finding is that accuracy saturation persists on v1.1 even after validity fixes (Nadgir et al., 23 Jun 2026). On the 39-task suite, the reported results include: CORE-Agent + Opus 4.6 (none), $k$8; Codex CLI + GPT-5.4 (high/xhigh), $k$9; Codex CLI + GPT-5.4 (medium), $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$0; Claude Code + Opus 4.6 (adaptive), $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$1; and OpenCode + GPT-5.4 (high), $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$2. Sensitivity to concurrent subagents is also reported for GPT-5.4 (medium) under Codex CLI: max_thr 1 gives 94.9%, 3 gives 97.4%, 6 gives 92.3%, and 9 gives 97.4%.

Because top-end accuracy is compressed, efficiency becomes discriminative. The paper reports that GPT-5.3-Codex (medium) achieved the same 97.4% accuracy as GPT-5.4 (high) at approximately 60% lower dollar cost, and that it was also the most efficient by tokens. Token usage and dollar cost do not align perfectly because pricing, caching, and scaffold design differ; CORE-Agent, for example, does not cache model outputs. This suggests that efficiency comparisons in saturated regimes can expose substantial differences even when headline success rates are nearly tied.

Reliability results are structurally different from accuracy results. In five-run samples per agent, more accurate agents are also more repeatable, but calibration is poor: the mean empirical pass rate is 93% whereas mean self-reported confidence is 32.1%. Confidence correlates with counts of failed bash tool calls, a metric reported as uncorrelated with success, and no agent consistently discriminates success from failure better than random on the basis of its own confidence. The benchmark therefore reveals a dissociation between execution competence and self-assessment.

The paper also reports large scaffold effects. With GPT-5.4 (medium), Codex CLI outperforms CORE-Agent by approximately 44 percentage points. Conversely, controlled model-versus-scaffold comparisons show complementarity: for both Opus 4.5 and GPT-5.4, an oracle that selects the best scaffold per task reaches 100%, implying that every task is solvable by at least one scaffold, but not by the same scaffold. Strategy analyses sharpen this observation. With Opus 4.6, Claude Code draws answers from code text output 41% versus 3% vision-read, whereas CORE-Agent is 21% code-output versus 31% vision-read. With GPT-5.4 the contrast is 1% vision-read for Codex CLI versus 56% for CORE-Agent. Direct localized fixes succeed 95.2% of the time compared with 67.8% for rewrites, and Codex CLI uses direct fixes 82% of the time versus 49% for CORE-Agent. A plausible implication is that scaffold design changes not only tool-use efficiency but also the agent’s epistemic route to the answer.

5. Out-of-distribution testing and human–agent collaboration

The follow-up study introduces CORE-Bench OOD as a separate out-of-distribution suite built with the same log-analysis pipeline (Nadgir et al., 23 Jun 2026). The OOD suite contains 19 tasks: 10 engineering, 5 physics, 2 economics, and 2 computer science, with one computer-science task having an approximately 50-minute runtime. It began from 30 tasks and was iteratively edited and reduced to 19 final tasks. The rationale is that disciplines differ in repository organization, ecosystems, and workflows, so OOD testing probes whether saturated in-distribution performance transfers under field shift.

The reported OOD results indicate that saturation largely transfers. Among 12 Codex CLI agents, the top five are statistically indistinguishable. The paper reports 100.0% for GPT-5.2 (medium) and 100.0% for GPT-5.4 (xhigh), with GPT-5.1 (medium) at 94.7% and many GPT-5.x configurations at 89.5%. Under the same Akhtar-style test, OOD also satisfies the indistinguishability condition, with $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$3, $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$4, and $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$5. This suggests that field shift alone does not restore strong ranking resolution once agent capability has crossed the relevant threshold.

The paper’s human–agent collaboration study adds a distinct measurement axis. It uses 20 papers, 5 evaluators, and 50 total replication attempts, with each evaluator performing 10 runs: 5 with an agent and 5 manually. The human–agent condition uses Codex CLI + GPT-5.4 with extra-high thinking and standardized Docker images; the manual condition uses traditional web search with no generative AI and AI overviews suppressed. The main analysis is a fixed-effects regression,

$$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$6

where $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$7 is 0 for human–agent collaboration and 1 for Manual, with paper and researcher fixed effects and CR2 standard errors clustered by researcher. The reported coefficient is $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$8 with CR2 SE $$SE_{\Delta} \approx \sqrt{\frac{s_1(1-s_1)}{n_{\mathrm{eff}}} + \frac{s_k(1-s_k)}{n_{\mathrm{eff}}}},$$9, so $n_{\mathrm{eff}} = n^{\alpha}$0, $n_{\mathrm{eff}} = n^{\alpha}$1: manual runs took approximately 2.11 times as long as collaborative runs on average.

The collaboration results are qualified by right-censoring. A 3-hour cap applied, and 5 of 25 manual runs hit the limit whereas 0 of 25 human–agent runs did, so the reported uplift is likely conservative. Process observations are also reported: 19 of 25 human–agent runs completed fully autonomously aside from launching infrastructure; across the 25 collaborative runs, agents added value in environment setup (25), running code (23), identifying main scripts (20), navigating README/files (19), and debugging runtime errors (14).

6. Limitations, interpretation, and benchmark identity

CORE-Bench v1.1 and CORE-Bench OOD are explicitly presented as improved but not perfect (Nadgir et al., 23 Jun 2026). Log analysis is target-dependent and not exhaustive, LLM-based auditing still requires manual validation, and the authors treat both suites as active benchmarks subject to future updates if new threats emerge. Coverage is limited to computer science, medical science, and social science in v1.1, and to physics, engineering, economics, and computer science in OOD; the programming languages are only Python and R; runtime bounds exclude very long pipelines.

The human–agent experiment also has constrained external validity. It uses a small sample of 20 papers and 5 evaluators; the evaluators are coauthors, introducing possible demand effects; there is no external ground truth beyond the paper values; and right-censoring is not modeled, which makes the estimated uplift conservative rather than definitive.

The benchmark’s broader significance lies in its redefinition of what a “saturated” benchmark can still measure. In the original 2024 release, CORE-Bench was already motivated by construct validity: tasks were intended to mirror what researchers actually do to reproduce papers rather than present toy problems (Siegel et al., 2024). The v1.1 program extends that logic. Once accuracy ceases to separate strong agents, the benchmark remains useful for measuring whether solutions are genuine, whether performance survives field shift, how much it costs to obtain the same accuracy, whether repeated runs agree, how much the scaffold matters independently of the model, and how much time a human saves when paired with an agent.

A final point of interpretation concerns nomenclature. The label “CORE-Bench” now spans multiple unrelated arXiv benchmarks, including the computational reproducibility benchmark discussed here and a repository-state–aware code retrieval benchmark for agentic coding (Zhang et al., 10 Jun 2026). In current usage, “CORE-Bench v1.1” is specific to the computational reproducibility benchmark introduced in the saturation study. That distinction is necessary for bibliographic precision, because the versioning, task semantics, and evaluation methodology differ fundamentally across these lines.