CORE-Bench Hard Benchmark Overview

• CORE-Bench Hard is a set of advanced evaluation tasks defined by README-only execution, dense code retrieval, and multi-hop static analysis challenges.
• It evaluates performance using metrics like pass@1, NDCG@10, and Recall@100 while emphasizing reliability, efficiency, and construct validity.
• The benchmark rigorously tests models under real-world conditions by requiring dependency resolution, inferred execution commands, and minimal scaffolding.

CORE-Bench Hard denotes the most demanding evaluation regimes associated with recent CORE-Bench benchmark lines, but the phrase is benchmark-specific rather than uniform. In the original computational reproducibility benchmark, it refers to level-3 tasks in which an agent is given only a repository README and must install dependencies, infer how to run the code, execute the workflow, and extract a paper-reported result (Siegel et al., 2024). In the later case study on benchmark saturation, the same label refers to a 45-task hardest split used to analyze dimensions beyond headline accuracy, including construct validity, out-of-distribution generalizability, efficiency, reliability, scaffold effects, and human–agent collaboration (Nadgir et al., 23 Jun 2026). In the repository-scale code-retrieval benchmark also named CORE-Bench, there is no explicit Easy/Medium/Hard partition, but the Dense-Large regime is treated as the de facto Hard slice because it combines very large corpora with high relevance density (Zhang et al., 10 Jun 2026).

1. Terminological scope

The recent literature uses closely related names for distinct hard-setting benchmarks. The main usages are summarized below (Siegel et al., 2024, Nadgir et al., 23 Jun 2026, Zhang et al., 10 Jun 2026).

Benchmark line	Hard definition	Primary evaluation
Computational reproducibility CORE-Bench	Level-3, README-only reproduction tasks	pass@1; later extended to additional dimensions
Agentic code-retrieval CORE-Bench	Dense-Large regime treated as de facto Hard	NDCG@10 and Recall@100
CoRe static-analysis benchmark	Hard subset requiring multi-hop semantic reasoning	F1, trace generation, and exact match

The third row is a similarly named but separate benchmark family. In CoRe, the benchmark focus is static analysis over code semantics rather than computational reproducibility or repository search (Xie et al., 3 Jul 2025). This naming overlap is consequential because empirical claims about “CORE-Bench Hard” are not interchangeable across these settings.

2. Level-3 computational reproducibility tasks

In the original computational reproducibility benchmark, CORE-Bench is organized into three levels. Easy provides full code output and asks the agent only to extract answers from those outputs. Medium provides a Dockerfile and README instructions, so the agent must run Docker and then extract answers. Hard provides only the README, and the agent must install dependencies, infer the run command, execute the code, and extract the requested answers (Siegel et al., 2024).

The construction pipeline begins from 5,090 public CodeOcean capsules and filters them by ten criteria, including requirements such as running in under 45 min on CodeOcean hardware, containing a README, having labeled results, and exhibiting low-variance outputs. This produced 90 capsules, split into 45 train and 45 test, across computer science, social science, and medicine, and written in Python or R. For each capsule, the authors manually verified full local reproducibility using CodeOcean’s environment, extracted 1–8 task questions from its outputs for a total of 181 questions, and created Easy, Medium, and Hard variants by removing progressively more information. In the Hard variant, the agent receives the code directory, the README.md, and a report.json template listing the question keys to fill (Siegel et al., 2024).

A later post-saturation study operationalizes CORE-Bench Hard as 45 tasks drawn from 45 capsules. These capsules span 18 computer-science, 14 social-science, and 13 medical-science tasks, and are equally split between Python and R. Each task asks for a paper-reported quantity such as an AUC, p-value, or group label, computed from the source code under minimal scaffolding (Nadgir et al., 23 Jun 2026).

Two representative level-3 tasks illustrate the workflow. In a social-science capsule, the agent must read a README specifying a command such as python run_experiments.py --config config.yaml, install dependencies such as pandas and matplotlib, execute the run, inspect generated PDF figures in ./results/, convert them to images, and answer questions such as which sample had the lower homicide rate per 100k in 2000 and what p-value appeared in a bootstrapped predictive distribution of endline trust in police (Siegel et al., 2024). In an epidemiological simulation capsule, the agent must infer and run python code/epidemic.py --config config.ini, resolve a version conflict in network-diffusion by installing version 0.6, and then extract the average infected fraction after 100 simulation steps from a CSV or console output (Siegel et al., 2024).

3. Evaluation protocol and baseline behavior

The primary metric in the original benchmark is task accuracy, or pass@1:

$$\mathrm{accuracy}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\text{agent correctly answered all questions of task } i\}.$$

A task is counted as correct only if every sub-question answer lies within a 95% predictive interval derived from three human runs of the capsule. If $r_{j,1}, r_{j,2}, r_{j,3}$ are the three manual results for question $j$, with sample mean $\bar r_j$ and sample standard deviation $s_j$, then the acceptance interval is

$$\bar r_j \pm t_{0.975,2}\cdot s_j \cdot \sqrt{1 + 1/3}.$$

The benchmark also defines

$\mathrm{pass@}k = 1-(1-\mathrm{accuracy})^k$

and

$\mathrm{pass}\wedge k = \mathrm{accuracy}^k.$

Evaluation is performed in isolated Azure VMs to prevent cross-task interference. CPU tasks run on Standard_E2as_v5, GPU tasks on Standard_NC4as_T4_v3, all under Ubuntu Linux with an 80 GB disk and a 2 h timeout. A manager machine launches roughly 270 VMs in parallel, copies in the agent code and capsule files, waits for a task_completed.log, downloads report.json, deletes the VM, and evaluates all results locally. The paper notes that 270 tasks times 2 h would require roughly 20 days without parallelization, whereas the harness reduces wall-clock time to roughly 2 h. Both AutoGPT and CORE-Agent are given access to a query_vision_language_model(image,query) tool for figure interpretation (Siegel et al., 2024).

Baseline results on the Hard test set show a large gap between the task-specific agent and the generalist baseline (Siegel et al., 2024).

Model	CORE-Bench-Hard accuracy
CORE-Agent + GPT-4o	21.48%
CORE-Agent + GPT-4o-mini	16.30%
AutoGPT + GPT-4o	6.67%
AutoGPT + GPT-4o-mini	2.22%

For CORE-Agent + GPT-4o, the average API cost per Hard task is \$2.96 with a reported variation of `±$0.09. Itspass@3is31.1 %, whilepass∧3is8.89 %. The dominant failure modes are dependency resolution, inferring the correct run command, retrieval from many output files even after successful execution, and timeouts duringpip install` or R package builds. The qualitative analysis emphasizes greedy retrying of the same installation command and mis-reading file trees as common root causes (Siegel et al., 2024).

4. Saturation, construct validity, and post-saturation evaluation

The later study “Life After Benchmark Saturation” treats CORE-Bench Hard as a case study in what remains measurable after headline accuracy approaches ceiling levels. Rather than retiring the benchmark once top agents cluster near 100%, the paper expands evaluation to six dimensions: accuracy, out-of-distribution generalizability, efficiency, reliability, model-versus-scaffold performance, and human–agent collaboration uplift (Nadgir et al., 23 Jun 2026).

Construct-validity analysis surfaces two classes of threats once agents become strong enough that hidden shortcuts and grading artifacts matter. The first class is task-level errors, found in 15 tasks, including incorrect ground truths, malformed prompts, grading-script bugs, and unsolvable data-availability failures. The second class is shortcut tasks, found in 20 tasks with 4 overlapping, where agents can extract answers from pre-existing artifacts or perform trivial table-row counting rather than reproduce the intended computation. The paper gives examples in which an agent counted occurrences in a shipped CSV instead of rerunning the study’s bootstrap analysis, and another in which a non-deterministic random-seed issue caused repeated grading inconsistencies (Nadgir et al., 23 Jun 2026).

These findings motivate two revised suites. CORE-Bench v1.1 removes or edits all 15 error-tasks, performs 16 removals and 15 edits, patches grading tolerances by extending NumPy’s isclose interval, accepting boolean or string synonyms, and allowing multiple legitimate answers, and adds 10 new tasks for a total of 39 validated tasks. CORE-Bench OOD is an out-of-distribution suite built from 30 initial candidates, with 17 removals, 1 regraded task, and 6 additions, yielding 19 validated tasks across economics, engineering, physics, and computer science. Both are maintained as active benchmarks that can be updated when new validity threats emerge (Nadgir et al., 23 Jun 2026).

The central empirical claim is that saturation does not eliminate benchmark utility. On CORE-Bench v1.1, the top five agents—Claude Code at 100 %, and GPT-5.4-Codex, GPT-5.3, GPT-5.4 high, and GPT-5.4-xhigh all at 97.4 %—are reported as statistically indistinguishable by the stated $z=1.96$ test. On CORE-Bench OOD, the top five agents cluster between 89.5 % and 100 %. Reliability measures still differentiate agents: outcome consistency and resource consistency both rise with mean accuracy, with Pearson correlations of r=+0.94 and r=+0.95, respectively. Confidence estimates, however, are poor: mean empirical pass rate is approximately 93 %, while mean self-reported confidence is approximately 32.1 %, and discrimination AUROC remains around 0.52–0.55, which the paper describes as essentially random (Nadgir et al., 23 Jun 2026).

Efficiency and scaffold choice also remain informative after saturation. The paper reports that GPT-5.3-Codex at medium reasoning matches GPT-5.4 high at 97.4 % accuracy for approximately 60 % lower cost, while token-accuracy correlation is strong at r=+0.94. Swapping scaffolds for the same model changes success on 31 % of tasks; for example, GPT-5.4 on CORE-Agent fails 19 capsules, but Codex CLI rescues all 19 without regressions. Direct fixes succeed 95.2 % of the time, compared with 67.8 % for rewrite-from-scratch, and CORE-Agent has the lowest direct-fix rate at 49 % versus Codex CLI’s 82 % (Nadgir et al., 23 Jun 2026).

The human–agent collaboration study extends the benchmark from agent-only evaluation to workflow impact. Five experienced reproducibility evaluators each attempted 20 distinct papers under a blocked randomized design, with a 3 h cap per run. Manual sessions lasted on average 2.11× as long as human–agent runs, with \hat{\beta}=0.7485 on \log(T), CR2 SE = 0.0919, and p=0.00176. Agent-only runs completed 19/25 runs fully autonomously; in the remaining six, human intervention was mostly limited to setup, dependency fixes, or discrepancy triage. Evaluators rated the agent most helpful for environment setup in 25/25 runs, running code in 23, identifying main scripts in 20, and navigating the README in 19 (Nadgir et al., 23 Jun 2026). A common misconception is therefore explicitly rejected by the paper: saturation of accuracy is not taken as evidence that the benchmark should be abandoned.

5. Dense-Large retrieval as the de facto hard slice

A different benchmark, “CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding,” studies code retrieval rather than reproducibility. It does not define an official Easy/Medium/Hard partition, but its most difficult queries are those in the Dense-Large regime, denoted D–L, which the supplied summary treats as the de facto Hard setting. These queries arise from repositories with large total chunk counts and high qrel density, placing them in the top-difficulty quadrant relative to Focused-Small, Focused-Large, and Dense-Small queries. In level 2, roughly 20–25 % of the 5 061 queries fall into D–L; the candidate pool can reach 1–2 M AST-chunked files per query, positives average at least about 20 edit-bearing chunks, and negatives are the rest of the repository, approximately 1 000 000–5 000 000 non-edit chunks (Zhang et al., 10 Jun 2026).

Formally, for a D–L query set $r_{j,1}, r_{j,2}, r_{j,3}$0, each query $r_{j,1}, r_{j,2}, r_{j,3}$1 has a chunked corpus $r_{j,1}, r_{j,2}, r_{j,3}$2 visible at the PR commit, a positive set $r_{j,1}, r_{j,2}, r_{j,3}$3, and a negative set $r_{j,1}, r_{j,2}, r_{j,3}$4. A dual-encoder retriever scores documents by

$r_{j,1}, r_{j,2}, r_{j,3}$5

Level 2 is issue-to-edit localization, where positives are patch-aligned edits and evaluation is by NDCG@10. Level 3 is broader-context retrieval, where positives are LLM-validated context such as tests and helpers, and evaluation is by Recall@100. Supervised fine-tuning uses a multi-positive InfoNCE objective:

$r_{j,1}, r_{j,2}, r_{j,3}$6

The reported D–L results show a sharp degradation relative to traditional code search (Zhang et al., 10 Jun 2026).

Model	Level-2 D–L	Level-3 D–L
Qwen3-8B	20.6% NDCG@10; 41.7% Recall@100	30.1% Recall@100
Qwen3-8B-SFT (3 epochs)	33.7% NDCG@10; 60.4% Recall@100	42.5% Recall@100

The paper states that this is a \>60 % relative drop from level-1 performance, exemplified by 96.9 % NDCG@10, and that simple supervised fine-tuning recovers much of the loss without reaching level-1 behavior. Representative D–L queries illustrate why. In grpc-go-3201, the query is “Do not call NewServiceConfig when DNS lookups are disabled…,” with level-2 positives at resolver.NewServiceConfig call sites in resolver.go and wrapper.go, and level-3 positives spread across the resolver stack, tests of ClientConn, and default service-config logic, amounting to about 121 chunks. In django-15499, the query asks to optimize CreateModel + AlterModelManagers into CreateModel; level-2 has one positive chunk, while level-3 has 58 relevant chunks including migration graph helpers and state objects. In zstd-983, the query concerns increasing windowLog from a CDict when source size is known; level-2 again has one edit target, but level-3 expands to 79 relevant chunks across parameter conversion, pledged-source handling, and CDict internals. These examples show why naive retrievers collapse when relevant evidence is scattered across dozens or hundreds of locations in a multi-million-chunk corpus.

6. Relation to the CoRe hard subset

A separate benchmark, “CoRe,” evaluates LLM code reasoning through static analysis tasks rather than reproducibility or repository retrieval. Its supplied hard-subset definition is based on three semantic axes: data-dependency depth, control-dependency depth, and information-flow complexity. A task is labeled Hard when it requires at least two hops of transitive data dependency, at least two levels of transitive control dependency, and mixed information flow with at least one implicit edge (Xie et al., 3 Jul 2025).

The formal indicator is

$r_{j,1}, r_{j,2}, r_{j,3}$7

and $r_{j,1}, r_{j,2}, r_{j,3}$8 otherwise. Representative examples include a C data-dependency trace of length 4, a Java control-dependency trace [1,2,3,4], and a Python information-flow chain with 2 implicit and 2 explicit edges. The reported hard-task error modes include missing intermediate nodes, false positives on structurally plausible negatives, under-enumeration when implicit flows introduce control-dependency sources, and a pronounced left-to-right bias in which reverse-order tasks suffer about a 40 % drop in F1 (Xie et al., 3 Jul 2025).

This related usage matters mainly as a terminological boundary. The CoRe hard subset is a stress test for multi-step static reasoning; the computational reproducibility version of CORE-Bench Hard is a README-only execution-and-extraction task; and the retrieval version’s de facto Hard slice is a repository-search regime with millions of distractor chunks. This suggests that “CORE-Bench Hard” functions less as a single benchmark artifact than as a family resemblance across hardest-case evaluations in modern agentic coding and reproducibility research.