HumanExec: Executable Specification Benchmark

• HumanExec is a benchmark that evaluates executable intermediate specifications by inserting Boolean assertions at key program checkpoints.
• It supports three tasks—specification generation, program correctness checking, and program repair—using competitive programming problems enhanced with human buggy submissions and comprehensive test suites.
• The benchmark employs formal metrics like validity, correctness, and completeness, with notable improvements observed when using the SpecCoder training framework.

Searching arXiv for the cited HumanExec paper and closely related work to ground the article. HumanExec is a benchmark for evaluating executable intermediate specifications in code reasoning, introduced alongside SpecCoder in "Teaching Code LLMs to Reason with Intermediate Formal Specifications" (Le-Anh et al., 5 Jul 2026). It is built from recent Codeforces competitive-programming problems with test suites, reference solutions, and human buggy submissions, and it supports three tasks: specification generation, program correctness checking, and program repair. Its central object is the executable checkpoint specification: a Boolean assertion inserted at a semantically meaningful internal program point to describe an expected intermediate state rather than only a whole-program precondition or postcondition.

1. Motivation and conceptual basis

Recent work on CodeLLMs has shifted attention from code generation toward verification, debugging, and repair, but executable formal specifications remain labor-intensive to write. HumanExec is motivated by two limitations identified in prior practice: existing LLM-based methods mainly infer whole-program pre/postconditions, thereby missing the intermediate semantic commitments used in algorithmic reasoning, and prompting current CodeLLMs often produces executable assertions that are syntactically invalid, trivial, or too weak to reject behavior-changing faults (Le-Anh et al., 5 Jul 2026).

HumanExec therefore evaluates a stricter capability than end-to-end code production. The benchmark asks whether a model can describe expected internal states at meaningful checkpoints and whether those descriptions survive instrumentation and execution. A common misconception is that HumanExec is merely another competitive-programming benchmark. It is instead an execution-centric benchmark in which the primary evidence is operational: assertions are run, reached, satisfied, or violated during concrete executions.

2. Construction and corpus

HumanExec is constructed primarily from recent Codeforces competitive-programming problems. Problem selection keeps only tasks with more than 30 test cases. Human-written submissions that pass all available tests are treated as reference solutions. Buggy solutions are human-submitted buggy solutions rather than synthetic mutants, and they are retained only if they pass at least one test case and fail at least one test case. Submissions that do not compile, have malformed I/O, or fail due to shallow syntactic errors are discarded so that the benchmark concentrates on semantic bugs (Le-Anh et al., 5 Jul 2026).

Item	Value	Notes
Programming tasks	150	Recent Codeforces tasks
Test cases per problem	48–279	Average 79.53; median 70
Buggy solutions per task	Avg. 53.19	Min 3; max 188; median 44
Difficulty level	800–2500	Mean 1348; median 1300

Each task includes at least one reference solution, numerous buggy human submissions, and a comprehensive executable test suite. The use of real human mistakes is a defining design choice. HumanExec is intended to reflect genuine programming failure modes rather than shallow perturbations, and incomplete correctness is established by execution against the available test suite.

3. Checkpoint specifications and supported tasks

The primary annotation type in HumanExec is the executable checkpoint specification. These are Boolean assertions inserted at semantically meaningful internal program points and are meant to describe expected intermediate states. HumanExec also supports final-state, or postcondition, specifications that check values or state at the end of execution (Le-Anh et al., 5 Jul 2026).

The benchmark supports three tasks. In specification generation, a model receives a problem statement and a reference solution or program and must insert executable checkpoint specifications at internal program points. In program correctness checking, a model receives a problem statement and a candidate solution, either reference or buggy, and must determine whether the solution is correct; checkpoint specifications are used for localizing and justifying errors. In program repair, a model receives a problem and a buggy submission and must generate a repaired program that passes the test suite, potentially using violated executable checkpoint specifications as clues.

Because assertions are instrumented and executed on both correct and buggy runs, HumanExec supports dynamic, step-by-step verification rather than only surface comparison. This suggests a benchmark philosophy in which internal semantic commitments are treated as first-class artifacts for reasoning.

4. Evaluation protocol and formal metrics

HumanExec evaluates each generated assertion with three machine-checkable metrics: validity, correctness, and completeness (Le-Anh et al., 5 Jul 2026). Validity is the fraction of generated assertions that can be safely instrumented and executed; valid assertions must be syntactically valid, refer to in-scope variables, and avoid unsafe operations.

For an assertion $s_{ij}$ in task $i$, correctness is defined as

$$\mathrm{Corr}_{ij} = \frac{ \sum_{t\in T_i} \mathbf{1}[r_{ijt}>0 \wedge p_{ijt}=r_{ijt}] }{ \sum_{t\in T_i} \mathbf{1}[r_{ijt}>0] }.$$

Here, $r_{ijt}$ is the number of times the assertion is reached in test $t$, and $p_{ijt}$ is the number of times it holds in those evaluations. Correctness therefore asks whether the assertion always holds on correct executions of the reference program.

Completeness measures whether a correct assertion rejects any faulty, behavior-changing mutant at the same checkpoint:

$$\mathrm{Comp}_{ij} = \frac{ \#\{m\in E_{ij}: s_{ij}\text{ rejects }m\} }{ \#E_{ij} }.$$

Here, $E_{ij}$ denotes the eligible mutants that reach the assertion point. Completeness is thus a discriminative property: a specification is valuable insofar as it rules out faulty behavior, not merely insofar as it is true on correct runs.

Downstream tasks introduce further metrics. Program repair uses Pass@k and APR, the average pass rate across generated repairs. Program correctness checking reports MCC, Balanced Accuracy, FNR, FPR, and related classification measures. The benchmark also tracks whether assertions are actually exercised during execution, so evaluation depends on reachability and behavior rather than textual resemblance alone.

5. Empirical results and the role of SpecCoder

HumanExec was introduced together with SpecCoder, a verification-guided CodeLLM training framework that learns from validated reference programs, behavior-changing mutants, and multi-turn specification-refinement traces (Le-Anh et al., 5 Jul 2026). The reported results indicate that executable checkpoints improve both the quality of generated specifications and downstream reasoning over buggy code.

For specification generation, SpecCoder improves inline-specification correctness, completeness, and executable assertion validity across Qwen2.5-Coder model sizes. Correctness rises from 0.279 to 0.435 for Qwen2.5-Coder-7B, from 0.523 to 0.647 for Qwen2.5-Coder-14B, and from 0.631 to 0.810 for Qwen2.5-Coder-32B. Completeness rises from 0.171 to 0.782 for 7B, from 0.199 to 0.771 for 14B, and from 0.264 to 0.684 for 32B. Validity rises from 0.790 to 1.000 for 14B and from 0.899 to 0.997 for 32B. The same study reports checkpoint quality improving from 68.6% to 87.1% and the non-trivial assertion rate improving from 39.3% to 83.4%.

For program correctness checking, MCC improves from 0.236 to 0.307 for Qwen2.5-Coder-7B, from 0.283 to 0.384 for Qwen2.5-Coder-14B, and from 0.394 to 0.493 for Qwen2.5-Coder-32B. Balanced Accuracy reaches 0.733 for Qwen2.5-Coder-32B with SpecCoder. For program repair, Pass@1 improves from 0.089 to 0.140 for 7B, from 0.237 to 0.241 for 14B, and from 0.261 to 0.283 for 32B; APR improves from 0.578 to 0.598 for 14B and from 0.615 to 0.622 for 32B.

The HumanExec study also compares against vanilla instruction-tuned code models, Chain-of-Thought prompting, HoarePrompt, and natural-language specification-based baselines such as SpecRover and HoarePrompt. Chain-of-Thought prompting did not reliably improve performance, while HoarePrompt helped but still trailed executable specifications in most settings. The benchmark’s main empirical claim is therefore not just that more assertions are generated, but that the assertions become more executable, more correct on reference runs, and more effective at rejecting faulty behavior.

6. Relation to broader execution-centric research

HumanExec belongs to a broader research pattern in which intermediate representations are made executable, inspectable, and operationally consequential. In "Understanding Human Actions through the Lens of Executable Models," ExAct represents human motions as underspecified motion programs, interprets them as reward-generating functions for zero-shot policy inference using forward-backwards representations, and uses these executable action models for action segmentation and anomaly detection (Rubavicius et al., 20 Apr 2026). In "KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents," execution is decoupled from reasoning through an executive kernel and Intent-Gated Execution, with scheduling, tool dispatch, dependency resolution, failures, and security enforced outside the LLM reasoning layer (Guerin et al., 31 Mar 2026). In a workflow setting, the Collaborative Innovation Framework introduces asynchronous checkpoints for human input in HPC environments, allowing workflows to pause for review without halting underlying compute jobs (Mendoza et al., 5 May 2026). Earlier work on interactive execution monitoring for agent teams likewise emphasized plan-based monitoring, alert categorization, and the value of an alert as mechanisms for supporting human responsibility in dynamic, data-rich domains (Berry et al., 2011).

Against that background, HumanExec is distinctive in its executable object. It is neither a motion program, nor a workflow checkpoint, nor a tool-authorization policy, but an inline program assertion over intermediate state. The common thread is that execution itself becomes the medium of validation and control. A plausible implication is that HumanExec marks a shift in code-LLM evaluation from output plausibility toward operational semantics: models are assessed by whether they can state, check, and exploit internal invariants during execution.