DexBench: Dual-Path Code Reasoning Benchmark

• DexBench is a benchmark for dynamic code reasoning that pairs forward execution analysis with backward counterfactual input mutation.
• It uses Python programs from multiple datasets to test causal reasoning by simulating both observed execution and alternative branch exploration.
• Empirical findings show dual-path tasks are significantly harder than individual tasks, emphasizing the need for robust execution-grounded reasoning.

Searching arXiv for the specified DexBench paper and closely related benchmark context. DexBench is a benchmark for evaluating dynamic code reasoning in LLMs through a dual-path formulation of program execution. Introduced in “The Path Not Taken: Duality in Reasoning about Program Execution” (Hasanov et al., 22 Apr 2026), it is designed to test whether a model can reason not only about the execution path that a program actually takes under a given input, but also about the path not taken—that is, how the input must be altered to realize a specified counterfactual behavior. In contrast to prior execution benchmarks that concentrate on a single observed path, DexBench pairs forward / execution reasoning with backward / counterfactual reasoning over the same program and seed input. The benchmark contains 445 paired instances and was evaluated on 13 LLMs (Hasanov et al., 22 Apr 2026).

1. Conceptual basis and motivation

DexBench was proposed in response to a limitation in existing dynamic code reasoning benchmarks: many focus on only one direction of reasoning on a single observed execution path, such as predicting outputs, coverage, intermediate states, or path identifiers for a fixed input (Hasanov et al., 22 Apr 2026). This probes what happened during execution, but not why that path occurred or what would need to change for another path to be taken.

The benchmark is built on the claim that program-execution understanding is inherently dual. One component is forward / execution reasoning, in which a model predicts the observed behavior of a program for a given input. The other is backward / counterfactual reasoning, in which the model infers how the input should be mutated so that execution reaches a different path and satisfies a target behavioral objective. DexBench operationalizes these as paired tasks over the same causal context, rather than treating them as unrelated subtasks (Hasanov et al., 22 Apr 2026).

The motivating concern is also methodological. Benchmarks based only on fixed input-output or input-coverage mappings are more vulnerable to contamination and shortcutting. DexBench attempts to reduce this by requiring causal reasoning over execution flow, especially in the backward task, where the model must produce an input mutation that causes an uncovered branch to execute (Hasanov et al., 22 Apr 2026). This suggests that the benchmark is intended less as a test of memorized coding patterns than as a probe of state-aware reasoning about branching behavior.

2. Dual-path task formulation

The paper defines an execution trace induced by a program $P$ and input $I_{exec}$ as

$$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$

where $s^{(t)}$ is the statement executed at step $t$, $\sigma^{(t)}$ is the program state at step $t$, and $\mathcal{T}$ is the space of execution traces (Hasanov et al., 22 Apr 2026). Observable runtime behavior is represented by a mapping

$\phi:\!\mathcal{T}\!\to\!\mathcal{O}.$

The forward / execution reasoning task is then defined as

$$\mathcal{R}_{exec}: (P, I_{exec}) \mapsto \phi(\tau_{exec}).$$

In DexBench’s concrete instantiation, $I_{exec}$0 is statement coverage, represented as the set of executed line numbers (Hasanov et al., 22 Apr 2026).

For the backward / counterfactual reasoning task, the target behavior is defined as

$I_{exec}$1

where $I_{exec}$2 transforms the observed behavior into a desired counterfactual objective (Hasanov et al., 22 Apr 2026). The task is

$I_{exec}$3

with the requirement that executing $I_{exec}$4 on the mutated input $I_{exec}$5 produces a trace $I_{exec}$6 satisfying

$I_{exec}$7

DexBench instantiates this objective as branch-targeted input mutation: the model must generate a mutation of the seed input that causes a specific uncovered branch $I_{exec}$8 to be reached, i.e.

$I_{exec}$9

The joint benchmark task is expressed as

$$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$0

This paired construction is central to the benchmark’s purpose. The forward task tests whether a model can simulate the observed execution, while the backward task tests whether it can reason from a desired branch outcome back to a causally effective input change (Hasanov et al., 22 Apr 2026).

3. Concrete benchmark instantiation

DexBench uses Python programs drawn from three existing datasets: CruxEval, HumanEval, and PythonSaga (Hasanov et al., 22 Apr 2026). These sources were selected because they support input-dependent reasoning about executions while differing in complexity: CruxEval contains shorter programs with simpler control flow, HumanEval has moderate control-flow complexity, and PythonSaga includes deeper nesting, recursion, and more advanced constructs.

Programs were retained only if they satisfied two constraints: they had at least one conditional or loop, and they had a test case with partial code coverage $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$1 (Hasanov et al., 22 Apr 2026). These constraints ensure the existence of nontrivial control flow and at least one uncovered branch suitable for the counterfactual task.

Each benchmark instance is a pair built from the same program and seed input. The construction pipeline is explicit. First, a seed test case satisfying the filtering criteria is randomly selected. Second, the program is executed on that seed input and statement coverage is collected using Slipcover, which provides the ground truth for the forward task. Third, among uncovered branches, the benchmark selects a counterfactual target intended to increase execution diversity and increase code coverage (Hasanov et al., 22 Apr 2026).

The appendix specifies the target-selection heuristic more concretely. Let $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$2 denote the coverage map, $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$3 the AST, $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$4 the uncovered lines, $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$5 the candidate branches, $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$6 the selected target branch, and $$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$7 the target line number. The procedure is:

1. collect uncovered lines:

$$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$8

1. gather branches whose bodies are entirely uncovered
2. score each candidate by body size:

$$\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},$$9

1. choose the branch with maximal score
2. if no such branch exists, choose an uncovered standalone line not in any branch body, preferring the smallest line number; otherwise discard the program (Hasanov et al., 22 Apr 2026)

This means DexBench defaults to targeting counterfactual paths with maximal coverage gain.

The resulting benchmark contains 445 unique paired instances total: 298 from CruxEval, 100 from HumanEval, and 47 from PythonSaga. Programs range from 10 to 78 lines of code, with mean cyclomatic complexity

$s^{(t)}$0

and complexity range 2 to 19 (Hasanov et al., 22 Apr 2026).

4. Prompting interface, evaluation, and scoring

DexBench uses two dedicated prompt templates. In the forward prompt, the model sees a line-numbered Python program and is asked to determine all executable line numbers run at least once. It must return JSON containing executed_lines as a sorted list of integers (Hasanov et al., 22 Apr 2026). In the backward prompt, the model sees the same line-numbered program, the original seed input, and a target line, and it must return only the comma-separated argument values for a mutated input that causes the target line or body to execute (Hasanov et al., 22 Apr 2026). All experiments use one-shot prompting with one illustrative example.

For both tasks, the evaluation metric is $s^{(t)}$1. The forward success indicator is

$s^{(t)}$2

meaning that at least one of the $s^{(t)}$3 candidate predictions must exactly match the true coverage set (Hasanov et al., 22 Apr 2026). The backward success indicator is

$s^{(t)}$4

meaning that at least one generated input must actually reach the target branch (Hasanov et al., 22 Apr 2026).

The key aggregate metric is the joint dual success

$s^{(t)}$5

An instance is counted as solved only if the model succeeds on both the execution and counterfactual tasks (Hasanov et al., 22 Apr 2026). This is the benchmark’s main scoring idea and the basis for its claim to be more discriminative than isolated evaluation.

For analysis, the paper also introduces a relaxed execution score based on Jaccard similarity:

$s^{(t)}$6

This is not the main score, but it captures partial overlap between predicted and true coverage sets and is useful for diagnosing near misses (Hasanov et al., 22 Apr 2026).

A notable feature of DexBench is that verification is execution-grounded. Forward answers are checked by exact match to observed coverage, while backward answers are checked by actual execution of the generated input (Hasanov et al., 22 Apr 2026). This makes the benchmark stricter than tasks that score only plausibility or heuristic similarity.

5. Empirical findings and model behavior

The paper evaluates 13 LLMs: nine open-source models—Jamba Reasoning 3B, Llama-3.1 Nemotron Nano, Llama-3.2-3B-Instruct, Mistral Small 24B, Magistral Small, QwQ-32B, Qwen2.5-32B, Llama-3.3-70B-Instruct, and Qwen2.5-72B—and four closed-source models—Gemini 2.5 Flash, GPT-5 Mini, Grok-4 Reasoning, and Claude Sonnet 4 (Hasanov et al., 22 Apr 2026).

A central result is that dual-path reasoning is substantially harder than either component alone. The paper explicitly reports large populations of execution-only successes and counterfactual-only successes, indicating that strong performance on a single task does not imply robust causal understanding of execution flow (Hasanov et al., 22 Apr 2026). This is one of the benchmark’s main empirical arguments: isolated task performance can overestimate true program-execution understanding.

Performance declines with program complexity. Across the three source datasets, dual performance generally worsens from

$s^{(t)}$7

reflecting the increasing difficulty of maintaining a consistent causal model over programs with deeper control flow, more nesting, recursion, and more advanced constructs (Hasanov et al., 22 Apr 2026).

Small models perform poorly on the joint task. All sub-10B models show near-zero dual performance, largely because they fail on exact code coverage prediction even when they occasionally succeed at branch-reaching mutation (Hasanov et al., 22 Apr 2026). This suggests that accurate execution simulation remains a major bottleneck.

Scaling is not monotonic. A prominent result is that Qwen2.5-32B outperforms Qwen2.5-72B on dual-path reasoning across all datasets (Hasanov et al., 22 Apr 2026). Likewise, reasoning-focused post-training does not reliably help: Mistral Small 24B outperforms Magistral Small, and Qwen2.5-32B outperforms QwQ-32B (Hasanov et al., 22 Apr 2026). A plausible implication is that current reasoning-oriented post-training procedures do not necessarily transfer to causal reasoning about program execution.

Among frontier systems, the best overall dual-path performance came from closed-source models. The reported ranking is

$s^{(t)}$8

(Hasanov et al., 22 Apr 2026).

Representative dual reasoning, $s^{(t)}$9 results illustrate both absolute capability and residual difficulty:

Dataset	Claude Sonnet 4	Grok-4 Reasoning	GPT-5 Mini	Gemini 2.5 Flash
CruxEval	87.6	79.9	73.8	51.0
HumanEval	69.0	65.0	59.0	41.0
PythonSaga	55.3	51.1	29.8	4.3

Even the strongest models degrade substantially on PythonSaga (Hasanov et al., 22 Apr 2026). This indicates that DexBench remains challenging even for high-performing systems.

6. Failure modes, ablations, and relation to prior benchmarks

DexBench’s qualitative analyses identify multiple failure types: computation errors, control flow errors, lack of fact verification, native API misevaluation, prompt misread, null response, output format errors, misreporting final output, skipping statements, and hallucinations (Hasanov et al., 22 Apr 2026). The distribution of failures differs by task. Execution failures often involve prompt misreading, skipping statements, hallucinated lines, or null responses, whereas counterfactual failures often involve null responses, formatting mistakes, and semantic misunderstandings of APIs or control flow.

The paper’s examples illustrate what the benchmark is intended to measure. In one case, a model correctly identifies the relevant branch structure in a character-counting program but miscounts occurrences of a character, leading to an invalid mutation; this is categorized as a computation error (Hasanov et al., 22 Apr 2026). In another, a model incorrectly reasons about Python’s splitlines() behavior and proposes a mutation that would not trigger the target break; this is classified as native API misevaluation (Hasanov et al., 22 Apr 2026). These examples show that DexBench is not merely about branch guessing, but about whether the model causally understands how an input mutation changes actual runtime behavior.

An ablation comparing input mutation with unconstrained input generation found that direct generation can perform better than mutation for GPT-5 Mini and Grok-4 Reasoning (Hasanov et al., 22 Apr 2026). The authors interpret this as support for DexBench’s harder design: mutation is more causally demanding because it requires the model to reason about how specific changes to an existing input propagate through the execution path. This suggests that DexBench intentionally prioritizes counterfactual faithfulness over raw branch-reaching convenience.

Prompt sensitivity is also evident. A stronger execution prompt from CRISPE improves execution-only performance, especially when finite-option coverage choices are available (Hasanov et al., 22 Apr 2026). However, the paper argues that the dual structure remains informative because it continues to test joint reasoning ability even when prompt engineering shifts absolute scores.

The paper situates DexBench relative to prior execution benchmarks. It explicitly contrasts the benchmark with CruxEval, which pairs output prediction and input prediction but not around shared execution-path causality in the same way; R-Eval / IC-Score, which focus on execution-related tasks but still on single-path behaviors; and CES, which reasons over multiple inputs but treats execution paths independently rather than as divergences in a shared execution space (Hasanov et al., 22 Apr 2026). DexBench’s contribution, in this framing, is not merely a new dataset, but a different causal organization of evaluation.

7. Limitations, scope, and significance

The paper notes several limitations. First, programs are drawn from public datasets, so contamination cannot be eliminated entirely, even if the dual-path design mitigates simple memorization (Hasanov et al., 22 Apr 2026). Second, DexBench is currently Python-only, largely because its instrumentation pipeline relies on tools such as Slipcover (Hasanov et al., 22 Apr 2026). Third, the benchmark uses one specific pair of behaviors—statement coverage for forward reasoning and branch-targeted mutation for backward reasoning—although the broader dual-path framework could in principle be instantiated with other observables, such as outputs, intermediate states, or post-conditions (Hasanov et al., 22 Apr 2026). Fourth, the default counterfactual target-selection heuristic favors paths with maximal coverage gain, which may bias the benchmark toward certain kinds of divergence (Hasanov et al., 22 Apr 2026).

These limits are substantive but do not negate the benchmark’s conceptual contribution. DexBench advances the evaluation of code-capable LLMs by formalizing the idea that execution understanding involves both the realized path and the unrealized alternative. Its emphasis on paired instances, execution-grounded verification, and the joint metric $t$0 makes it more stringent than benchmarks that test only one side of execution reasoning (Hasanov et al., 22 Apr 2026).

DexBench is therefore best understood as a benchmark for causal, state-aware reasoning about execution flow rather than a generic code-generation or code-completion benchmark. It is relevant to research on debugging, testing, coverage-guided exploration, fuzzing-like reasoning, and execution-aware code agents, because these settings require more than predicting outputs: they require understanding why a branch was or was not taken, and how to manipulate inputs to induce desired execution outcomes (Hasanov et al., 22 Apr 2026). A plausible implication is that future benchmark design in code intelligence may increasingly center such bidirectional formulations, especially where causal understanding rather than surface regularity is the target capability.