---
title: 'DexBench: Dual-Path Code Reasoning Benchmark'
url: https://www.emergentmind.com/topics/dexbench
type: topic
---

# DexBench: Dual-Path Code Reasoning Benchmark

Searching arXiv for the specified DexBench paper and closely related benchmark context.
DexBench is a benchmark for evaluating **dynamic code reasoning** in large language models through a **dual-path** formulation of program execution. Introduced in “The Path Not Taken: Duality in Reasoning about Program Execution” [2604.20917], it is designed to test whether a model can reason not only about the execution path that a program actually takes under a given input, but also about the **path not taken**—that is, how the input must be altered to realize a specified counterfactual behavior. In contrast to prior execution benchmarks that concentrate on a single observed path, DexBench pairs **forward / execution reasoning** with **backward / counterfactual reasoning** over the same program and seed input. The benchmark contains **445 paired instances** and was evaluated on **13 LLMs** [2604.20917].

## 1. Conceptual basis and motivation

DexBench was proposed in response to a limitation in existing dynamic code reasoning benchmarks: many focus on only **one direction** of reasoning on a **single observed execution path**, such as predicting outputs, coverage, intermediate states, or path identifiers for a fixed input [2604.20917]. This probes what happened during execution, but not why that path occurred or what would need to change for another path to be taken.

The benchmark is built on the claim that program-execution understanding is inherently **dual**. One component is **forward / execution reasoning**, in which a model predicts the observed behavior of a program for a given input. The other is **backward / counterfactual reasoning**, in which the model infers how the input should be mutated so that execution reaches a different path and satisfies a target behavioral objective. DexBench operationalizes these as paired tasks over the same causal context, rather than treating them as unrelated subtasks [2604.20917].

The motivating concern is also methodological. Benchmarks based only on fixed input-output or input-coverage mappings are more vulnerable to contamination and shortcutting. DexBench attempts to reduce this by requiring **causal reasoning over execution flow**, especially in the backward task, where the model must produce an input mutation that causes an uncovered branch to execute [2604.20917]. This suggests that the benchmark is intended less as a test of memorized coding patterns than as a probe of state-aware reasoning about branching behavior.

## 2. Dual-path task formulation

The paper defines an execution trace induced by a program \(P\) and input \(I_{exec}\) as

\[
\tau_{exec}\!=\!\big(\langle s^{(1)}, \sigma^{(1)} \rangle ...\langle s^{(|\tau_{exec}|)}, \sigma^{(|\tau_{exec}|)} \rangle \big) \in \mathcal{T},
\]

where \(s^{(t)}\) is the statement executed at step \(t\), \(\sigma^{(t)}\) is the program state at step \(t\), and \(\mathcal{T}\) is the space of execution traces [2604.20917]. Observable runtime behavior is represented by a mapping

\[
\phi:\!\mathcal{T}\!\to\!\mathcal{O}.
\]

The **forward / execution reasoning** task is then defined as

\[
\mathcal{R}_{exec}: (P, I_{exec}) \mapsto \phi(\tau_{exec}).
\]

In DexBench’s concrete instantiation, \(\phi(\tau_{exec})\) is **statement coverage**, represented as the set of executed line numbers [2604.20917].

For the **backward / counterfactual reasoning** task, the target behavior is defined as

\[
\phi^\star = f\!\left(\phi(\tau_{exec})\right),
\]

where \(f:\mathcal{O}\to\mathcal{O}\) transforms the observed behavior into a desired counterfactual objective [2604.20917]. The task is

\[
\mathcal{R}_{cf} : (P, I_{exec}, \phi^\star) \mapsto I_{cf},
\]

with the requirement that executing \(P\) on the mutated input \(I_{cf}\) produces a trace \(\tau_{cf}\) satisfying

\[
\phi(\tau_{cf}) = \phi^\star.
\]

DexBench instantiates this objective as **branch-targeted input mutation**: the model must generate a mutation of the seed input that causes a specific uncovered branch \(b\) to be reached, i.e.

\[
b \in \phi(\tau_{cf}).
\]

The joint benchmark task is expressed as

\[
\mathcal{R}_{dual}: \mathcal{R}_{exec} \oplus \mathcal{R}_{cf}.
\]

This paired construction is central to the benchmark’s purpose. The forward task tests whether a model can simulate the observed execution, while the backward task tests whether it can reason from a desired branch outcome back to a causally effective input change [2604.20917].

## 3. Concrete benchmark instantiation

DexBench uses **Python** programs drawn from three existing datasets: **CruxEval**, **HumanEval**, and **PythonSaga** [2604.20917]. These sources were selected because they support input-dependent reasoning about executions while differing in complexity: CruxEval contains shorter programs with simpler control flow, HumanEval has moderate control-flow complexity, and PythonSaga includes deeper nesting, recursion, and more advanced constructs.

Programs were retained only if they satisfied two constraints: they had **at least one conditional or loop**, and they had **a test case with partial code coverage** \((<100\%)\) [2604.20917]. These constraints ensure the existence of nontrivial control flow and at least one uncovered branch suitable for the counterfactual task.

Each benchmark instance is a pair built from the **same program and seed input**. The construction pipeline is explicit. First, a seed test case satisfying the filtering criteria is randomly selected. Second, the program is executed on that seed input and statement coverage is collected using **Slipcover**, which provides the ground truth for the forward task. Third, among uncovered branches, the benchmark selects a counterfactual target intended to increase execution diversity and increase code coverage [2604.20917].

The appendix specifies the target-selection heuristic more concretely. Let \(\mathcal{C}\) denote the coverage map, \(\mathcal{A}\) the AST, \(\mathcal{U}\) the uncovered lines, \(\mathcal{T}\) the candidate branches, \(b^\ast\) the selected target branch, and \(\ell^\ast\) the target line number. The procedure is:

1. collect uncovered lines:
   \[
   \mathcal{U} \leftarrow \{\ell \mid \text{coverage\_count}(\ell) = 0 \}
   \]
2. gather branches whose bodies are entirely uncovered
3. score each candidate by body size:
   \[
   s_b \leftarrow |LinesInBranchBody(b)|
   \]
4. choose the branch with maximal score
5. if no such branch exists, choose an uncovered standalone line not in any branch body, preferring the smallest line number; otherwise discard the program [2604.20917]

This means DexBench defaults to targeting counterfactual paths with **maximal coverage gain**.

The resulting benchmark contains **445 unique paired instances total**: **298** from CruxEval, **100** from HumanEval, and **47** from PythonSaga. Programs range from **10 to 78 lines of code**, with mean cyclomatic complexity

\[
3.63 \pm 2.18
\]

and complexity range **2 to 19** [2604.20917].

## 4. Prompting interface, evaluation, and scoring

DexBench uses two dedicated prompt templates. In the **forward prompt**, the model sees a line-numbered Python program and is asked to determine all executable line numbers run at least once. It must return JSON containing `executed_lines` as a sorted list of integers [2604.20917]. In the **backward prompt**, the model sees the same line-numbered program, the original seed input, and a target line, and it must return only the comma-separated argument values for a mutated input that causes the target line or body to execute [2604.20917]. All experiments use **one-shot prompting** with one illustrative example.

For both tasks, the evaluation metric is **\(\mathrm{pass@}k\)**. The forward success indicator is

\[
S_{exec}\!=\!\mathbbm{1}\!\left(\exists \tilde{\phi}_i\!\in\!\{\tilde{\phi}_1, ..., \tilde{\phi}_k\} \; \text{s.t.} \; \tilde{\phi}_i\!=\!\phi(\tau_{exec}) \right),
\]

meaning that at least one of the \(k\) candidate predictions must exactly match the true coverage set [2604.20917]. The backward success indicator is

\[
S_{cf}\!=\!\mathbbm{1} \left( \exists I_{cf, i}\!\in\!\{I_{cf, 1}, ..., I_{cf, k}\} \; \text{s.t.} \; b\!\in\!\phi(\tau_{cf, i}) \right),
\]

meaning that at least one generated input must actually reach the target branch [2604.20917].

The key aggregate metric is the **joint dual success**

\[
S_{dual} = S_{exec} \wedge S_{cf}.
\]

An instance is counted as solved only if the model succeeds on **both** the execution and counterfactual tasks [2604.20917]. This is the benchmark’s main scoring idea and the basis for its claim to be more discriminative than isolated evaluation.

For analysis, the paper also introduces a relaxed execution score based on Jaccard similarity:

\[
S_{exec}^{r} = \frac{|\tilde{\phi} \cap \phi(\tau_{exec})|}{|\tilde{\phi} \cup \phi(\tau_{exec})|}.
\]

This is not the main score, but it captures partial overlap between predicted and true coverage sets and is useful for diagnosing near misses [2604.20917].

A notable feature of DexBench is that verification is **execution-grounded**. Forward answers are checked by exact match to observed coverage, while backward answers are checked by actual execution of the generated input [2604.20917]. This makes the benchmark stricter than tasks that score only plausibility or heuristic similarity.

## 5. Empirical findings and model behavior

The paper evaluates **13 LLMs**: nine open-source models—Jamba Reasoning 3B, Llama-3.1 Nemotron Nano, Llama-3.2-3B-Instruct, Mistral Small 24B, Magistral Small, QwQ-32B, Qwen2.5-32B, Llama-3.3-70B-Instruct, and Qwen2.5-72B—and four closed-source models—Gemini 2.5 Flash, GPT-5 Mini, Grok-4 Reasoning, and Claude Sonnet 4 [2604.20917].

A central result is that **dual-path reasoning is substantially harder than either component alone**. The paper explicitly reports large populations of **execution-only successes** and **counterfactual-only successes**, indicating that strong performance on a single task does not imply robust causal understanding of execution flow [2604.20917]. This is one of the benchmark’s main empirical arguments: isolated task performance can overestimate true program-execution understanding.

Performance declines with program complexity. Across the three source datasets, dual performance generally worsens from

\[
\text{CruxEval} < \text{HumanEval} < \text{PythonSaga},
\]

reflecting the increasing difficulty of maintaining a consistent causal model over programs with deeper control flow, more nesting, recursion, and more advanced constructs [2604.20917].

Small models perform poorly on the joint task. All sub-10B models show near-zero dual performance, largely because they fail on exact code coverage prediction even when they occasionally succeed at branch-reaching mutation [2604.20917]. This suggests that accurate execution simulation remains a major bottleneck.

Scaling is not monotonic. A prominent result is that **Qwen2.5-32B** outperforms **Qwen2.5-72B** on dual-path reasoning across all datasets [2604.20917]. Likewise, reasoning-focused post-training does not reliably help: **Mistral Small 24B** outperforms **Magistral Small**, and **Qwen2.5-32B** outperforms **QwQ-32B** [2604.20917]. A plausible implication is that current reasoning-oriented post-training procedures do not necessarily transfer to causal reasoning about program execution.

Among frontier systems, the best overall dual-path performance came from closed-source models. The reported ranking is

\[
\text{Claude Sonnet 4} > \text{Grok-4 Reasoning} > \text{GPT-5 Mini} > \text{Gemini 2.5 Flash}
\]

[2604.20917].

Representative **dual reasoning, \(\mathrm{pass@1}\)** results illustrate both absolute capability and residual difficulty:

| Dataset | Claude Sonnet 4 | Grok-4 Reasoning | GPT-5 Mini | Gemini 2.5 Flash |
|---|---:|---:|---:|---:|
| CruxEval | 87.6 | 79.9 | 73.8 | 51.0 |
| HumanEval | 69.0 | 65.0 | 59.0 | 41.0 |
| PythonSaga | 55.3 | 51.1 | 29.8 | 4.3 |

Even the strongest models degrade substantially on PythonSaga [2604.20917]. This indicates that DexBench remains challenging even for high-performing systems.

## 6. Failure modes, ablations, and relation to prior benchmarks

DexBench’s qualitative analyses identify multiple failure types: **computation errors, control flow errors, lack of fact verification, native API misevaluation, prompt misread, null response, output format errors, misreporting final output, skipping statements, and hallucinations** [2604.20917]. The distribution of failures differs by task. Execution failures often involve prompt misreading, skipping statements, hallucinated lines, or null responses, whereas counterfactual failures often involve null responses, formatting mistakes, and semantic misunderstandings of APIs or control flow.

The paper’s examples illustrate what the benchmark is intended to measure. In one case, a model correctly identifies the relevant branch structure in a character-counting program but miscounts occurrences of a character, leading to an invalid mutation; this is categorized as a **computation error** [2604.20917]. In another, a model incorrectly reasons about Python’s `splitlines()` behavior and proposes a mutation that would not trigger the target `break`; this is classified as **native API misevaluation** [2604.20917]. These examples show that DexBench is not merely about branch guessing, but about whether the model causally understands how an input mutation changes actual runtime behavior.

An ablation comparing **input mutation** with unconstrained **input generation** found that direct generation can perform better than mutation for GPT-5 Mini and Grok-4 Reasoning [2604.20917]. The authors interpret this as support for DexBench’s harder design: mutation is more causally demanding because it requires the model to reason about how specific changes to an existing input propagate through the execution path. This suggests that DexBench intentionally prioritizes counterfactual faithfulness over raw branch-reaching convenience.

Prompt sensitivity is also evident. A stronger execution prompt from CRISPE improves execution-only performance, especially when finite-option coverage choices are available [2604.20917]. However, the paper argues that the dual structure remains informative because it continues to test joint reasoning ability even when prompt engineering shifts absolute scores.

The paper situates DexBench relative to prior execution benchmarks. It explicitly contrasts the benchmark with **CruxEval**, which pairs output prediction and input prediction but not around shared execution-path causality in the same way; **R-Eval / IC-Score**, which focus on execution-related tasks but still on single-path behaviors; and **CES**, which reasons over multiple inputs but treats execution paths independently rather than as divergences in a shared execution space [2604.20917]. DexBench’s contribution, in this framing, is not merely a new dataset, but a different causal organization of evaluation.

## 7. Limitations, scope, and significance

The paper notes several limitations. First, programs are drawn from **public datasets**, so contamination cannot be eliminated entirely, even if the dual-path design mitigates simple memorization [2604.20917]. Second, DexBench is currently **Python-only**, largely because its instrumentation pipeline relies on tools such as Slipcover [2604.20917]. Third, the benchmark uses one specific pair of behaviors—**statement coverage** for forward reasoning and **branch-targeted mutation** for backward reasoning—although the broader dual-path framework could in principle be instantiated with other observables, such as outputs, intermediate states, or post-conditions [2604.20917]. Fourth, the default counterfactual target-selection heuristic favors paths with maximal coverage gain, which may bias the benchmark toward certain kinds of divergence [2604.20917].

These limits are substantive but do not negate the benchmark’s conceptual contribution. DexBench advances the evaluation of code-capable LLMs by formalizing the idea that execution understanding involves both the realized path and the unrealized alternative. Its emphasis on **paired instances**, **execution-grounded verification**, and the **joint metric** \(S_{dual}\) makes it more stringent than benchmarks that test only one side of execution reasoning [2604.20917].

DexBench is therefore best understood as a benchmark for **causal, state-aware reasoning about execution flow** rather than a generic code-generation or code-completion benchmark. It is relevant to research on debugging, testing, coverage-guided exploration, fuzzing-like reasoning, and execution-aware code agents, because these settings require more than predicting outputs: they require understanding why a branch was or was not taken, and how to manipulate inputs to induce desired execution outcomes [2604.20917]. A plausible implication is that future benchmark design in code intelligence may increasingly center such bidirectional formulations, especially where causal understanding rather than surface regularity is the target capability.

Source: https://www.emergentmind.com/topics/dexbench