---
title: Execution Accuracy (EX)
url: https://www.emergentmind.com/topics/execution-accuracy-ex
type: topic
---

# Execution Accuracy (EX)

Execution Accuracy (EX) is a primary metric for evaluating the functional correctness of generated code or program predictions, measuring the fraction of examples for which the output of an executable artifact—code, query, or intermediate state—matches the expected ground truth under execution. EX provides a robust evaluation of models’, algorithms’, or systems’ ability to capture not only the syntax but also the precise semantics of the target computational behavior. Across different domains, EX may refer to the output-level, program-trace–level, intermediate step–level, or query-result–level match against an authoritative reference, and is often operationalized as a pass@k (typically k=1) aggregated over a benchmark suite.

## 1. Formal Definitions of Execution Accuracy

Across code reasoning, code generation, code translation, and semantic parsing, EX is consistently defined as the mean over a dataset of the indicator that a candidate output passes all execution-based correctness requirements. Several representative formalisms are in use:

- **Standard code output prediction (pass@1):**
  $$
  \mathrm{EX} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}[\hat y_i = y_i]
  $$
  where $N$ is the number of examples, $y_i$ the ground-truth output, and $\hat y_i$ the model’s prediction [2503.05703], [2408.13001].

- **Unit-test harness for code translation:**
  $$
  \mathrm{EX} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\big[\forall j \in \{1...K\},\, \hat f_i(x_{i,j}) = f_i(x_{i,j})\big]
  $$
  with $K$ test inputs per example [2501.18460].

- **Trace-level or stepwise accuracy (white-box):**
  $$
  \mathrm{EX} = \frac{1}{M} \sum_{i=1}^M \frac{1}{|Q^i|} \sum_{q \in Q^i} \mathbb{I}[a_q = a_q^*]
  $$
  where $Q^i$ is a set of intermediate execution questions, $a_q$ model prediction, and $a_q^*$ ground truth [2603.11226].

- **SQL query execution:**
  $$
  \mathrm{EX} = \frac{1}{|X|} \sum_{x \in X} \mathbb{I}[R_\mathrm{gen}(x) = R_\mathrm{gt}(x)]
  $$
  where $R_\mathrm{gen}(x)$ and $R_\mathrm{gt}(x)$ are result sets from executing the generated and ground truth SQL on the underlying database [2409.19014].

- **Selective execution in D²NNs:**
  $$
  \mathrm{EX}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathbb{I}[\,\hat y_i(\theta) = y_i\,]
  $$
  measured under a learned input-dependent partial network execution policy [1701.00299].

These definitions enable direct aggregation and comparison of models’ capability to produce outputs that are not only syntactically plausible but also functionally correct by the criterion of actual execution.

## 2. Methodologies for Measuring Execution Accuracy

Typical EX assessment requires constructing or employing a benchmark in which each task or example is equipped with either a test oracle (unit tests, desired output, assertion cases) or a procedure to compute an authoritative trace/result. The methodologies differ based on application area:

- **Unit test–based harnessing:** Used in code translation and multilingual code reasoning benchmarks—candidate code is run on multiple input test cases, and EX is the fraction of candidates passing all cases [2501.18460], [2408.13001].
- **White-box execution questions:** For code reasoning tasks, EX may be computed over intermediate interpreter state (next-statement or data-flow questions), requiring automated extraction of ground-truth traces and efficient batchwise evaluation [2603.11226].
- **SQL and database queries:** EX is determined by executing both the ground-truth and predicted SQL on a database instance and comparing the result sets. Extensions (see FLEX) invoke LLM-based adjudication to resolve result-set equivalence and annotation flaws [2409.19014].
- **Domain-specific execution (dynamic evaluation):** For transaction script generation or chain-specific code, EX involves executing code on a sandboxed (possibly forked) environment and checking for semantic/functional correctness (details omitted here due to lack of explicit metric definitions in [2601.06565]).

The widespread use of pass@k, especially pass@1, unifies EX metrics across languages, paradigms, and benchmarks, facilitating rigorous empirical comparison.

## 3. Extensions and Limitations of EX

Despite its widespread adoption and apparent objectivity, EX has significant limitations:

- **False Positives:** EX may assign credit when a predicted output appears correct for incidental reasons (e.g., overfitted query or code matches on spurious database state or trivial test cases), without reflecting true semantic correctness [2409.19014].
- **False Negatives:** EX may penalize correct alternatives (e.g., different but semantically equivalent SQL, or code with permuted variable names or column order) due to annotation or output format inflexibility [2409.19014].
- **Intrinsic undecidability:** For code or logic tasks, EX only approximates semantic equivalence by using finite test suites and cannot guarantee completeness.
- **Step-level faithfulness:** In code reasoning, EX at the final output level may mask failures or “lucky” guesses at intermediate steps; thus, fine-grained white-box EX is needed to assess true program-comprehension [2603.11226].

To address these, new metrics such as FLEX (False-Less EXecution) augment EX by leveraging LLMs with explicit reasoning criteria and adjudication contexts to reduce both false positives and negatives, bringing automated assessment closer to human expert judgment [2409.19014].

## 4. Empirical Benchmarks and Model Results

Execution accuracy is universally used in benchmarks for code generation, translation, execution reasoning, and text-to-SQL:
  
| Benchmark           | Domain                | EX Results (SOTA)          | Notable Features                |
|---------------------|----------------------|----------------------------|---------------------------------|
| TransCoder-test-X   | Code translation     | ExeCoder: 83.04%           | 6 directions, 948 functions     |
| CRUXEval-X          | Multilingual code    | GPT-4o: 85–98% (Python)    | 19 languages, pass@1 metric     |
| MBPP                | Code synthesis       | MBR-EXEC: 58.2%            | 3 assertion test inputs         |
| Spider/BIRD         | Text-to-SQL          | FLEX raises EX by 2.6 pts  | LLM-based semantic adjudication |
| D²NN (LFW-B)        | Dynamic NNs          | F₁: 0.87@cost0.43–1.0      | Input-dependent selective exec  |

Top-tier LLMs (GPT-4o, DeepseekCoder-V2) achieve near-perfect EX in familiar languages; cross-lingual EX is lower for languages with unique semantics (e.g., Racket, D) [2408.13001]. Use of execution-tuning and fine-grained trace modeling substantially raises EX relative to direct function output prediction [2503.05703].

## 5. Role in Training Objectives and Model Selection

EX not only functions as an evaluation metric but is increasingly integrated into model selection and training:

- **Execution-based Minimum Bayes Risk (MBR-EXEC):** Sampling diverse program candidates and selecting the one with maximal empirical execution consistency on held-out test inputs significantly boosts EX over standard likelihood-based methods [2204.11454].
- **Reinforcement Learning with stepwise EX rewards:** Rewards that blend EX at the output and intermediate step levels, as in ExecVerify, directly align the learning objective with semantic program execution [2603.11226].
- **Selective execution for efficiency–accuracy tradeoff:** D²NNs employ EX within composite reward functions that balance computation cost and per-input module gating, optimizing for high EX under resource constraints [1701.00299].

These developments indicate a trend towards coupling functional correctness with model probability and policy optimization, moving beyond mere token-level loss.

## 6. Evolving Best Practices and Future Directions

Recent advances prompt several recommendations for the use and interpretation of EX:

- **Augment raw EX with LLM-based expert evaluation (FLEX):** This approach better captures semantic equivalence, error modes, and ambiguous specifications, yielding leaderboard re-calibrations and greater correlation with human consensus [2409.19014].
- **Context-aware harness and test construction:** Benchmarks must minimize test-case and annotation noise, ensure cross-language meaningfulness, and provide sufficiently challenging/edge-case inputs to improve EX validity [2501.18460], [2408.13001].
- **Integration with long-context and trace-level modeling:** Especially for code reasoning with deep/nested traces, efficient representations (dynamic scratchpad, compact diffing) enable models to maintain high EX even for thousands of steps [2503.05703].
- **Differentiation across task types:** While output-level EX suffices for simple program synthesis, finer-grained (white-box, stepwise) EX is necessary for reasoning, verification, and explainability tasks [2603.11226].
- **Transparent reporting of aggregation, harness details, and error exemplars:** Ensures replicability and enables precise diagnostics across models, datasets, and languages [2408.13001], [2501.18460].

Plausibly, as model architectures and evaluation scenarios continue to diversify, EX—and its augmented forms—will remain central to functional correctness assessment, with ongoing refinement required to match the increasing sophistication and subtlety of generative systems.

Source: https://www.emergentmind.com/topics/execution-accuracy-ex