Execution Accuracy (EX)

• Execution Accuracy (EX) is a metric that quantifies how often a generated code or query produces the expected output under execution tests.
• It is operationalized via methods like pass@1, unit tests, and white-box execution to assess both syntactic and semantic correctness across various domains.
• Despite its robustness, EX can yield false positives or negatives, necessitating augmented metrics such as FLEX for improved evaluation of true functional performance.

Execution Accuracy (EX) is a primary metric for evaluating the functional correctness of generated code or program predictions, measuring the fraction of examples for which the output of an executable artifact—code, query, or intermediate state—matches the expected ground truth under execution. EX provides a robust evaluation of models’, algorithms’, or systems’ ability to capture not only the syntax but also the precise semantics of the target computational behavior. Across different domains, EX may refer to the output-level, program-trace–level, intermediate step–level, or query-result–level match against an authoritative reference, and is often operationalized as a pass@k (typically k=1) aggregated over a benchmark suite.

1. Formal Definitions of Execution Accuracy

Across code reasoning, code generation, code translation, and semantic parsing, EX is consistently defined as the mean over a dataset of the indicator that a candidate output passes all execution-based correctness requirements. Several representative formalisms are in use:

• Standard code output prediction (pass@1):

$$\mathrm{EX} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}[\hat y_i = y_i]$$

where $N$ is the number of examples, $y_i$ the ground-truth output, and $\hat y_i$ the model’s prediction (Armengol-Estapé et al., 10 Feb 2025, Xu et al., 2024).

• Unit-test harness for code translation:

$$\mathrm{EX} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\big[\forall j \in \{1...K\},\, \hat f_i(x_{i,j}) = f_i(x_{i,j})\big]$$

with $K$ test inputs per example (He et al., 30 Jan 2025).

• Trace-level or stepwise accuracy (white-box):

$$\mathrm{EX} = \frac{1}{M} \sum_{i=1}^M \frac{1}{|Q^i|} \sum_{q \in Q^i} \mathbb{I}[a_q = a_q^*]$$

where $Q^i$ is a set of intermediate execution questions, $a_q$ model prediction, and $a_q^*$ ground truth (Tang et al., 11 Mar 2026).

• SQL query execution:

$$\mathrm{EX} = \frac{1}{|X|} \sum_{x \in X} \mathbb{I}[R_\mathrm{gen}(x) = R_\mathrm{gt}(x)]$$

where $R_\mathrm{gen}(x)$ and $R_\mathrm{gt}(x)$ are result sets from executing the generated and ground truth SQL on the underlying database (Kim et al., 2024).

• Selective execution in D²NNs:

$$\mathrm{EX}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathbb{I}[\,\hat y_i(\theta) = y_i\,]$$

measured under a learned input-dependent partial network execution policy (Liu et al., 2017).

These definitions enable direct aggregation and comparison of models’ capability to produce outputs that are not only syntactically plausible but also functionally correct by the criterion of actual execution.

2. Methodologies for Measuring Execution Accuracy

Typical EX assessment requires constructing or employing a benchmark in which each task or example is equipped with either a test oracle (unit tests, desired output, assertion cases) or a procedure to compute an authoritative trace/result. The methodologies differ based on application area:

• Unit test–based harnessing: Used in code translation and multilingual code reasoning benchmarks—candidate code is run on multiple input test cases, and EX is the fraction of candidates passing all cases (He et al., 30 Jan 2025, Xu et al., 2024).
• White-box execution questions: For code reasoning tasks, EX may be computed over intermediate interpreter state (next-statement or data-flow questions), requiring automated extraction of ground-truth traces and efficient batchwise evaluation (Tang et al., 11 Mar 2026).
• SQL and database queries: EX is determined by executing both the ground-truth and predicted SQL on a database instance and comparing the result sets. Extensions (see FLEX) invoke LLM-based adjudication to resolve result-set equivalence and annotation flaws (Kim et al., 2024).
• Domain-specific execution (dynamic evaluation): For transaction script generation or chain-specific code, EX involves executing code on a sandboxed (possibly forked) environment and checking for semantic/functional correctness (details omitted here due to lack of explicit metric definitions in (Yang et al., 10 Jan 2026)).

The widespread use of pass@k, especially pass@1, unifies EX metrics across languages, paradigms, and benchmarks, facilitating rigorous empirical comparison.

3. Extensions and Limitations of EX

Despite its widespread adoption and apparent objectivity, EX has significant limitations:

• False Positives: EX may assign credit when a predicted output appears correct for incidental reasons (e.g., overfitted query or code matches on spurious database state or trivial test cases), without reflecting true semantic correctness (Kim et al., 2024).
• False Negatives: EX may penalize correct alternatives (e.g., different but semantically equivalent SQL, or code with permuted variable names or column order) due to annotation or output format inflexibility (Kim et al., 2024).
• Intrinsic undecidability: For code or logic tasks, EX only approximates semantic equivalence by using finite test suites and cannot guarantee completeness.
• Step-level faithfulness: In code reasoning, EX at the final output level may mask failures or “lucky” guesses at intermediate steps; thus, fine-grained white-box EX is needed to assess true program-comprehension (Tang et al., 11 Mar 2026).

To address these, new metrics such as FLEX (False-Less EXecution) augment EX by leveraging LLMs with explicit reasoning criteria and adjudication contexts to reduce both false positives and negatives, bringing automated assessment closer to human expert judgment (Kim et al., 2024).

4. Empirical Benchmarks and Model Results

Execution accuracy is universally used in benchmarks for code generation, translation, execution reasoning, and text-to-SQL:

Benchmark	Domain	EX Results (SOTA)	Notable Features
TransCoder-test-X	Code translation	ExeCoder: 83.04%	6 directions, 948 functions
CRUXEval-X	Multilingual code	GPT-4o: 85–98% (Python)	19 languages, pass@1 metric
MBPP	Code synthesis	MBR-EXEC: 58.2%	3 assertion test inputs
Spider/BIRD	Text-to-SQL	FLEX raises EX by 2.6 pts	LLM-based semantic adjudication
D²NN (LFW-B)	Dynamic NNs	F₁: [email protected]–1.0	Input-dependent selective exec

Top-tier LLMs (GPT-4o, DeepseekCoder-V2) achieve near-perfect EX in familiar languages; cross-lingual EX is lower for languages with unique semantics (e.g., Racket, D) (Xu et al., 2024). Use of execution-tuning and fine-grained trace modeling substantially raises EX relative to direct function output prediction (Armengol-Estapé et al., 10 Feb 2025).

5. Role in Training Objectives and Model Selection

EX not only functions as an evaluation metric but is increasingly integrated into model selection and training:

• Execution-based Minimum Bayes Risk (MBR-EXEC): Sampling diverse program candidates and selecting the one with maximal empirical execution consistency on held-out test inputs significantly boosts EX over standard likelihood-based methods (Shi et al., 2022).
• Reinforcement Learning with stepwise EX rewards: Rewards that blend EX at the output and intermediate step levels, as in ExecVerify, directly align the learning objective with semantic program execution (Tang et al., 11 Mar 2026).
• Selective execution for efficiency–accuracy tradeoff: D²NNs employ EX within composite reward functions that balance computation cost and per-input module gating, optimizing for high EX under resource constraints (Liu et al., 2017).

These developments indicate a trend towards coupling functional correctness with model probability and policy optimization, moving beyond mere token-level loss.

6. Evolving Best Practices and Future Directions

Recent advances prompt several recommendations for the use and interpretation of EX:

• Augment raw EX with LLM-based expert evaluation (FLEX): This approach better captures semantic equivalence, error modes, and ambiguous specifications, yielding leaderboard re-calibrations and greater correlation with human consensus (Kim et al., 2024).
• Context-aware harness and test construction: Benchmarks must minimize test-case and annotation noise, ensure cross-language meaningfulness, and provide sufficiently challenging/edge-case inputs to improve EX validity (He et al., 30 Jan 2025, Xu et al., 2024).
• Integration with long-context and trace-level modeling: Especially for code reasoning with deep/nested traces, efficient representations (dynamic scratchpad, compact diffing) enable models to maintain high EX even for thousands of steps (Armengol-Estapé et al., 10 Feb 2025).
• Differentiation across task types: While output-level EX suffices for simple program synthesis, finer-grained (white-box, stepwise) EX is necessary for reasoning, verification, and explainability tasks (Tang et al., 11 Mar 2026).
• Transparent reporting of aggregation, harness details, and error exemplars: Ensures replicability and enables precise diagnostics across models, datasets, and languages (Xu et al., 2024, He et al., 30 Jan 2025).

Plausibly, as model architectures and evaluation scenarios continue to diversify, EX—and its augmented forms—will remain central to functional correctness assessment, with ongoing refinement required to match the increasing sophistication and subtlety of generative systems.