Execution-Based Evaluation Metrics

• Execution-Based Evaluation Metrics are methods that assess system outputs by running them in controlled environments, ensuring they meet real-world task specifications.
• They employ techniques like pass@k, container-based harnesses, and dense scoring to measure accuracy, efficiency, and consistency in fields such as code generation and program repair.
• Empirical results show these metrics better capture functional correctness compared to surface similarity measures, revealing meaningful performance gaps in complex computational tasks.

Execution-based evaluation metrics assess the functional correctness, reliability, and real-world utility of models or systems by running their outputs in situ—on code, artifacts, workflows, or task environments—and scoring the success or fidelity of the actual execution rather than proxying via pattern-matching or surface similarity. In computational domains such as code generation and program repair, execution-based metrics have become the de facto standard for measuring whether candidate outputs meet task specifications, pass unit tests, or achieve goals in their designated environment. These metrics are now core to benchmarks in programming languages, data science, scripting, dialog systems, workflow scheduling, LLM subjective task-following, and robotics, enabling a shift from brittle symbolic matching toward robust, semantically grounded evaluation.

1. Formal Definitions of Core Execution-Based Metrics

The foundation of execution-based code evaluation is the pass@k metric, which quantifies the probability that at least one of k independently sampled outputs from a LLM (or program repair system) passes all available unit tests for a problem. The canonical unbiased estimator (for sampling without replacement) is: $$\mathrm{pass}@k = 1 - \prod_{i=0}^{k-1} \frac{n-c-i}{n-i}$$ where $n$ is the total number of samples, $c$ is the number that pass all tests, and $k$ is the pool size used for metric calculation. For $k=1$, pass@1 (“execution accuracy”) reduces to the proportion of problems where the single best output passes all unit tests: $$\mathrm{ExecAcc} = \mathrm{pass}@1 = \frac{1}{M}\sum_{j=1}^M \mathbf{1}\{\text{model’s top-1 output for problem }j\text{ passes all tests}\}$$ These metrics are always computed relative to a set of human-written or curated test cases, which execute the generated code on concrete, bounded input/output pairs and ascertain both correctness and failure modes (Wang et al., 2022, Haque et al., 2022, Huang et al., 2022, Gu et al., 2024, Yan et al., 2023, Khan et al., 2023).

In non-code domains, such as task-oriented dialogue or enterprise workflows, execution-based correctness can extend to complex metrics: for DataFlow systems, “Execution Accuracy” (EA) is the fraction of dialogue turns for which the entire predicted program, when executed, produces both the correct world state and the correct agent response (He et al., 2022). For workflow scheduling and systems, the primary execution-based metrics are wall-clock application makespan, per-task latency, task throughput, and the fraction of successfully completed tasks under real execution conditions (Pauloski et al., 2024).

2. Methodologies for Metric Computation and Benchmark Design

Execution-based metrics rely on practical, standardized test harnesses. In code domains, annotators refactor code snippets into callable functions, specify immutably defined imports and environments, and author diverse, domain-representative test suites to exhaustively challenge submitted solutions, typically enforcing edge-case robustness (e.g., bounding numeric precision, library-aware equality, mocking, and randomized assertions). For each sampled completion, the entire test suite is run, and a sample is marked “pass” only if no exceptions are raised and all assertions succeed (Wang et al., 2022, Khan et al., 2023, Yan et al., 2023, Huang et al., 2022).

Execution platforms such as MultiCodeEngine (Yan et al., 2023), ExecEval (Khan et al., 2023), or container-based harnesses (Vo et al., 2024, González et al., 23 Feb 2026, Pauloski et al., 2024), instrument every phase—from compilation and I/O capture to error handling and metrics aggregation—across multiple programming languages, scripting environments, or system APIs. For text-based LLM instruction following, controlled parallel reference code is used to determine not only final answer correctness but also adherence to detailed rubric steps (format, step-by-step following, depth of intermediate result reporting) (Moon et al., 9 Oct 2025).

Benchmark splits use custom data balancing schemes (e.g., geometric-mean tag balancing, bounded-flow/circulation-based sampling) to control for domain, tag, language, and difficulty skew (Khan et al., 2023).

3. Extensions Beyond Binary Correctness: Fine-Grained, Subjective, and Dense Metrics

Recent advances stretch execution-based evaluation into multi-dimensional and more subjective domains:

• Multidimensional and Multilingual Code Assessment: CodeScope augments pass@k with “Same-Output Rate” (SOR) to measure output consistency, and Opt@k to directly quantify efficiency, such as time and memory usage improvements, by actually executing code and comparing with reference or original solutions (Yan et al., 2023). These metrics support per-language, per-length, and per-difficulty aggregation, crucial for robust evaluation in multilingual, multitask settings.
• Stepwise and Intermediate Artifact Scoring: Frameworks such as LH-Bench deploy expert-designed rubrics and ground-truth artifact annotation to create scalar, tiered reward signals covering both intermediate process and final output fidelity in long-horizon or open-ended workflows. Stepwise rewards ($r_t$ per episode step), rubric-weighted process scores, and pairwise human preferences (statistically aggregated via Bradley–Terry or Elo modeling) enable reliable, execution-aligned evaluation even on tasks where no standard unit test applies (Chandwani et al., 24 Mar 2026).
• Dense Robotic Evaluation (OPD Metrics): Beyond binary success, dense evaluation via Process Reward Models (PRMs) learns a potential function $\Phi: X \rightarrow [0,1]$ over observation streams, decomposing trajectories into outcome, process, and diagnosis layers (e.g., Milestone Coverage, Path-weighted Progress Length, Stagnation Ratio), enforcing macro-consistency and micro-resolution for granular, additive, and path-invariant analysis of robotic policies (Ji et al., 23 Mar 2026).
• Robustness and Proxy Metrics: As executing all test cases may be costly or unsafe, automated learned proxies such as CodeScore (Dong et al., 2023) and CodeScore-R (Yang et al., 2024) use LLMs or neural encoders trained to approximate pass@k via contrastive learning on reference/predicted code pairs, AST sketching, and mutation-based negatives, attaining high correlation with true execution outcomes on a fraction of the execution cost.

4. Empirical Findings, Model Behavior, and Analysis

Consistent empirical findings across code generation, bug repair, data science, and scripting tasks highlight several key points:

• Execution-based metrics such as pass@k, TCA@k, and execution accuracy are robust to surface-level differences: Functionally correct but lexically or structurally divergent solutions are rewarded, while “surface matches” that fail to run or yield the required output are penalized (Wang et al., 2022, Huang et al., 2022, Haque et al., 2022).
• Surface-form metrics can be systematically misleading: High BLEU or CodeBLEU scores are compatible with zero pass@k, exposing major false positives and false negatives in evaluation and motivating exclusive reliance on execution-based measures.
• Performance under these metrics varies sharply across domains, languages, and scaling: Open vs. closed domain accuracy gaps, language-dependent pass rates, and scale-driven performance shifts are consistently observed (e.g., Codex’s open/closed domain gap increases with model size, while CodeGen’s narrows) (Wang et al., 2022).
• Difficulty and problem length create steep drop-offs: On Codeforces-derived “hard” problems, pass@k in generation and repair tasks plunges, with high-performing models on easy problems often dropping to near-zero on hard ones (Yan et al., 2023, Khan et al., 2023).
• Robustness to perturbations: Embedding- and contrastive-learning-based proxies (e.g., CodeScore-R) outperform text- or tree-based metrics under identifier, syntax, and semantic mutations, maintaining close alignment to execution-based ground truth (Yang et al., 2024).
• For workflows or robotic tasks, dense metrics capture nuances invisible to binary outcomes: Fine-grained OPD scores distinguish between “late-stage” and “early” failures, execution hesitancy, or recovery, providing actionable diagnostics for model improvement (Ji et al., 23 Mar 2026, Chandwani et al., 24 Mar 2026).

5. Limitations, Pitfalls, and Open Challenges

Despite their advantages, execution-based evaluation faces inherent trade-offs:

• Annotation and infrastructure cost: Building high-quality, comprehensive, environment-specific test suites is labor-intensive, particularly in open-domain or data-rich settings (Wang et al., 2022, Huang et al., 2022, Yan et al., 2023).
• Coverage and corner cases: While single test cases may gate main functionality, under-coverage of edge-case or adversarial situations is a common risk, and full correctness is only as strong as the test suite (Wang et al., 2022, Huang et al., 2022).
• Metric sensitivity to sampling/diversity: pass@k conflates semantic diversity, sampling noise, and model quality; higher k values mechanically inflate scores even without underlying model improvement (Wang et al., 2022).
• Potential overfitting to tests: Models may exploit spurious cues or overfit to publicized benchmarks/tests, distorting real-world performance.
• Non-functional properties: Standard execution-based metrics are agnostic to code performance, resource consumption, maintainability, and side effects unless explicitly instrumented (e.g., efficiency metrics in CodeScope or TaPS) (Yan et al., 2023, Pauloski et al., 2024).
• Proxy limitations: LLM-based execution-free metrics rely on the availability of high-quality references and may underperform on unseen languages or tasks due to representation bias or incomplete code-embedding robustness (Yang et al., 2024, Yadavally et al., 28 Jan 2025).

6. Future Directions and Best Practices

Recommendations and best practices emerging from current literature include:

• Always report pass@k (and related metrics) in addition to any surface similarity scores to ensure evaluation reflects functional outcomes (Wang et al., 2022, Haque et al., 2022, Huang et al., 2022).
• Standardize candidate/sample sizes and use unbiased estimators to enable pipelined, comparable benchmarks (Wang et al., 2022, Haque et al., 2022, Khan et al., 2023).
• Leverage trial-based filtering and execution-based reranking at inference to close most of the gap to oracle performance, and, where possible, employ self-debugging routines on multiple candidates to harvest extra headroom (Li et al., 2024).
• Expand evaluation to dense, process- and artifact-level metrics for long-horizon, subjective, or multi-agent tasks, ensuring that intermediate steps and partial progress are scored (Ji et al., 23 Mar 2026, Chandwani et al., 24 Mar 2026).
• Complement execution-based metrics with artifact and behavior rubrics, as well as human and pairwise judgments, especially in subjective or creative domains (Chandwani et al., 24 Mar 2026, Moon et al., 9 Oct 2025).
• Maintain modular, reproducible evaluation infrastructure: containerization, isolated execution, and careful handling of state and dependencies are critical for correctness and safety (Vo et al., 2024, González et al., 23 Feb 2026).
• Continue to develop robust, rapid, and test-free proxy metrics to scale evaluation in the absence of test cases, with an emphasis on robustness to code mutations and adversarial variation (Dong et al., 2023, Yang et al., 2024, Yadavally et al., 28 Jan 2025).

Execution-based evaluation metrics now underpin objective, scalable, and functionally faithful assessment across a broad array of computational tasks, and are under active extension to more subjective, open-domain, and artifact-rich settings in both code intelligence and embodied AI research.