Execution-Grounded Evaluation

• Execution-grounded evaluation framework is an approach where model outputs are validated by executing them in controlled environments, focusing on concrete state transitions and operational semantics.
• It employs substrates such as Docker sandboxes, forked blockchains, and database views to capture and replay the execution trace for auditability and performance verification.
• Empirical studies show that this approach exposes subtle failure modes and improves synthesis accuracy, though it challenges traditional proxy evaluations with issues like environment fidelity.

An execution-grounded evaluation framework is an evaluation paradigm in which model outputs are assessed through execution, state change, or replayable environment interaction rather than through narrative plausibility, static inspection, or final-answer text alone. Recent work uses this paradigm for natural-language transaction-script generation on EVM-compatible chains, financial-agent security evaluation with state-writable databases and explicit compliance constraints, autonomous software engineering with a mandatory Docker sandbox, automated AI research with an implementation $\rightarrow$ compile $\rightarrow$ run loop, DAG-structured agent trace evaluation, deterministic trajectory checking against grounded evidence, research evaluation that examines code and data alongside the paper, and Text-to-SQL schema refinement verified by downstream execution accuracy (Yang et al., 10 Jan 2026, Yang et al., 9 Jan 2026, Kumar et al., 13 Apr 2026, Si et al., 20 Jan 2026, Guo et al., 26 Apr 2026, Flynt, 22 Jun 2026, Bai et al., 5 Feb 2026, Wang et al., 1 May 2026). Across these settings, execution is not an auxiliary check; it is the evaluation object.

1. Definitional core

Several formulations make the same shift explicit. In autonomous software engineering, execution-grounded verification is introduced as a first-class principle: every code change must survive sandboxed execution before propagation (Kumar et al., 13 Apr 2026). In automated AI research, execution grounding is defined as pairing each natural-language research idea with an automatic, end-to-end implementation $\rightarrow$ compile $\rightarrow$ run pipeline so that the ideator model receives real experimental feedback (Si et al., 20 Jan 2026). In financial-agent safety, a successful attack is one that actually flips state or bypasses a rule, not merely changes the model’s text (Yang et al., 9 Jan 2026). In deterministic agent evaluation, the framework scores both the final answer and the recorded trajectory against grounded, time-bounded, and access-controlled evidence (Flynt, 22 Jun 2026). In research evaluation, the framework verifies research beyond narrative review by examining code and data alongside the paper (Bai et al., 5 Feb 2026).

This shift is also visible in benchmark design. EVM-QuestBench is described as an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains; its validators verify outcomes against instantiated values, and its runner executes scripts on a forked EVM chain with snapshot isolation (Yang et al., 10 Jan 2026). CRUXEval defines code-reasoning tasks whose correctness is determined by actual execution of the model’s answer, and DOCE centers execution-based code generation through trial unit-test filtering, execution-based reranking, and self-debugging (Gu et al., 2024, Li et al., 2024).

A plausible implication is that execution grounding replaces proxy evaluation with environment-coupled verification. The object being measured is no longer only whether an answer looks correct, but whether it survives the operational semantics of the target domain.

2. Architectural substrates and validation loops

Execution-grounded frameworks are typically built around an executable substrate, preserved artifacts, and a validator that observes concrete outcomes. The substrate may be a forked blockchain, a Docker-like “vault,” a disposable Docker container, an isolated interpreter, a benchmark judge, or a database view layer (Yang et al., 10 Jan 2026, Yang et al., 9 Jan 2026, Kumar et al., 13 Apr 2026, Huang et al., 30 Oct 2025, Arefin, 7 Jun 2026, Wang et al., 1 May 2026).

Setting	Execution substrate	Grounded signal
EVM-QuestBench	forked EVM chain with snapshot isolation	validators verify outcomes against instantiated values
FinVault	isolated Docker-like “vault” with state-writable database	state change or compliance bypass
AgentForge	disposable Docker sandbox	FAIL$\rightarrow$PASS with no PASS$\rightarrow$FAIL
Nexus	isolated interpreter or Docker container	oracle assertion execution plus runtime errors
Beyond Pass Rate	LeetCode official judge via Selenium + Chromium	judge verdict and per-testcase pass counts
EGRefine	temporary SQL views and downstream execution	execution accuracy under candidate renames

The validation loop is often dynamic rather than fixed. EVM-QuestBench samples instructions from template pools, draws numeric parameters from predefined intervals, and verifies outcomes against those instantiated values (Yang et al., 10 Jan 2026). FinVault builds 31 regulatory, case-driven sandbox scenarios, each with a toolset, privilege levels, compliance rules, and immutable audit logging (Yang et al., 9 Jan 2026). AgentForge applies candidate patches to repository states, executes them in a mandatory Docker sandbox, stores stdout, stderr, and test results in execution history, and propagates only successful patches (Kumar et al., 13 Apr 2026). EGRefine creates a temporary view for each candidate rename, runs Text-to-SQL models on the modified schema, executes predicted SQL, and accepts a rename only under a conservative selection rule (Wang et al., 1 May 2026).

Artifact preservation is a second recurring pattern. The multilingual code-evaluation framework behind “Beyond Pass Rate” stores prompt text, raw model response, extracted code, submission metadata, LeetCode judge result tags, static-analysis signals, and structural metrics, explicitly preserving the entire chain of artifacts for auditability and downstream analysis (Arefin, 7 Jun 2026). AgentEval likewise evaluates each step with access to upstream outputs but not upstream scores, which is intended to prevent bias toward agreeing with prior attributions (Guo et al., 26 Apr 2026). GroundEval emits structured per-question diagnostics that pair tool activity with the agent’s turn-level narration, making each score inspectable rather than merely reported (Flynt, 22 Jun 2026).

This suggests that execution grounding is not only about running code or tools. It is equally about capturing the operational trace in a form that supports replay, inspection, and attribution.

3. Formalizations, metrics, and scoring functions

The literature does not use a single mathematical template, but several formalizations recur. AgentForge models software engineering with LLMs as an iterative decision process over repository states, with state $s_t = (\mathcal{R}_t,\mathcal{M}_t,\mathcal{H}_t)$, action $a_t$ as a candidate patch, transition $s_{t+1}=E(s_t,a_t)$ through isolated execution, and binary reward $r_t=1$ only if all previously failing tests now pass and no existing tests regress (Kumar et al., 13 Apr 2026). “Towards Execution-Grounded Automated AI Research” defines execution grounding by mapping an idea $\rightarrow$0 to a code diff $\rightarrow$1 and then to a measured performance metric $\rightarrow$2, with reward $\rightarrow$3; in its experiments, $\rightarrow$4 is accuracy, $\rightarrow$5, or $\rightarrow$6 if execution failed (Si et al., 20 Jan 2026).

Security and agent-evaluation frameworks often separate outcome scoring from trajectory or root-cause analysis. FinVault defines Attack Success Rate over adversarial inputs by checking whether a vulnerability predicate $\rightarrow$7 at some turn, and supplements ASR with Vulnerability Compromise Rate, True Positive Rate, False Positive Rate, average tokens consumed per case, and average time per case (Yang et al., 9 Jan 2026). GroundEval defines two deterministic scores per question, $\rightarrow$8 and $\rightarrow$9, combines them with a track-specific weight, and then applies a compliance adjustment

$\rightarrow$0

where $\rightarrow$1 is the aggregate violation rate across actor-gate, subsystem, and horizon violations (Flynt, 22 Jun 2026).

Step-level frameworks introduce graph structure and typed metrics. AgentEval formalizes an agent execution as an evaluation DAG whose nodes correspond to single agent steps, whose edges encode uses-output-of dependencies, and whose node scores are produced by a calibrated LLM judge over typed metric sets (Guo et al., 26 Apr 2026). It reports Failure Detection Recall, False Positive Rate, Human Agreement measured by Cohen’s $\rightarrow$2, and Root Cause Accuracy (Guo et al., 26 Apr 2026). In contrast, the multilingual code-evaluation framework “Beyond Pass Rate” emphasizes per-job and per-problem statistics, including Mean Correctness, Pass Rate, Accepted-Problem Coverage, and Lint Pass Rate, with slice-dependent variants by difficulty and language (Arefin, 7 Jun 2026). MechEvalAgent uses binary checklist aggregation across three independent runs, logical AND over checklist items, and an acceptable-tolerance reproducibility check with $\rightarrow$3 for reported numeric metrics (Bai et al., 5 Feb 2026).

A plausible synthesis is that execution-grounded evaluation often decomposes into at least two layers: a functional layer that measures whether the externally visible task succeeded, and an operational layer that measures whether the correct state transitions, evidence accesses, or intermediate steps produced that success.

4. Domain-specific realizations

Execution-grounded evaluation has been instantiated across transaction systems, regulated agent settings, software engineering, code-generation assessment, automated research, research evaluation, and database interfaces.

In transaction and finance settings, the emphasis is on mutable state and irreversible consequences. EVM-QuestBench contains 107 tasks, with 62 atomic and 45 composite, and evaluates natural-language transaction-script generation on EVM-compatible chains through execution on a forked chain with snapshot isolation; composite tasks apply step-efficiency decay, and the reported split scores reveal persistent asymmetry between single-action precision and multi-step workflow completion (Yang et al., 10 Jan 2026). FinVault extends the same logic to financial agents through 31 regulatory sandbox scenarios spanning Credit & Lending, Insurance Services, Securities & Investment, Payments & Settlement, AML & Compliance, and Risk Management, with 107 real-world vulnerabilities and 963 test cases (Yang et al., 9 Jan 2026).

In software engineering, execution grounding operates at the level of repositories, tests, and repair loops. AgentForge coordinates Planner, Coder, Tester, Debugger, and Critic agents through shared memory and a mandatory Docker sandbox, and frames patching as a verified state transition process (Kumar et al., 13 Apr 2026). Nexus uses multi-agent deliberation for test oracle synthesis, then generates a plausible candidate implementation of the function under test and executes each oracle in a secure sandbox; any failed oracle enters an automated self-refinement loop driven by runtime errors (Huang et al., 30 Oct 2025). DOCE integrates candidate generation, trial unit-test filtering, execution-based reranking, minimum Bayes risk decoding, and self-debugging as a unified framework for execution-based code generation (Li et al., 2024). CRUXEval provides a more compact execution benchmark in which correctness is determined by assert f(x)==y or by executing a predicted input and checking that it maps to the desired output (Gu et al., 2024).

In open-ended research and research evaluation, the executed object may be an experiment or an entire repository. “Towards Execution-Grounded Automated AI Research” builds an automated executor with Implementer, Scheduler, and Worker components, launches parallel GPU experiments, and uses measured performance as the feedback signal for evolutionary search or reinforcement learning (Si et al., 20 Jan 2026). “The Story is Not the Science” packages research outputs as narrative plus execution resources, then evaluates Coherence, Reproducibility, and Generalizability by executing code blocks, rerunning experiments, and testing on up to 3 new models or data (Bai et al., 5 Feb 2026).

Database-interface work shows that execution grounding can target infrastructure rather than model generations alone. EGRefine frames schema refinement as an optimization over admissible renaming functions, verifies candidate names through downstream Text-to-SQL execution accuracy, and materializes accepted changes as non-destructive SQL views so that read-only query equivalence is preserved (Wang et al., 1 May 2026).

5. Empirical regularities

A consistent empirical finding is that execution-grounded evaluation exposes failure modes that end-to-end outcome checks, judge-based review, or static metrics understate. AgentEval reports Failure Detection Recall of .89 compared with .41 for end-to-end evaluation, Human Agreement $\rightarrow$4, and Root Cause Accuracy of .72 against an $\rightarrow$5 human ceiling; its ablation shows that removing the DAG costs $\rightarrow$6 percentage points of FDRec and $\rightarrow$7 percentage points of RCA (Guo et al., 26 Apr 2026). GroundEval presents a case in which two frontier LLM judges scored a plausible response above 0.85, while the trajectory had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000 (Flynt, 22 Jun 2026). EVM-QuestBench reports persistent asymmetry between single-action precision and multi-step workflow completion, indicating that multi-step execution remains a separate bottleneck even when single-action precision is strong (Yang et al., 10 Jan 2026).

A second regularity is that execution grounding often reveals materially worse safety than language-only defenses suggest. In FinVault, average attack success rates still reach up to $\rightarrow$8 on state-of-the-art models and remain $\rightarrow$9 even for the most robust systems; insurance workflows see up to $\rightarrow$0 ASR, and no defense achieves both high TPR $\rightarrow$1 and low FPR $\rightarrow$2 (Yang et al., 9 Jan 2026). The framework also finds that semantic attacks such as role-playing and hypothetical framing outperform low-level obfuscation, which suggests that rule-based or regex defenses will struggle (Yang et al., 9 Jan 2026).

A third regularity is that execution feedback can improve synthesis or search when it is used inside the generation loop rather than only after the fact. AgentForge achieves $\rightarrow$3 resolution on SWE-BENCH Lite, compared with $\rightarrow$4 for a single-agent GPT-4o baseline, $\rightarrow$5 for ReAct, and $\rightarrow$6 for SWE-agent; its ablations show that execution feedback and role decomposition each independently drive performance (Kumar et al., 13 Apr 2026). DOCE reports that on HumanEval+ with CodeLlama-7B, filtering plus MBR-Exec reaches $\rightarrow$7, while SD-Multi raises this to $\rightarrow$8; filtering alone gives $\rightarrow$9, and MBR-Exec alone gives $\rightarrow$0 (Li et al., 2024). In automated AI research, execution-guided evolutionary search finds a method that significantly outperforms the GRPO baseline $\rightarrow$1 vs $\rightarrow$2 on post-training and a pre-training recipe that outperforms the nanoGPT baseline $\rightarrow$3 minutes vs $\rightarrow$4 minutes$\rightarrow$5 within ten search epochs, whereas reinforcement learning from execution reward suffers from mode collapse (Si et al., 20 Jan 2026).

Artifact-preserving evaluation also changes what is considered the dominant failure mode. In “Beyond Pass Rate,” the best open model reaches $\rightarrow$6 mean correctness against a $\rightarrow$7 human acceptance baseline, rankings are slice-dependent, and compile errors account for $\rightarrow$8 of non-accepted best submissions (Arefin, 7 Jun 2026). Static quality diverges from functional correctness: Gemma-2-27B-IT has the highest all-language lint pass rate, while Qwen2.5-Coder-14B-Instruct is second in mean correctness but much lower in all-language LPR (Arefin, 7 Jun 2026). In research evaluation, MechEvalAgent attains above $\rightarrow$9 agreement with human judges and surfaces 51 additional issues that human reviewers miss (Bai et al., 5 Feb 2026).

6. Limitations, methodological debates, and extensions

Execution-grounded evaluation does not eliminate methodological trade-offs. One debate concerns LLM-as-judge versus deterministic or environment-derived scoring. AgentEval relies on a calibrated LLM judge and reports strong agreement with humans, but GroundEval is explicitly introduced as a deterministic replacement for LLM-as-judge in stateful agent evaluation, targeting failures that final-answer and judge-based scoring struggle to detect (Guo et al., 26 Apr 2026, Flynt, 22 Jun 2026). A related issue is the status of attribution: AgentEval’s root-cause logic uses a greedy lowest-parent heuristic and is not formal causal inference (Guo et al., 26 Apr 2026).

A second limitation is environment fidelity. FinVault notes that sandbox $\rightarrow$0 production scale, that real systems have richer toolsets and deeper integrations, and that its ASR treats all violations equally rather than weighting by monetary or reputational impact (Yang et al., 9 Jan 2026). AgentEval handles non-DAG traces by loop-unrolling or timestamp-based branch resolution, but its details note that the DAG advantage wanes past $\rightarrow$1 non-DAG (Guo et al., 26 Apr 2026). MechEval identifies brittleness in binary checklists, occasional optimistic agent behavior, instruction-following gaps, and dependence on a single-model implementation (Bai et al., 5 Feb 2026).

A third limitation is observable scope. Many execution-grounded frameworks still emphasize pass/fail or compliance/no-compliance outcomes. “Towards Evaluation of Implicit Software World Models in Coding LLMs” broadens the observable axis to execution resources by asking models to predict test outcome, exception class, peak memory, wall-clock time, and ranked profiler outputs at method and line granularity; all tested models, frontier ones included, show modest performance and brittle behaviour (Bogomolov et al., 25 Jun 2026). This suggests that pass rate is only one slice of execution understanding. The multilingual code-evaluation study makes a related point from a different angle: a single aggregate pass rate obscures language coverage, difficulty stratification, failure modes, and static-analysis quality (Arefin, 7 Jun 2026).

Current extensions point toward more expressive operational semantics and stronger safety guarantees. TGEO models reasoning as an executable state-transition process with states, operators, predicates, contracts, and replayable execution graphs, and reports layer-wise auditability, replay success, and failure localization through an execution funnel (Anantharangachar, 14 Jun 2026). EGRefine demonstrates a different form of safety by construction: column-local non-degradation follows from its conservative selection rule, while database-level query equivalence is enforced by view-based materialization (Wang et al., 1 May 2026). In parallel, AgentEval proposes future work on cycle-aware graphs, automated taxonomy evolution, and closing the loop into agent training, while automated AI research emphasizes improving the implementer agent and adding richer trajectory signals or diversity bonuses to avoid RL mode collapse (Guo et al., 26 Apr 2026, Si et al., 20 Jan 2026).

Taken together, these developments indicate that the execution-grounded evaluation framework is not a single benchmark format but a methodological family. Its unifying property is that correctness, safety, and attribution are established through executed behavior, preserved traces, and environment-coupled validation rather than through narrative coherence or surface plausibility alone.