EVM-QuestBench Evaluation Suite

• EVM-QuestBench is an execution-grounded benchmarking suite that assesses the generation of transaction scripts on EVM-compatible blockchains.
• It features a modular architecture with dynamic task instantiation, addressing both atomic and composite transaction workflows via runtime validations.
• The benchmark employs snapshot isolation and step-efficiency metrics to ensure execution accuracy, offering actionable insights for model safety in blockchain environments.

EVM-QuestBench is an execution-grounded benchmarking suite for the evaluation of natural-language-driven transaction-script generation specifically on EVM-compatible blockchain platforms. It addresses the critical limitations of prior assessments by enforcing execution accuracy and outcome validation, essential in domains where even minor errors can result in irreversible financial or operational losses. EVM-QuestBench provides a modular, scalable architecture for the dynamic assessment of LLMs and related systems in transaction code generation, distinguishing itself through its emphasis on runtime validation and workflow completion fidelity (Yang et al., 10 Jan 2026).

1. Motivation and Background

LLMs have been increasingly applied to automate development workflows, including the generation of transaction scripts for blockchain applications. On-chain transactions, however, present heightened challenges: correctness is paramount, as mistakes can lead to user asset loss or system misbehavior. Prior natural language to code benchmarks have typically relied on static code comparison or unit tests, failing to capture the stringent execution correctness and safety required in blockchain settings. EVM-QuestBench was introduced to fill this methodological gap, offering a benchmark that directly measures the transaction outcome within an EVM (Ethereum Virtual Machine) execution context, thereby providing rigorous assessment of both functional correctness and real-world viability (Yang et al., 10 Jan 2026).

2. Benchmark Design and Task Suite

EVM-QuestBench comprises 107 transaction-generation tasks, partitioned into 62 atomic (single-step) and 45 composite (multi-step) scripts. Each task is constructed from robust template pools, where instructions are instanced via sampling and numeric parameters are drawn from pre-specified intervals. This design enables dynamic task instantiation, enhancing generalizability and resistance to model memorization. Validators associated with each task type execute postcondition checks: after script deployment in the EVM-forked environment, the system verifies on-chain state changes, balances, and other task-specific outcomes against the instantiated ground-truth values. This dynamic template and value instantiation framework supports rapid benchmark expansion and fine-grained skill assessment, while the inclusion of composite tasks ensures that macro-level reasoning and workflow execution can be robustly evaluated (Yang et al., 10 Jan 2026).

3. Execution-Grounded Evaluation Methodology

The core of EVM-QuestBench's evaluation pipeline is execution-grounded assessment. Scripts generated by candidate systems are executed on a forked EVM chain using snapshot isolation. This ensures full state encapsulation and repeatability of evaluation. Each transaction script is executed in a controlled environment, and a suite of validators ascertain whether the final chain state matches expected outcomes. For composite tasks, a step-efficiency decay metric is applied, which penalizes incomplete execution and measures the ability of models to successfully perform multi-step workflows. This approach contrasts markedly with previous benchmarks that only checked for the presence of code patterns or superficial artifacts, enabling a high-fidelity assessment of both model safety and utility in the blockchain ecosystem (Yang et al., 10 Jan 2026).

4. Modular Architecture and Task Development

EVM-QuestBench is architected for extensibility via a modular task and validation system. New task types can be rapidly incorporated through the template/value pool mechanism, and validators can be customized per task to accommodate complex logical or state requirements. The execution runner orchestrates script deployment, execution, and validation within the EVM fork, minimizing overhead and ensuring the isolation of each benchmark interaction. This modularity optimizes the benchmark for rapid development in tandem with advances in model architectures and for the ongoing evolution of blockchain transaction primitives (Yang et al., 10 Jan 2026).

5. Model Evaluation and Metrics

Twenty distinct models were evaluated using EVM-QuestBench, revealing significant performance variability. Metric reporting includes both single-action (atomic) task precision and composite (multi-step) workflow completion rates, with split scoring to surface differences in localized versus sequential reasoning. Empirical findings demonstrate a persistent asymmetry: models often excel at atomic, single-action tasks but exhibit substantial degradation in multi-step transaction workflows. The runner architecture’s snapshot isolation and step-efficiency decay scoring allow for granular diagnosis of failure modes—including incomplete execution, transaction reversion, or silent logical error—thereby yielding a comprehensive understanding of model capabilities and safety in real EVM-deployed scenarios (Yang et al., 10 Jan 2026).

6. Practical Significance and Applications

The introduction of EVM-QuestBench establishes a new standard for assessing LLM competence in high-stakes transaction code generation. It enables practitioners and researchers to benchmark systems under dynamic, execution-grounded workloads and to quantify both their atomic action reliability and their robustness in orchestrating complex workflows. The modular, fork-based architecture ensures that assessments are valid across evolving task requirements and blockchain standards, supporting trustworthy advances in both LLM safety and transaction automation tooling (Yang et al., 10 Jan 2026).

7. Code Availability and Future Directions

The reference implementation for EVM-QuestBench is openly available (code repository), supporting reproducibility and extensibility by the research community. Future directions include the expansion of the composite task suite, increased coverage for novel transaction constructs, and refinement of scoring methodologies for even finer-grained failure classification. A plausible implication is that EVM-QuestBench may serve as the foundation for standardized evaluation in transaction-oriented code generation, influencing both the design of future models and the assessment criteria for on-chain automation technologies (Yang et al., 10 Jan 2026).