SolContractEval: Solidity Code Generation Benchmark

Updated 14 July 2026

SolContractEval is a benchmark that evaluates full-contract Solidity code synthesis using real on-chain tasks and contextual inputs.
It employs a dynamic evaluation framework based on historical transaction replay to verify compilation, event logs, and storage state evolution.
Empirical results show high compilation rates but lower functional correctness, highlighting challenges in handling complex inter-contract dependencies.

Searching arXiv for the benchmark and closely related Solidity evaluation work. SolContractEval is a benchmark for evaluating contract-level Solidity code generation by LLMs. It was introduced to address a specific gap in prior evaluation practice: existing Solidity benchmarks had focused on isolated functions, synthetic inputs, or incomplete context, whereas real smart contract development requires whole-contract reasoning under version-sensitive syntax, strict state semantics, inter-contract dependencies, and on-chain behavioral constraints. The benchmark comprises 124 generation tasks drawn from real on-chain contracts across nine major domains, and it couples context-rich generation inputs with an automated dynamic evaluation framework based on historical transaction replay to assess functional correctness under realistic execution conditions (Ye et al., 28 Sep 2025).

1. Concept and motivation

SolContractEval was proposed in response to the mismatch between the apparent success of LLMs on general-purpose code generation and the particular demands of Solidity. The benchmark paper states that Solidity constitutes only 0.01% of datasets like The Stack v2, and emphasizes its version-sensitive syntax, gas management, strict state layering (storage vs memory), and immutable logic as reasons to question whether existing models can be relied upon for real-world smart contract generation (Ye et al., 28 Sep 2025).

A central premise of SolContractEval is that contract generation cannot be meaningfully approximated by isolated function completion. The benchmark therefore targets full target-contract generation with explicit dependencies and scaffolding. This positioning is consistent with the broader evolution of Solidity benchmarking. "SolEval" (Peng et al., 26 Feb 2025) introduced the first repository-level benchmark designed for Solidity smart contract generation, while "SolBench" (Chen et al., 3 Mar 2025) focused on functional correctness in Solidity code completion and repair at the function level. SolContractEval extends this progression by centering evaluation on complete contracts and replayed on-chain behavior rather than only repository-local tests or differential fuzzing (Ye et al., 28 Sep 2025).

This suggests that SolContractEval is best understood not merely as a dataset, but as a shift in evaluation granularity: from function bodies and repository snippets toward full-contract synthesis under realistic deployment constraints.

2. Benchmark design and task construction

SolContractEval contains 124 Generation Tasks, each requiring a model to generate a full, functional target contract with context, not just isolated functions. All tasks are based on contracts deployed on the Ethereum mainnet. Each task input includes three elements: Context Information, a Target Contract Framework, and a Natural Language Prompt. The context includes all necessary surrounding code; the framework provides the contract name, state variables, constructors, and API signatures with omitted function bodies; and the prompt specifies requirements, contract name, and compiler version (Ye et al., 28 Sep 2025).

The construction process is explicitly described. It includes Source File Selection from real contracts with ≥1,000 txs, Target Identification of one non-trivial non-interface non-abstract contract per file, Context Extraction using tools like Slither, Human Annotation with dual expert-written comments in strict NatSpec format, and Test Input Generation based on real transaction data (Ye et al., 28 Sep 2025).

The benchmark’s published statistics are as follows:

Category	Statistics
ERC20	52 tasks, 11.23 functions, 1206.79 tokens, 4.33 context contracts, 1074.04 context tokens, 1000 txs
DeFi	28 tasks, 9.75 functions, 1117.82 tokens, 4.75 context contracts, 702.50 context tokens, 1000 txs
ERC721	23 tasks, 17.26 functions, 1628.35 tokens, 16.22 context contracts, 4841.30 context tokens, 1000 txs
Others	21 tasks, 8.00 functions, 1124.38 tokens, 6.19 context contracts, 1549.33 context tokens, 1000 txs
Overall	124 tasks, 11.47 functions, 1250.93 tokens, 6.94 context contracts, 1769.40 context tokens, 1000 txs

The task set spans nine significant contract application domains: ERC20 Tokens, DeFi, ERC721 Tokens, GameFi, DAOs, Proxy Contracts, NFT, Lottery/Gambling Contracts, and Others (Ye et al., 28 Sep 2025). The concentration of tasks in ERC20, DeFi, and ERC721 makes those categories especially important for interpreting the benchmark’s empirical outcomes.

3. Dynamic evaluation by historical transaction replay

The most distinctive methodological component of SolContractEval is its automated dynamic validation framework based on historical transaction replay. For each contract, the benchmark collects the first 1,000 on-chain transactions, including execution context such as msg.sender and block params, and replays them on both the original and the model-generated contracts in a Hardhat local network with full state reset before replay (Ye et al., 28 Sep 2025).

Functional correctness is evaluated through three equivalence checks on every replayed transaction:

Execution Status/Output: matching success/fail status and return values.
Event Logs: signature/parameter/value matching for emitted events.
Storage State Evolution: SHA3-256-based checking for key variables after each transaction, with the complete state transition chain required to match the original contract (Ye et al., 28 Sep 2025).

The benchmark uses Compile@k and Pass@k as its primary metrics. Compile@k measures the fraction of tasks for which at least one of the top- $k$ samples compiles. Pass@k measures the fraction of tasks for which at least one of the top- $k$ samples compiles and passes full replay-based functional validation. The paper adopts the HumanEval-style unbiased estimator:

$\text{Pass@}k = \mathbb{E} \left[ 1 - \frac{ \binom{n-c}{k} }{ \binom{n}{k} } \right]$

where $n$ is the number of samples per task, $c$ is the number of successful samples, and $k$ is the threshold (Ye et al., 28 Sep 2025).

The paper also introduces Function Pass Rate (FPR) and Contract Full Pass Rate (CFPR) to compare function-level and contract-level settings:

$\mathrm{FPR}_c = \frac{P}{M}$ : compilable functions / total functions
$\mathrm{FPR}_p = \frac{P}{M}$ : passed functions / total functions
$\mathrm{CFPR}_c = \frac{K}{N}$ : contracts where all generated functions compile / total contracts
$\mathrm{CFPR}_p = \frac{K}{N}$ : contracts where all generated functions pass tests / total contracts (Ye et al., 28 Sep 2025)

A plausible implication is that SolContractEval’s evaluation protocol targets semantic equivalence at the contract execution level rather than only syntactic validity or unit-test satisfaction.

4. Empirical findings on LLM performance

SolContractEval reports a systematic evaluation of six state-of-the-art LLMs: GPT-4o, o4-mini, Claude-3-7-Sonnet, Gemini-2.0-flash-exp, DeepSeek-R1, and Qwen2.5-Coder-32b-instruct (Ye et al., 28 Sep 2025). The headline result is that Claude-3-7-Sonnet achieves the highest overall performance.

The benchmark’s reported overall results are:

Model	Compile@1 / Compile@5	Pass@1 / Pass@5
claude-3-7-sonnet	85.48% / 96.77%	40.65% / 49.19%
o4-mini	81.45% / 95.97%	35.00% / 44.35%
gpt-4o	68.54% / 94.35%	33.39% / 45.16%
deepseek-R1	67.74% / 86.29%	29.84% / 40.32%
qwen2.5-coder-32b	68.54% / 84.67%	25.97% / 35.48%
gemini-2.0-flash-exp	63.55% / 72.03%	29.03% / 33.87%

Several conclusions follow directly from these results. First, compile rates are significantly higher than functional correctness, indicating that a substantial fraction of generated contracts are syntactically acceptable but semantically incorrect under replay-based execution (Ye et al., 28 Sep 2025). Second, the paper finds that current models perform better on tasks that follow standard patterns but struggle with complex logic and inter-contract dependencies, and that they show limited understanding of Solidity-specific features and contextual dependencies (Ye et al., 28 Sep 2025).

The paper further reports that function-level Pass rates (FPR $k$ 0) can reach up to ~80%, while contract-level Pass@1 reaches only ~40% at best. It gives the example of Claude-3-7-Sonnet FPR $k$ 1: 80.42% vs. contract-level Pass@1: 40.65% (Ye et al., 28 Sep 2025). This discrepancy is central to the benchmark’s argument: inter-function consistency and whole-contract context create failure modes that are largely invisible in function-level evaluations.

5. Category effects and characteristic failure modes

SolContractEval identifies clear domain-level variation. ERC20 is the category with the highest performance; both compilation and functional correctness are reported as above the overall average. ERC721 often yields high compilation rates but the lowest functional correctness, which the paper attributes to complex, less standardized logic and heavy dependencies, including dependencies on ERC165 and IERC721Metadata. DeFi shows a slight dip in compilation due to complex syntax and operations, while functional correctness remains near the benchmark average (Ye et al., 28 Sep 2025).

The benchmark also provides a concrete breakdown of error types. Among compilation failures:

ParserError (37.21%) is the most common category.
TypeError (36.43%) is the second-largest.
DeclarationError (21.71%) accounts for a substantial share as well (Ye et al., 28 Sep 2025).

The paper states that 49.61% of compile errors trace to Solidity-specific features such as versioning and types, and 21.71% result from poor context/contextual dependency handling (Ye et al., 28 Sep 2025). On replay-based testing failures, 45.18% are return value mismatches, 43.05% are storage state mismatches, and 11.86% are event log mismatches (Ye et al., 28 Sep 2025).

Qualitative examples further emphasize two recurring patterns. One is implicit condition miss: models fail to implement constraints described in comments, such as adjusting token decimals as implied by a docstring. The other is context misunderstanding: models omit or misread state flags and guards, especially in access-control and one-time-action logic (Ye et al., 28 Sep 2025).

A common misconception in Solidity code generation is that successful compilation is already a strong proxy for correctness. SolContractEval directly contradicts this view, since the reported Compile@1 values are substantially higher than the corresponding Pass@1 values across all evaluated models (Ye et al., 28 Sep 2025).

6. Position within the Solidity benchmark ecosystem

SolContractEval belongs to a rapidly developing family of Solidity benchmarks, but its scope and evaluation target are distinct. The following comparison summarizes the benchmark landscape described in the provided literature.

Benchmark	Scope	Primary evaluation
SolBench (Chen et al., 3 Mar 2025)	4,178 functions from 1,155 deployed contracts	pass@1 via differential fuzzing with Diffusc and Echidna
SolEval (Peng et al., 26 Feb 2025)	1,125 samples from 9 repositories	Pass@k, Compile@k, Gas Fee, Vulnerability Rate
SolContractEval (Ye et al., 28 Sep 2025)	124 contract-level tasks from real on-chain contracts	Compile@k, Pass@k, historical transaction replay

"SolEval" (Peng et al., 26 Feb 2025) had already shown that repository-level Solidity code generation remained difficult, with the best-performing LLM reaching only 26.29% Pass@10. "SolBench" (Chen et al., 3 Mar 2025) showed that Solidity completion is highly context dependent and that retrieval-augmented repair can substantially improve function-level correctness. "SolAgent" (Chen et al., 30 Jan 2026) then demonstrated that tool-augmented multi-agent generation can push file-level Pass@1 to 64.39% on SolEval+, using a dual-loop refinement mechanism with Forge for functional correctness and Slither for security. Together, these results place SolContractEval in a broader methodological trajectory: increasingly realistic tasks, increasingly execution-grounded evaluation, and growing emphasis on the distinction between compilability, functional correctness, and security (Peng et al., 26 Feb 2025, Chen et al., 3 Mar 2025, Chen et al., 30 Jan 2026).

This suggests that SolContractEval’s contribution is not only the introduction of a new benchmark, but also the consolidation of a stricter notion of evaluation in Solidity generation: complete context, full-contract synthesis, and replay-based semantic validation.

7. Significance, interpretation, and research implications

SolContractEval establishes that contract-level Solidity code generation is demonstrably harder for current LLMs than analogous class-level generation tasks in mainstream languages. The paper explicitly contrasts GPT-3.5 on ClassEval (Python, class-level): Pass@1 ≈ 30% with GPT-3.5 on SolContractEval (Solidity, contract-level): Pass@1 ≈ 12% (Ye et al., 28 Sep 2025). It also concludes that success on Python or Java generation does not transfer directly to Solidity, particularly at contract scope.

The benchmark’s findings align with adjacent research stressing that Solidity evaluation must go beyond lexical or shallow behavioral criteria. "SmartEval" (Goel et al., 10 May 2026) uses a five-dimensional rubric for smart contracts generated from natural language specifications, while "SCDBench" (Qin et al., 27 May 2026) evaluates decompilation outputs through format completeness, compilability, ABI recovery, and semantic consistency via differential replay. Although these works target different tasks, they reinforce the same broader principle: source-like output quality and even compilability are insufficient without stronger semantic checks (Goel et al., 10 May 2026, Qin et al., 27 May 2026).

For practitioners, SolContractEval supports a cautious interpretation of current model capabilities. The benchmark paper’s practical conclusion is that models handle standardized contracts such as ERC20 better than complex categories, but remain unreliable on complex logic, inter-contract dependencies, and Solidity-specific contextual requirements (Ye et al., 28 Sep 2025). For researchers, it provides a reproducible basis for studying compiler-version sensitivity, context extraction, contract-level reasoning, and dynamic validation by historical replay.

A plausible implication is that future progress on Solidity generation will likely depend on combining stronger context modeling with execution-aware or tool-augmented generation loops, rather than relying on single-pass text generation alone.