---
title: 'SolContractEval: Solidity Code Generation Benchmark'
url: https://www.emergentmind.com/topics/solcontracteval
type: topic
---

# SolContractEval: Solidity Code Generation Benchmark

Searching arXiv for the benchmark and closely related Solidity evaluation work.
SolContractEval is a benchmark for evaluating **contract-level Solidity code generation** by large language models. It was introduced to address a specific gap in prior evaluation practice: existing Solidity benchmarks had focused on isolated functions, synthetic inputs, or incomplete context, whereas real smart contract development requires whole-contract reasoning under version-sensitive syntax, strict state semantics, inter-contract dependencies, and on-chain behavioral constraints. The benchmark comprises **124 generation tasks** drawn from **real on-chain contracts across nine major domains**, and it couples context-rich generation inputs with an automated **dynamic evaluation framework based on historical transaction replay** to assess functional correctness under realistic execution conditions [2509.23824].

## 1. Concept and motivation

SolContractEval was proposed in response to the mismatch between the apparent success of LLMs on general-purpose code generation and the particular demands of Solidity. The benchmark paper states that Solidity constitutes only **0.01%** of datasets like **The Stack v2**, and emphasizes its **version-sensitive syntax**, **gas management**, **strict state layering (storage vs memory)**, and **immutable logic** as reasons to question whether existing models can be relied upon for real-world smart contract generation [2509.23824].

A central premise of SolContractEval is that contract generation cannot be meaningfully approximated by isolated function completion. The benchmark therefore targets full target-contract generation with explicit dependencies and scaffolding. This positioning is consistent with the broader evolution of Solidity benchmarking. "SolEval" [2502.18793] introduced the **first repository-level benchmark designed for Solidity smart contract generation**, while "SolBench" [2503.01098] focused on **functional correctness in Solidity code completion and repair** at the function level. SolContractEval extends this progression by centering evaluation on complete contracts and replayed on-chain behavior rather than only repository-local tests or differential fuzzing [2509.23824].

This suggests that SolContractEval is best understood not merely as a dataset, but as a shift in evaluation granularity: from function bodies and repository snippets toward full-contract synthesis under realistic deployment constraints.

## 2. Benchmark design and task construction

SolContractEval contains **124 Generation Tasks**, each requiring a model to generate **a full, functional target contract with context, not just isolated functions**. All tasks are based on contracts deployed on the **Ethereum mainnet**. Each task input includes three elements: **Context Information**, a **Target Contract Framework**, and a **Natural Language Prompt**. The context includes all necessary surrounding code; the framework provides the contract name, state variables, constructors, and API signatures with omitted function bodies; and the prompt specifies requirements, contract name, and compiler version [2509.23824].

The construction process is explicitly described. It includes **Source File Selection** from real contracts with **≥1,000 txs**, **Target Identification** of one non-trivial non-interface non-abstract contract per file, **Context Extraction** using tools like **Slither**, **Human Annotation** with dual expert-written comments in strict **NatSpec** format, and **Test Input Generation** based on real transaction data [2509.23824].

The benchmark’s published statistics are as follows:

| Category | Statistics |
|---|---|
| **ERC20** | **52** tasks, **11.23** functions, **1206.79** tokens, **4.33** context contracts, **1074.04** context tokens, **1000** txs |
| **DeFi** | **28** tasks, **9.75** functions, **1117.82** tokens, **4.75** context contracts, **702.50** context tokens, **1000** txs |
| **ERC721** | **23** tasks, **17.26** functions, **1628.35** tokens, **16.22** context contracts, **4841.30** context tokens, **1000** txs |
| **Others** | **21** tasks, **8.00** functions, **1124.38** tokens, **6.19** context contracts, **1549.33** context tokens, **1000** txs |
| **Overall** | **124** tasks, **11.47** functions, **1250.93** tokens, **6.94** context contracts, **1769.40** context tokens, **1000** txs |

The task set spans **nine significant contract application domains**: **ERC20 Tokens**, **DeFi**, **ERC721 Tokens**, **GameFi**, **DAOs**, **Proxy Contracts**, **NFT**, **Lottery/Gambling Contracts**, and **Others** [2509.23824]. The concentration of tasks in ERC20, DeFi, and ERC721 makes those categories especially important for interpreting the benchmark’s empirical outcomes.

## 3. Dynamic evaluation by historical transaction replay

The most distinctive methodological component of SolContractEval is its **automated dynamic validation framework** based on **historical transaction replay**. For each contract, the benchmark collects the **first 1,000 on-chain transactions**, including execution context such as **msg.sender** and **block params**, and replays them on both the original and the model-generated contracts in a **Hardhat local network with full state reset before replay** [2509.23824].

Functional correctness is evaluated through three equivalence checks on every replayed transaction:

1. **Execution Status/Output**: matching success/fail status and return values.  
2. **Event Logs**: signature/parameter/value matching for emitted events.  
3. **Storage State Evolution**: **SHA3-256**-based checking for key variables after each transaction, with the complete state transition chain required to match the original contract [2509.23824].

The benchmark uses **Compile@k** and **Pass@k** as its primary metrics. **Compile@k** measures the fraction of tasks for which at least one of the top-\(k\) samples compiles. **Pass@k** measures the fraction of tasks for which at least one of the top-\(k\) samples compiles **and** passes full replay-based functional validation. The paper adopts the HumanEval-style unbiased estimator:

$$
\text{Pass@}k = \mathbb{E} \left[ 1 - \frac{ \binom{n-c}{k} }{ \binom{n}{k} } \right]
$$

where \(n\) is the number of samples per task, \(c\) is the number of successful samples, and \(k\) is the threshold [2509.23824].

The paper also introduces **Function Pass Rate (FPR)** and **Contract Full Pass Rate (CFPR)** to compare function-level and contract-level settings:

- \( \mathrm{FPR}_c = \frac{P}{M} \): compilable functions / total functions  
- \( \mathrm{FPR}_p = \frac{P}{M} \): passed functions / total functions  
- \( \mathrm{CFPR}_c = \frac{K}{N} \): contracts where all generated functions compile / total contracts  
- \( \mathrm{CFPR}_p = \frac{K}{N} \): contracts where all generated functions pass tests / total contracts [2509.23824]

A plausible implication is that SolContractEval’s evaluation protocol targets semantic equivalence at the contract execution level rather than only syntactic validity or unit-test satisfaction.

## 4. Empirical findings on LLM performance

SolContractEval reports a systematic evaluation of **six state-of-the-art LLMs**: **GPT-4o**, **o4-mini**, **Claude-3-7-Sonnet**, **Gemini-2.0-flash-exp**, **DeepSeek-R1**, and **Qwen2.5-Coder-32b-instruct** [2509.23824]. The headline result is that **Claude-3-7-Sonnet** achieves the highest overall performance.

The benchmark’s reported overall results are:

| Model | Compile@1 / Compile@5 | Pass@1 / Pass@5 |
|---|---|---|
| **claude-3-7-sonnet** | **85.48% / 96.77%** | **40.65% / 49.19%** |
| **o4-mini** | **81.45% / 95.97%** | **35.00% / 44.35%** |
| **gpt-4o** | **68.54% / 94.35%** | **33.39% / 45.16%** |
| **deepseek-R1** | **67.74% / 86.29%** | **29.84% / 40.32%** |
| **qwen2.5-coder-32b** | **68.54% / 84.67%** | **25.97% / 35.48%** |
| **gemini-2.0-flash-exp** | **63.55% / 72.03%** | **29.03% / 33.87%** |

Several conclusions follow directly from these results. First, **compile rates are significantly higher than functional correctness**, indicating that a substantial fraction of generated contracts are syntactically acceptable but semantically incorrect under replay-based execution [2509.23824]. Second, the paper finds that current models **perform better on tasks that follow standard patterns** but **struggle with complex logic and inter-contract dependencies**, and that they show **limited understanding of Solidity-specific features and contextual dependencies** [2509.23824].

The paper further reports that **function-level Pass rates (FPR\(_p\))** can reach **up to ~80%**, while **contract-level Pass@1** reaches only **~40% at best**. It gives the example of **Claude-3-7-Sonnet FPR\(_p\): 80.42% vs. contract-level Pass@1: 40.65%** [2509.23824]. This discrepancy is central to the benchmark’s argument: inter-function consistency and whole-contract context create failure modes that are largely invisible in function-level evaluations.

## 5. Category effects and characteristic failure modes

SolContractEval identifies clear domain-level variation. **ERC20** is the category with the highest performance; both compilation and functional correctness are reported as above the overall average. **ERC721** often yields high compilation rates but the **lowest functional correctness**, which the paper attributes to **complex, less standardized logic and heavy dependencies**, including dependencies on **ERC165** and **IERC721Metadata**. **DeFi** shows a slight dip in compilation due to complex syntax and operations, while functional correctness remains near the benchmark average [2509.23824].

The benchmark also provides a concrete breakdown of error types. Among compilation failures:

- **ParserError (37.21%)** is the most common category.  
- **TypeError (36.43%)** is the second-largest.  
- **DeclarationError (21.71%)** accounts for a substantial share as well [2509.23824].

The paper states that **49.61% of compile errors trace to Solidity-specific features** such as versioning and types, and **21.71% result from poor context/contextual dependency handling** [2509.23824]. On replay-based testing failures, **45.18%** are **return value mismatches**, **43.05%** are **storage state mismatches**, and **11.86%** are **event log mismatches** [2509.23824].

Qualitative examples further emphasize two recurring patterns. One is **implicit condition miss**: models fail to implement constraints described in comments, such as adjusting token decimals as implied by a docstring. The other is **context misunderstanding**: models omit or misread state flags and guards, especially in access-control and one-time-action logic [2509.23824].

A common misconception in Solidity code generation is that successful compilation is already a strong proxy for correctness. SolContractEval directly contradicts this view, since the reported **Compile@1** values are substantially higher than the corresponding **Pass@1** values across all evaluated models [2509.23824].

## 6. Position within the Solidity benchmark ecosystem

SolContractEval belongs to a rapidly developing family of Solidity benchmarks, but its scope and evaluation target are distinct. The following comparison summarizes the benchmark landscape described in the provided literature.

| Benchmark | Scope | Primary evaluation |
|---|---|---|
| **SolBench** [2503.01098] | **4,178 functions** from **1,155** deployed contracts | **pass@1** via differential fuzzing with **Diffusc** and **Echidna** |
| **SolEval** [2502.18793] | **1,125 samples** from **9 repositories** | **Pass@k**, **Compile@k**, **Gas Fee**, **Vulnerability Rate** |
| **SolContractEval** [2509.23824] | **124 contract-level tasks** from real on-chain contracts | **Compile@k**, **Pass@k**, historical transaction replay |

"SolEval" [2502.18793] had already shown that repository-level Solidity code generation remained difficult, with the **best-performing LLM** reaching only **26.29% Pass@10**. "SolBench" [2503.01098] showed that Solidity completion is highly context dependent and that retrieval-augmented repair can substantially improve function-level correctness. "SolAgent" [2601.23009] then demonstrated that tool-augmented multi-agent generation can push file-level **Pass@1** to **64.39%** on **SolEval+**, using a **dual-loop refinement mechanism** with **Forge** for functional correctness and **Slither** for security. Together, these results place SolContractEval in a broader methodological trajectory: increasingly realistic tasks, increasingly execution-grounded evaluation, and growing emphasis on the distinction between compilability, functional correctness, and security [2502.18793; 2503.01098; 2601.23009].

This suggests that SolContractEval’s contribution is not only the introduction of a new benchmark, but also the consolidation of a stricter notion of evaluation in Solidity generation: complete context, full-contract synthesis, and replay-based semantic validation.

## 7. Significance, interpretation, and research implications

SolContractEval establishes that **contract-level Solidity code generation is demonstrably harder** for current LLMs than analogous class-level generation tasks in mainstream languages. The paper explicitly contrasts **GPT-3.5 on ClassEval (Python, class-level): Pass@1 ≈ 30%** with **GPT-3.5 on SolContractEval (Solidity, contract-level): Pass@1 ≈ 12%** [2509.23824]. It also concludes that success on Python or Java generation does not transfer directly to Solidity, particularly at contract scope.

The benchmark’s findings align with adjacent research stressing that Solidity evaluation must go beyond lexical or shallow behavioral criteria. "SmartEval" [2605.09610] uses a five-dimensional rubric for smart contracts generated from natural language specifications, while "SCDBench" [2605.29059] evaluates decompilation outputs through **format completeness**, **compilability**, **ABI recovery**, and **semantic consistency via differential replay**. Although these works target different tasks, they reinforce the same broader principle: source-like output quality and even compilability are insufficient without stronger semantic checks [2605.09610; 2605.29059].

For practitioners, SolContractEval supports a cautious interpretation of current model capabilities. The benchmark paper’s practical conclusion is that models handle **standardized contracts** such as ERC20 better than complex categories, but remain unreliable on **complex logic**, **inter-contract dependencies**, and **Solidity-specific contextual requirements** [2509.23824]. For researchers, it provides a reproducible basis for studying compiler-version sensitivity, context extraction, contract-level reasoning, and dynamic validation by historical replay.

A plausible implication is that future progress on Solidity generation will likely depend on combining stronger context modeling with execution-aware or tool-augmented generation loops, rather than relying on single-pass text generation alone.

Source: https://www.emergentmind.com/topics/solcontracteval