---
title: 'SolEval+: Benchmark for Solidity LLMs'
url: https://www.emergentmind.com/topics/soleval-benchmark
type: topic
---

# SolEval+: Benchmark for Solidity LLMs

The SolEval+ Benchmark is a comprehensive suite for evaluating the contract-level code generation capabilities of large language models (LLMs) in Solidity, focusing on realistic smart contract development with explicit functional, contextual, and security correctness criteria. It represents a major advancement beyond prior benchmarks, which predominantly address only function-level code generation or synthetic tasks, and is constructed on a foundation of real-world, on-chain contracts and rigorous dynamic validation. SolEval+ is referenced interchangeably with SolContractEval in the literature and is now the de facto standard for contract-level Solidity code generation benchmarking [2509.23824][2601.23009].

## 1. Motivation and Benchmark Evolution

Historically, evaluations of LLMs for Solidity code generation were limited to function-level or repository-level tasks, as exemplified by SolBench [2503.01098] and SolEval [2502.18793]. These earlier benchmarks neglected key phenomena in smart contract engineering, such as inter-function consistency, inheritance hierarchies, version-dependent syntax, and behaviors emergent only under realistic transaction sequences. Such gaps motivated the creation of SolEval+, the first contract-level benchmark. SolEval+ evaluates the capability to generate entire contracts reflective of real on-chain artifacts, integrates a realistic dependency context, and employs dynamic, transaction-driven evaluation—addressing the limitations of static or isolated function-level tests [2509.23824].

## 2. Benchmark Construction and Task Design

SolEval+ tasks are derived from verified mainnet contracts mined from Ajibola et al.’s Etherscan dataset. Contracts are filtered for ≥1,000 real on-chain transactions, deduplicated via SimHash, balanced for broad compiler version coverage (0.4.x–0.8.x), and restricted to contracts with substantive implementation (excluding interfaces and trivial constructors). This rigorous curation yields 124 contract-level tasks spanning nine critical application domains: DeFi, GameFi, ERC20/721 token standards, DAOs, proxy patterns, oracles, marketplaces, and "other" categories [2509.23824].

Each task is structured for LLM input as follows:
- **Dependency context**: A minimal but complete set of external definitions—imports, libraries, inherited contracts—extracted via a static call-graph analysis. Algorithm 1 formalizes this process, using tools such as Slither to enumerate and resolve all call dependencies.
- **Structured contract framework**: Includes license specifiers, pragma directives, original imports, state variable declarations, constructors, and full function signatures (retaining NatSpec documentation). Only function bodies are omitted for generation.
- **Natural-language prompt**: A concise, NatSpec-format textual description of each function's intended behavior, coupled with the required Solidity compiler version. All NatSpec annotations are independently authored and cross-validated by experienced developers to prevent leakage and ensure semantic accuracy.

## 3. Dynamic Multidimensional Evaluation Framework

SolEval+ employs dynamic, on-chain–inspired functional correctness evaluation. For each target contract, the first 1,000 transactions (with opcode traces, storage slot changes, event logs, and full call context) are replayed deterministically on a local Hardhat network using the same compiler version as the original deployment. Success criteria require that generated contracts:
- Produce identical execution status and return values for every transaction (byte-for-byte equivalence).
- Emit matching events in both signature and payload, including indexed and unindexed event fields.
- Maintain storage slot state evolution sequences that are SHA3-256 hash-identical to those of the original on-chain contract.

Key metrics are formalized as follows:
- **Pass@k**: The probability that at least one of $k$ generations passes both compilation and dynamic simulation. The unbiased estimator is $ \operatorname{Pass@}k = \mathbb{E}_i\left[1 - \frac{\binom{n-c_i}{k}}{\binom{n}{k}}\right] $, with $n$ samples and $c_i$ successful generations per task.
- **Compile@k**: As above, but counting compilable outputs only.
- **Function Pass Rate (FPR)**: The fraction of isolated functions passing their dynamic tests.
- **Contract Full Pass Rate (CFPR)**: The proportion of contracts for which all functions pass dynamic evaluation.

## 4. Experimental Protocols and Model Baselines

Six mainstream LLMs from Anthropic, OpenAI, Google DeepMind, DeepSeek-AI, and Alibaba are evaluated using a uniform pipeline:
- **Generation**: For each task, $n=5$ candidates are produced either by greedy decoding ($k=1$) or nucleus sampling ($k>1$, temperature=0.2, top_p=0.95).
- **Prompt composition**: Prompt includes full extracted context, contract scaffold, and cross-validated NatSpec descriptions.
- **Test environment**: All dynamic evaluations are performed on Ubuntu 20.04.5 with 128-core Xeon Platinum CPUs, 8×NVIDIA A800 GPUs, and a Hardhat network.
- **Compiler fidelity**: The Solidity compiler version matches the contract’s original pragma [2509.23824].

A selection of recent studies also evaluate agent-based frameworks, such as SolAgent, on a reduced SolEval+ benchmark (81 files, 1,125 functions, 1,188 tests). SolAgent utilizes a dual-loop refinement protocol, alternating between Forge-based compilation/testing and Slither-driven static security analysis. Comparative baselines include plain LLMs, AI IDE agents (Copilot, MetaGPT, DeepCode), and ablations of tool feedback [2601.23009].

## 5. Results, Analysis, and Challenges

SolEval+ reveals a marked performance gap between contract-level Solidity generation and general-purpose code generation:
- **LLM performance**: The best LLM (Claude-3.7-Sonnet) achieves Pass@1 of 40.7% and Compile@1 of 85.5% across 124 tasks. Even high-capacity models underperform compared to their class-level results on generic programming tasks (e.g., GPT-3.5: ClassEval Pass@1 ≈29.6% vs. SolEval+ Pass@1 ≈12.1%) [2509.23824].
- **Domain variance**: Models excel on standardized ERC20 contracts (reflecting pattern familiarity) but suffer on complex ERC721, DeFi, and contracts with intricate ownership logic or multi-level dependencies.
- **Primary error categories**: A significant proportion of failures arise from version-specific syntax (25.6% ParserErrors), the storage/memory distinction (12.4% TypeErrors), declaration resolution across inheritance hierarchies (21.7% DeclarationErrors), and inter-function consistency lapses. Notably, function-level FPRs (~95%) far exceed contract-level Pass@1 (~41%), highlighting the challenge of integrated contract behaviors under realistic state evolution.
- **Agent-based advances**: SolAgent (with Claude-Sonnet-4.5), using refinement guided by full Forge unit testing and Slither feedback, achieves a Pass@1 of 64.39% on the 81-file subset, with up to a 39.8% reduction in detected vulnerabilities versus the human-written baseline [2601.23009].

| Model/Framework         | Pass@1 (%) | Compile Rate | Vulnerability Reduction |
|------------------------|-----------:|-------------:|-----------------------:|
| Claude-3.7-Sonnet      |      40.7  |      85.5    | —                     |
| SolAgent + Claude      |      64.4  |      95.1    | −15.7%                |
| Copilot (Claude)       |      10.0  |      32.1    | N/A                   |
| Human-written baseline |     100.0  |     100.0    | —                     |

## 6. Limitations, Extensions, and Future Directions

SolEval+ has established multidimensional contract-level benchmarks with dynamic replay as a rigorous correctness criterion and highlighted persistent challenges for both model and evaluation methodology:
- **Version and context sensitivity**: Models struggle with the evolution of Solidity syntax and require precise context extraction to minimize undeclared identifier and type errors.
- **Inter-contract and cross-function dependencies**: The leap from function- to contract-level tasks introduces exponential complexity via inheritance, external contracts, and storage side effects.
- **Prospective enhancements** [2502.18793, 2509.23824]:
  - Include formal verification (SMT-based checks with tools like VerX, Certora Prover), runtime execution cost (dynamic gas and latency profiling), and more granular code complexity measurements.
  - Broaden linguistic and domain coverage (Vyper code, cross-chain patterns, multi-ecosystem tasks).
  - Integrate security adversary tests and automated continuous integration pipelines for evolving benchmarks.

*This suggests that LLM benchmarking for smart contract generation must evolve continuously to capture the complexity and security criticality of modern decentralized application development.*

## 7. Reproducibility and Community Contributions

SolEval+ supports detailed reproduction steps for the research community: exact source selection and annotation rules, explicit context extraction algorithms, hardware setup, prompt templates, and CI-ready test harnesses are provided. Extensions and new task contributions follow strictly defined filtering and validation protocols. Full benchmark code and agent frameworks such as SolAgent are released for end-to-end automated evaluation and model distillation, fostering open research in secure, correct Solidity code generation [2502.18793][2509.23824][2601.23009].

Source: https://www.emergentmind.com/topics/soleval-benchmark