ContractEval: Evaluating Contract Fidelity

Updated 8 July 2026

ContractEval is a family of evaluation regimes that verify contractual fidelity by aligning legal clauses with corresponding code representations.
Its applications include clause-level legal risk identification, e-contract to smart contract semantic validation, and end-to-end LLM synthesis scoring.
These frameworks employ metrics such as F1 scores, knowledge graph matching, and rubric-based audits to drive practical improvements in both legal and code accuracy.

ContractEval denotes a cluster of evaluation and validation approaches concerned with whether contractual intent is preserved across representations. In current literature, the name is used explicitly for a benchmark on clause-level legal risk identification in commercial contracts, and more broadly for systems that validate semantic alignment between e-contracts and smart contracts or score Solidity synthesized from natural-language requirements. Across these formulations, the central object is not merely syntactic well-formedness, but fidelity to clauses, obligations, permissions, temporal conditions, and executable behavior (Liu et al., 5 Aug 2025, Godboley et al., 27 Apr 2025, Goel et al., 14 Feb 2026).

1. Terminological scope and principal formulations

The literature supports three recurrent uses of “ContractEval”. One is a named benchmark for extracting legally relevant clauses from commercial agreements. A second is a semantic-validation framework that compares an e-contract and its intended Solidity implementation through knowledge graphs. A third is an evaluation pipeline for LLM-generated smart contracts that starts from natural-language requirements and ends with scored Solidity artifacts, security audits, and paired comparison against ground-truth contracts (Liu et al., 5 Aug 2025, Godboley et al., 27 Apr 2025, Goel et al., 14 Feb 2026).

Formulation	Object of evaluation	Core mechanism
Legal-risk ContractEval	Clause-level risk identification in commercial contracts	Question-conditioned span extraction over CUAD
E-contract / smart-contract ContractEval	Semantic alignment between legal text and Solidity	Knowledge-graph construction and discrepancy detection
Agentic smart-contract ContractEval	Quality of generated Solidity from NL specifications	Multi-agent synthesis, auditing, and rubric-based scoring

This terminological plurality implies that ContractEval is not a single canonical formalism. A more accurate characterization is a family of contract-centered evaluation regimes organized around alignment: clause-to-span alignment in legal review, clause-to-code alignment in smart-contract validation, and requirement-to-implementation alignment in LLM synthesis pipelines. This suggests that the unifying abstraction is contractual fidelity under representation change rather than any single dataset or metric.

2. Clause-level legal risk identification in commercial contracts

The benchmark explicitly titled “ContractEval” formulates contract review as clause-level legal risk identification over the CUAD test set. It uses 4,128 (contract, question) pairs drawn from 102 unique contracts and 41 clause categories, with about 30% positive and 70% negative labels. For each contract and legal-risk category, a model receives the full contract text as context and a natural-language question, and must return exact sentence spans from the contract or the exact string "No related clause" (Liu et al., 5 Aug 2025).

Its evaluation decomposes into three axes. Correctness is measured by $F_1$ and $F_2$ , where a true positive requires that the model’s prediction fully covers the labeled span; false negatives include both "No related clause" outputs on positive instances and extractions that fail to fully cover the label span. Output effectiveness is measured by token-level Jaccard similarity on positive instances,

$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$

and over-conservative retrieval behavior is isolated through the false “no related clause” rate on positive examples (Liu et al., 5 Aug 2025).

The benchmark evaluates 19 LLMs, comprising 4 proprietary and 15 open-source models. Proprietary systems lead overall correctness: GPT-4.1 reports $F_1=0.641$ , $F_2=0.672$ , $J=0.472$ , and false no-clause rate $=0.071$ ; GPT-4.1-mini reports $F_1=0.644$ , $F_2=0.678$ , $J=0.435$ , and false no-clause rate $F_2$ 0. Among open-source systems, Qwen3 8B (thinking) is the strongest reported configuration with $F_2$ 1, $F_2$ 2, $F_2$ 3, and false no-clause rate $F_2$ 4, while Gemma 3 12B is notable for $F_2$ 5 despite markedly lower correctness scores (Liu et al., 5 Aug 2025).

The benchmark’s five headline findings are structurally important. Proprietary models outperform open-source models in both correctness and output effectiveness. Larger open-source models generally improve, but the gains slow as scale increases. Reasoning mode improves output effectiveness but can reduce correctness. Open-source models produce "No related clause" more often even when relevant clauses exist, which the authors interpret as possible “laziness” or low confidence. Quantization improves efficiency at a performance cost, and in Qwen3 8B variants the degradation is especially sharp in thinking mode (Liu et al., 5 Aug 2025).

This formulation of ContractEval is narrowly focused but methodologically precise. It does not score legal reasoning in the abstract; it scores whether a model can localize risk-relevant textual evidence under long-context conditions. Its strongest categories are relatively straightforward fields such as Governing Law, Parties, Document Name, and Effective Date, whereas rare and nuanced categories such as Uncapped Liability, Joint IP Ownership, and Notice Period to Terminate Renewal remain difficult. The benchmark therefore operationalizes legal risk review as span-conditioned evidence recovery rather than free-form doctrinal analysis.

3. Semantic validation between e-contracts and smart contracts

A distinct ContractEval formulation appears in the validation framework for e-contracts and smart contracts. Its central problem is: given a human-readable e-contract and a Solidity smart contract intended to implement it, how can one systematically validate that the code captures all material terms, duties, and conditions of the original agreement? The stated goal is not bug-freedom in the narrow software-security sense, but semantic alignment between the legal agreement and its executable counterpart (Godboley et al., 27 Apr 2025).

The framework uses knowledge graphs as the common semantic layer. It constructs an e-contract graph

$F_2$ 6

whose nodes represent parties, obligations, dates, amounts, objects, and roles extracted from natural language, and a smart-contract graph

$F_2$ 7

whose nodes represent variables, functions, events, and semantic roles extracted from Solidity via compiler-generated ASTs and a custom grammar. Compliance is treated as a coverage relation: the smart contract is a correct implementation if $F_2$ 8 covers the entities and relations of $F_2$ 9, modulo naming and representation differences. The comparison stage computes matched entities and relations,

$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 0

and discrepancy sets

$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 1

The validation report $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 2 is the core output (Godboley et al., 27 Apr 2025).

Architecturally, the system is a four-stage pipeline: e-contract processing and knowledge-graph construction; smart-contract analysis and knowledge-graph construction; graph comparison; and reporting and visualization. E-contract processing consists of preprocessing, entity extraction, and relation extraction from natural language, using NLP and dependency parsing. Smart-contract processing extracts the Solidity version, selects the compiler, generates an AST, maps AST node types to semantic phrases through ApplyGrammar, and then extracts entities and relations from the resulting semantic structure. The implementation serializes both graphs as JSON, uses NetworkX and matplotlib for visualization, and produces a result view showing unmatched or missing entities and relations (Godboley et al., 27 Apr 2025).

The rental-agreement case study illustrates the intended semantics. The e-contract contains parties, property address, term dates, monthly rent of GBP 5000, security deposit of GBP 2000, utilities, property-use restrictions, maintenance duties, termination with one month written notice, and governing law. The corresponding Solidity contract includes state variables such as landlord, tenant, propertyAddress, rentAmount, securityDeposit, termStartDate, termEndDate, and terminated, along with functions payRent(), paySecurityDeposit(), terminateContract(), and getContractDetails(), plus events RentPaid, SecurityDepositPaid, and ContractTerminated. The comparison matches principal entities such as landlord, tenant, rent amount, deposit, and term dates, but identifies omissions for “monthly” periodicity, one-month notice, utilities, maintenance and repairs, use-of-property restrictions, and governing law. Those omissions are recorded as semantic gaps rather than compiler or security defects (Godboley et al., 27 Apr 2025).

Methodologically, this framework is notable for what it does not do. It does not introduce a full temporal logic or deontic logic semantics; indeed, the paper states that there is no heavy logical formalism like LTL or deontic logic. Obligations are instead approximated structurally as graph edges in the legal text and as functions plus require conditions and state changes on the code side. This yields a pragmatic ContractEval model: graph-based semantic compliance checking rather than theorem-proving over a unified legal-program logic.

4. End-to-end evaluation of natural-language-to-Solidity synthesis

A third major formulation treats ContractEval as an end-to-end evaluation system for LLM-generated smart contracts from natural-language specifications. The pipeline, orchestrated by IBMAgenticContractTranslator, proceeds through seven phases: Requirement Specification Agent, Solidity Generation Agent, Security Auditing Agent, Refiner loop, Quality Evaluator Agent, ABI Agent, and MCP Server Generator Agent. The first phase builds a UniversalContractSchema that structures parties, financial terms, dates, assets, obligations, conditions, and termination conditions, while preserving exact function names, variable names, and state names mentioned in the source specification (Goel et al., 14 Feb 2026).

The evaluation rubric is explicitly five-dimensional. $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 3 is Functional Completeness with weight 25%; $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 4 is Variable / Parameter Fidelity with weight 15%; $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 5 is State Machine Correctness with weight 15%; $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 6 is Business Logic Fidelity with weight 35%; and $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 7 is Code Quality with weight 10%. The composite score is

$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 8

For $J(A,B)=\frac{|A\cap B|}{|A\cup B|},$ 9, the paper gives an explicit scoring formula,

$F_1=0.641$ 0

where the fraction contributes 50 points and $F_1=0.641$ 1 covers implementation quality such as logic completeness, access control, events, and validation (Goel et al., 14 Feb 2026).

The system also includes an LLM-based static pseudo-auditor operating over eight vulnerability categories: reentrancy attacks, access control flaws, arithmetic safety issues, Ether handling vulnerabilities, denial-of-service risks, input validation gaps, timestamp dependence, and external call safety. Each issue receives a severity in none, low, medium, high, or critical. The refinement loop rewrites contracts when the audit report is not approved and the severity is medium or higher, with at most two refinement iterations (Goel et al., 14 Feb 2026).

Empirical evaluation uses the FSM-SCG dataset, which contains 21,976 entries, each with natural-language requirements, an FSM specification, and a ground-truth Solidity implementation. The experimental setup evaluates the pipeline on 9,000 contracts, processed in 6 batches of 1,500, using GPT-4o-mini for all agents, Solidity 0.8.x compilation, and up to two security refinements. The average composite score is 81.54, with average processing time 109.96 seconds per contract and total time 274.9 hours for the 9k run. Grade distribution is concentrated in the B range: 66.4% B-grade, 7.3% A-grade, 2.2% F-grade, and 1.5% D-grade. Metric averages are 84.45 for Functional Completeness, 84.62 for Variable Fidelity, 83.12 for State Machine Correctness, 76.73 for Business Logic Fidelity, and 83.85 for Code Quality (Goel et al., 14 Feb 2026).

Compilation and security numbers further illustrate the evaluation regime. Of 8,824 contracts checked for compilation, 7,637 compile successfully, 1,187 fail, 176 are not checked, and the success rate is 86.54%. The audit–refine loop reduces contracts with medium-or-higher severity issues from 4,127 to 1,203, average security issues per contract from 2.8 to 0.7, and critical vulnerabilities from 287 to 34, while improving compilation success rate from 81.2% to 86.5%. Performance degrades with complexity: low-complexity contracts score 87.2 with 94.1% compilation rate, medium-complexity 81.4 with 86.7%, and high-complexity 71.8 with 73.2% compilation rate (Goel et al., 14 Feb 2026).

This ContractEval formulation is therefore fundamentally rubric-based and artifact-rich. It evaluates not only whether code compiles or passes an isolated check, but whether generated Solidity preserves function names, variable fidelity, FSM structure, financial and temporal business logic, and security hygiene. Its paired evaluation against expert ground-truth contracts further converts ContractEval into a comparative benchmark rather than a one-sided validator.

5. Methodological extensions and adjacent contract-evaluation paradigms

Several adjacent systems broaden the design space that ContractEval inhabits. “SolContractEval” is a contract-level Solidity generation benchmark built from 124 tasks drawn from real Ethereum mainnet contracts across nine domains. Each task supplies complete context dependencies and a target contract framework, and evaluation is performed by replaying 1,000 historical transactions per task while comparing execution status, return values, emitted events, and storage-state evolution. Under this dynamic criterion, Claude-3.7-Sonnet achieves the highest reported overall performance with Compile@1 of 85.48% and Pass@1 of 40.65%, underscoring that contract-level Solidity generation remains substantially harder than function-level code completion (Ye et al., 28 Sep 2025).

PACT addresses a different but structurally related notion of contract adherence in general code generation. It extends HumanEval+ and MBPP+ with contract-violating test cases generated by an LLM-plus-SMT pipeline and introduces metrics such as Assert Violation Coverage, Target Specificity, Assertion Alignment Recall, and Assertion Alignment Precision. Its main empirical result is that Example-Augmented Specification, which includes explicit contract-violating test cases in the prompt, substantially improves runtime and static contract-adherence metrics relative to natural-language contract descriptions alone, albeit with slight pass@1 trade-offs (Lim et al., 14 Oct 2025).

DeCon contributes a runtime-monitoring interpretation of contract evaluation for smart contracts. It models a smart contract as relational tables of transaction records and derived views, expresses contract properties as violation query rules over those relations, and compiles the result to Solidity with instrumentation for runtime monitoring and provenance-based debugging. Across the evaluated contracts, DeCon incurs 14% median gas overhead relative to reference implementations and an additional 16% median gas overhead for runtime verification, showing that property-level ContractEval can be embedded directly into execution artifacts rather than treated purely as an offline benchmark (2207.13827).

Broader program-verification work extends the meaning of contract evaluation beyond legal documents and blockchains. ConVer uses LLM-synthesized function contracts and loop invariants inside a CEGAR-CEGIS loop for compositional verification of C programs, obtaining 82–96% success on 45 Frama-C benchmarks across three LLM backends and 67% overall success on LF-Hard benchmarks after LF-to-C preprocessing (Pirzada et al., 26 May 2026). Contract strengthening through constrained Horn clauses shows how valid but insufficient postconditions can be strengthened automatically using CHC transformation, SPACER models, and translation back to source-level contracts, enabling Stainless to verify a list-reversal example it cannot prove under the original contracts (Angelis et al., 2022). In gradual typing, “Corpse Reviver” treats typed–untyped boundary contracts as static specifications, verifies most of them with soft contract verification, and reduces worst-case overhead from 73.6× to 1.6× on a standard 12-program suite (Moy et al., 2020). Runtime verification in Erlang via EDBC adds executable preconditions, postconditions, decreasing-argument contracts, purity, timing, invariants, and cpre/3 request guards for concurrent servers, showing a host-language-centric contract-monitoring model (Fredlund et al., 2018).

Deeper antecedents contribute formal and representational foundations. Event-based contracts with circular enabling relations $F_1=0.641$ 2 and $F_1=0.641$ 3 define agreement, duties, and culpability over configurations, and establish a correspondence with a fragment of Propositional Contract Logic through the theorem that reachable events are exactly the provable atoms in the translated logic (Bartoletti et al., 2013). EROP represents contracts as Events, Rights, Obligations, and Prohibitions, compiles them to Augmented Drools for the Contract Compliance Checker, and thereby exemplifies a rule-engine view of machine-evaluable contractual obligations (Delchev et al., 2023). Certified compilation of financial contracts shows yet another axis: declarative financial contracts compiled in Coq to payoff expressions with proof that compiled evaluation equals discounted trace semantics, enabling certified pricing in Monte Carlo engines (Annenkov et al., 2021).

Taken together, these systems suggest that ContractEval is best understood as a methodological spectrum. Some instances evaluate evidence extraction from legal text, some validate legal-to-code alignment, some benchmark generated code under contract-aware rubrics or transaction replay, and others verify contracts as logical or executable artifacts. The commonality lies in treating contracts as semantically meaningful specifications whose preservation, violation, or adequacy can itself be evaluated.

6. Limitations, controversies, and future directions

Each major ContractEval formulation inherits a distinct limitation profile. The legal-risk benchmark is restricted to CUAD’s 41 clause categories and 102 test contracts, all from the CUAD test set, and the paper does not provide new quantitative human baselines or inter-annotator agreement figures. It also evaluates only off-the-shelf models with large context windows released around mid-2025 and uses a single prompt template. This makes its results highly informative for long-context clause extraction, but less general for multilingual, multi-jurisdiction, or richly structured risk analysis (Liu et al., 5 Aug 2025).

The knowledge-graph framework for e-contract and smart-contract alignment is intentionally lightweight in semantics. It depends on NLP quality for entity and relation extraction, sketches but does not fully specify MatchEntities and MatchRelations, and does not capture full temporal or deontic semantics such as “monthly”, “notice period”, or “upon termination, deposit returned minus damages” in a formal way. Its main worked example is a rental agreement, so scalability to complex commercial contracts remains open. The authors explicitly propose richer structured representations, including AST-like forms for e-contracts, as future work (Godboley et al., 27 Apr 2025).

The agentic Solidity-evaluation pipeline is strong on protocol and artifact structure but remains LLM-mediated at key points. The rubric weights and formulas are explicit, yet scoring of implementation quality and business-logic fidelity is still performed by an LLM. The audit stage is not Slither or Mythril, but an LLM-based pseudo-analyzer guided by their vulnerability taxonomy. The paper also notes literal-spec bias: models that hew closely to the given terminology and FSM can outperform human ground-truth implementations on the rubric even when the human implementation reflects sensible abstraction choices. Performance drops sharply on high-complexity contracts, and the dataset does not cover multi-contract systems, oracle-heavy protocols, or cross-chain behaviors (Goel et al., 14 Feb 2026).

Adjacent benchmarks reveal further open problems. SolContractEval shows that even top models remain below 50% Pass@5 on contract-level Solidity generation and are particularly brittle with version-sensitive syntax, inheritance, storage semantics, and DeFi logic (Ye et al., 28 Sep 2025). PACT shows that current models exhibit a contract–functionality trade-off: better enforcement of invalid-input behavior can slightly degrade functional pass@1 (Lim et al., 14 Oct 2025). DeCon’s gas measurements show that some global invariants, such as ERC20 balance-sum equality, are expensive to monitor online (2207.13827).

This suggests that future ContractEval systems will likely combine several presently separate ingredients: richer legal ontologies and deontic-temporal semantics for text–code alignment; hybrid static, symbolic, and dynamic checking for generated smart contracts; transaction replay plus formal verification for contract-level execution fidelity; and domain-specific compliance layers for settings such as securities tokens, DeFi KYC/AML, or real-estate jurisdictional rules. It also suggests that “contract evaluation” is converging toward a multi-layer stack in which retrieval quality, semantic representation quality, executable correctness, security posture, and regulatory conformance are evaluated jointly rather than in isolation.