CUAD: Legal Contract Benchmark

• CUAD is an expert-annotated legal contract benchmark comprising 510 contracts with 13,101 spans over 41 categories for extractive clause review.
• Its evaluation protocol uses sliding-window chunking and precise span matching to address the challenges of long, complex legal documents.
• The dataset underpins research in long-document NLP, structure-aware segmentation, OCR integration, and prompt-based retrieval for legal analysis.

The Contract Understanding Atticus Dataset (CUAD) is an expert-annotated benchmark for legal contract review introduced by Hendrycks et al. It contains 510 English-language, public-domain commercial contracts and 13,101 annotated spans across 41 categories, and its core task is extractive “highlight” prediction: given a contract and a clause query, a model must identify the salient text spans that a human reviewer would need to inspect (Hendrycks et al., 2021). CUAD has become a reference corpus for studying long-document legal NLP because it combines high-value expert supervision with the practical difficulties of lengthy, structurally complex agreements; later work has used it not only for extractive review, but also for structure-aware segmentation, OCR-augmented document understanding, prompt-based retrieval, and LLM-driven metadata extraction (Hegel et al., 2021, Klem et al., 2 Sep 2025, Han et al., 22 Oct 2025).

1. Motivation and intended scope

CUAD was created to address a specific bottleneck in legal practice: manual contract review is a low-level but costly “needles-in-a-haystack” activity in which dozens or hundreds of pages are scanned to locate a small number of high-salience clauses (Hendrycks et al., 2021). The dataset’s stated aim is to automate this “contract analysis” task by locating provisions such as definitions, indemnification, governing law, anti-assignment, and uncapped liability, while also providing a large, expert-annotated benchmark for evaluating how well pretrained Transformer models transfer into a highly specialized legal domain (Hendrycks et al., 2021).

The dataset was built in collaboration with The Atticus Project, a non-profit of practicing lawyers and law students. Dozens of annotators underwent 70–100 hours of training, including video lectures, live workshops, quizzes, and use of a 100-page annotation manual crafted by experienced lawyers (Hendrycks et al., 2021). Every clause annotation was independently verified by three additional annotators and then reviewed by senior attorneys. The summary provided for the original dataset states that the estimated annotation cost exceeds \$2 million if valued at \$500 per hour (Hendrycks et al., 2021).

CUAD’s scope is narrower than general legal reasoning. It is designed around clause finding and clause highlighting rather than higher-level counseling tasks. The original authors explicitly present expansion toward tasks such as automatically summarizing risk or prescribing remedial actions as future work rather than as part of the benchmark itself (Hendrycks et al., 2021). A common misconception is therefore to treat CUAD as a full-spectrum legal-advice benchmark; the source material instead places it at the level of expert-supported extractive review.

2. Corpus composition and annotation schema

CUAD contains 510 unique contracts drawn from 25 different contract types, including categories such as License, Services, Distribution, Manufacturing, Franchise, and Sponsorship (Hendrycks et al., 2021). A later long-document study reports an average contract length of approximately 9,594 words, with a minimum of 109 and a maximum of 103,923 words (Hegel et al., 2021). The original dataset summary also states that contracts range from a few pages to over 100 pages, with median length around 20–30 pages (Hendrycks et al., 2021).

Item	Value
Contracts	510 unique contracts
Contract types	25
Annotations	13,101 spans across 41 categories
Average length	Approximately 9,594 words
Length range	109 to 103,923 words
Split	80% train / 20% test; approximately 10% of train held out as dev

The 41 label categories are organized into three broad groups. “General Information” includes fields such as Document Name, Parties, Agreement Date, Effective Date, Expiration Date, Renewal Term, Notice to Terminate Renewal, and Governing Law. “Restrictive Covenants” includes Non-Compete, Exclusivity, No-Solicit of Customers, No-Solicit of Employees, Non-Disparagement, Termination for Convenience, ROFR/ROFO/ROFN, Change of Control, and Anti-Assignment. “Revenue-Risk Provisions” includes Revenue/Profit Sharing, Price Restriction, Minimum Commitment, Volume Restriction, IP Ownership Assignment, Joint IP Ownership, License Grants and related subtypes, Source-Code Escrow, Post-Termination Services, Audit Rights, Uncapped Liability, Cap on Liability, Liquidated Damages, Warranty Duration, Insurance, Covenant Not to Sue, and Third-Party Beneficiary (Hendrycks et al., 2021).

The annotation workflow was carried out in eBrevia, an online contract-review tool. The process included student labeling, a keyword-driven recall pass, category-by-category correction, attorney review and consensus, an “extras” pass using eBrevia’s suggestions, and final export (Hendrycks et al., 2021). The original summary does not report a single Cohen’s $\kappa$ score; instead, it emphasizes multi-stage verification and attorney sign-off as the basis for high consistency (Hendrycks et al., 2021).

CUAD is sparse in the sense relevant to extraction. The original summary states that labeled text comprises about 10% of each contract, while only about 0.25% of tokens are highlighted per category (Hendrycks et al., 2021). This sparsity is one reason why retrieval, chunking, and negative-example handling matter so strongly in downstream models.

3. Task formulation and evaluation protocol

CUAD’s canonical formulation is extractive question answering in the style of SQuAD 2.0. Given a contract text $x$ and a query $q$ consisting of a label category name plus a one- or two-sentence description, the system predicts a set $S$ of zero or more answer spans within the contract (Hendrycks et al., 2021). Transformer baselines produce token representations and then apply two independent softmax classifiers for start and end positions, with loss

$$L = -\log P_{\text{start}}(i^*) - \log P_{\text{end}}(j^*),$$

where $(i^*, j^*)$ are the true start and end indices, and a special null index is used when no span exists (Hendrycks et al., 2021).

Because contracts routinely exceed the 512-token limit of standard BERT-style encoders, the original benchmark uses a sliding-window procedure with stride around 128 tokens to generate overlapping chunks (Hendrycks et al., 2021). Queries and passages are concatenated as [CLS] q [[SEP](https://www.emergentmind.com/topics/semantic-entropy-production-sep-metric)] text_chunk [SEP]. More than 99% of windows contain no relevant span, so negative windows are down-sampled to make positive and negative chunks roughly balanced. When multiple spans for a label appear in a single chunk, separate training instances are created per span (Hendrycks et al., 2021).

Evaluation is based on approximate span matching. Two spans $A$ and $B$ are treated as a match when their word-level Jaccard similarity,

$J(A,B)=\frac{|A\cap B|}{|A\cup B|},$

is at least $\kappa$0, after punctuation removal and case folding (Hendrycks et al., 2021). From these matches, CUAD computes precision, recall, and $\kappa$1, and also traces a precision–recall curve whose area is reported as AUPR. The benchmark additionally reports Precision @ 80% Recall and Precision @ 90% Recall to characterize model usefulness under recall-constrained review settings (Hendrycks et al., 2021).

This evaluation design has consequences. It rewards extractive localization rather than paraphrastic correctness, and it is sensitive to boundary quality. A plausible implication is that systems which retrieve approximately correct regions but quote incompletely may be undervalued by exacting span metrics, a concern that becomes more explicit in later LLM work on CUAD (Klem et al., 2 Sep 2025).

4. Baseline modeling results in the original benchmark

The original CUAD study evaluates BERT-base, BERT-large, ALBERT-base/large/xlarge/xxlarge, RoBERTa-base, RoBERTa-large, DeBERTa-xlarge, and RoBERTa-base with contracts-only pretraining on 8 GB of unlabeled EDGAR data (Hendrycks et al., 2021). The main test-set results show that stronger architectures improve performance materially, but also that the benchmark remains difficult.

BERT-base achieves 32.4% AUPR, 8.2% Precision @ 80% Recall, 0.0% Precision @ 90% Recall, and 61.5% Exact Match. RoBERTa-large reaches 48.2% AUPR, 38.1% Precision @ 80% Recall, 0.0% Precision @ 90% Recall, and 75.1% Exact Match. DeBERTa-xlarge records 47.8% AUPR, 44.0% Precision @ 80% Recall, 17.8% Precision @ 90% Recall, and 74.5% Exact Match (Hendrycks et al., 2021). The strongest AUPR in the reported table is therefore RoBERTa-large, whereas DeBERTa-xlarge gives the best reported precision under the fixed-recall conditions emphasized for review workflows.

The benchmark also isolates two important effects. First, model design matters: DeBERTa-xlarge improves Precision @ 80% Recall from 8.2% for BERT-base to 44.0% (Hendrycks et al., 2021). Second, data scale matters comparably. In a RoBERTa-base ablation, AUPR rises from 27.6 with 3% of the training annotations to 35.0 with 10%, 40.1 with 30%, and 42.6 with the full training set (Hendrycks et al., 2021). The authors characterize this 10-fold increase in supervision as yielding gains comparable to moving to a larger model.

Domain-adaptive masked-language-model pretraining is helpful but limited in the reported setup. RoBERTa-base plus 8 GB of EDGAR pretraining improves from 42.6% to 45.2% AUPR, a gain of 2.6 percentage points over vanilla RoBERTa-base (Hendrycks et al., 2021). The original summary interprets this as evidence that, at least for CUAD as studied there, high-quality supervised annotations are more valuable than additional contracts-only MLM pretraining.

Per-category difficulty is highly uneven. For DeBERTa-xlarge, some labels exceed 90% AUPR, including Document Name and Governing Law, while others remain around 20% (Hendrycks et al., 2021). This heterogeneity is central to CUAD’s identity as a benchmark: it is not a single homogeneous extraction problem, but a family of clause-localization problems spanning short headers, highly formulaic provisions, and legally nuanced or cross-clause concepts.

5. Structure-aware long-document modeling and visual cues

A major line of CUAD research asks whether long-document performance improves when models recover document structure before running standard 512-token transformers. Hegel et al. study this question by combining OCR-derived visual metadata with a pre-trained BERT-style backbone (Hegel et al., 2021). Their setup is motivated by the observation that legal agreements encode structure through layout, style, and placement as much as through lexical content.

In the reported pipeline, PDFs are first passed through an off-the-shelf OCR engine, such as Tesseract or commercial OCR, which returns token text, page numbers, character-level bounding boxes $\kappa$2, font size, style, indentation, justification, and paragraph/table/layout grouping (Hegel et al., 2021). Four feature groups are then computed for each token or line: page layout features, text placement features, visual grouping features, and stylistic features. These are concatenated into a visual vector $\kappa$3 and fused with the text embedding $\kappa$4 using an implied early-fusion form

$\kappa$5

after which the fused representations are passed through a standard transformer encoder (Hegel et al., 2021).

The section-splitting model predicts labels at token or line level over the set $\kappa$6 with

$\kappa$7

and is trained with multi-class cross-entropy (Hegel et al., 2021). Contiguous tokens with the same label form a section, and each section is guaranteed to fit beneath the 512-token limit. Importantly, CUAD itself does not provide section-boundary labels; the segmentation model is trained and evaluated on a separate proprietary corpus, while the downstream extraction and classification tasks are run on CUAD (Hegel et al., 2021).

The segmentation ablation shows mixed but mostly positive effects from OCR features. Clause F1 rises from 0.900 to 0.910, Sub-clause F1 from 0.907 to 0.911, and Footer F1 from 0.800 to 0.872, while Header F1 falls from 0.927 to 0.906 (Hegel et al., 2021). The downstream CUAD results are stronger. For entity extraction, Expiration Date improves from 0.83 F1 with DeBERTa-xlarge to 0.87 with the structure-aware model, and Governing Law improves from 0.88 to 0.98. For classification, Termination for Convenience improves from 0.51 to 0.76 F1, and Anti-Assignment from 0.78 to 0.89 (Hegel et al., 2021). A dedicated visual-cue ablation on Anti-Assignment classification reports $\kappa$8 without visual cues and $\kappa$9 with visual cues (Hegel et al., 2021).

These results directly counter the assumption that CUAD can be treated as a purely linear-text benchmark. The reported gains are attributed to the fact that contracts rely heavily on formatting, numbering, indentation, bold headings, and page artifacts, and that cleaner section splits produce cleaner context windows for BERT (Hegel et al., 2021).

6. Structured prompting, chunk heuristics, and retrieval-oriented use of CUAD

A later research direction treats CUAD less as a fine-tuning benchmark and more as a testbed for prompt-based long-document retrieval. Klem et al. apply a structured prompting methodology to CUAD using QWEN-2 7B without any fine-tuning (Klem et al., 2 Sep 2025). Their pipeline begins by splitting each contract into chunks of $x$0 words, approximately 1500 tokens, with overlap ratio $x$1, and then augmenting the document with overlapping “reduplication” chunks constructed from adjacent chunk boundaries (Klem et al., 2 Sep 2025).

The final prompt template instructs the model to identify the part corresponding to a query if the information is directly present, and otherwise respond only with “Does not exist”; it also demands that the answer be quoted word for word, exactly as it appears in the document (Klem et al., 2 Sep 2025). This design explicitly combines domain specification, verbatim extraction, and a forced null-answer token. After candidate answers are generated per chunk and per question, the system ranks them using two heuristics.

The first heuristic, Distribution-Based Localisation (DBL), partitions each document into $x$2 equal-length slices and estimates where answers for a given question tend to appear in relative document position over the training corpus (Klem et al., 2 Sep 2025). Each chunk then receives a DBL weight according to the aggregate density of the slices it covers. The second heuristic, Inverse Cardinality Weighting (ICW), embeds candidate answers with GritLM, clusters them with DBSCAN using cosine distance, and assigns each candidate a weight of $x$3 where $x$4 is the size of its cluster; noise points are treated as singleton clusters (Klem et al., 2 Sep 2025). The combined score is

$x$5

On 41 questions over a 102-document test set, the QWEN-2 system with structured prompting and these heuristics matches or beats the DeBERTa-large fine-tuned extractive baseline on 39 of 41 questions, with an average correctness gain of approximately 9 percentage points per question, corresponding to a total of 250 additional correct spans (Klem et al., 2 Sep 2025). When true negatives are excluded and only clause-present cases are considered, the average gain is approximately 6 percentage points, or 34 additional cases (Klem et al., 2 Sep 2025). In a separate $x$6 study of prompt complexity and augmentation using automatic metrics, the “Aug Comp” condition is reported as the strongest, with absolute score 10,868 and percentage score 0.519, compared with 6,775 and 0.324 for “Basic” (Klem et al., 2 Sep 2025).

The same study also emphasizes evaluation problems. It reports that ROUGE, METEOR, and cosine-based metrics are too coarse for mixed-length legal spans, no statistical-significance tests are reported, and human span validation remains important (Klem et al., 2 Sep 2025). This is not merely a critique of one system; it highlights a recurrent issue in CUAD research, namely that span correctness and legal usefulness are not perfectly captured by generic automatic text-overlap metrics.

7. LLM-based metadata extraction, benchmark interpretation, and persistent limitations

CUAD has also been used as a public benchmark for end-to-end metadata extraction pipelines driven by modern LLMs. One such study evaluates Text-Unicorn, Claude 3.5 Sonnet, and Gemini 1.5 Flash on a CUAD-based pipeline that combines Azure Document Intelligence for text conversion, strategic chunk selection, and LLM prompting with JSON-structured tool calling (Han et al., 22 Oct 2025). The study states that CUAD comprises just over 500 real-world contracts yielding more than 13,000 expert annotations over 41 clause categories, and that its own experiments focus on metadata-centric fields including Document Name, Parties, Agreement Date, Effective Date, Expiration Date, Notice to Terminate/Renewal, and Governing Law (Han et al., 22 Oct 2025).

The reported pipeline retains structural markers such as section headers, numbered subclauses, and whitespace cues during OCR-based text conversion. It then uses either 4,096-token or 8,192-token sliding windows with 25% overlap, followed by section-aware ranking. In the first ranking phase, named-entity recognition boosts chunks containing field-relevant entities, and four signals—per-field cosine similarity, total-template similarity, per-field NER counts, and total-template NER counts—are fused with a Borda-count re-ranking scheme (Han et al., 22 Oct 2025). In the second phase, a lightweight neural reranker consumes embeddings, BM25 scores, NER/POS-tag features, and chunk-position metadata to predict chunk relevance.

The main CUAD results reported in that study are field-averaged $x$7 scores: 0.66 for a 4,096-token baseline with Text-Unicorn, 0.74 with NER-boosted re-ranking, 0.76 with a neural chunk reranker, 0.75 for an 8,192-token baseline, 0.80 for the reranked 8,192-token approach, 0.77 for a 64K-token Claude 3.5 Sonnet baseline, and 0.82 for a chunk-reranked 16K-token Gemini 1.5 Flash system (Han et al., 22 Oct 2025). Against an oracle upper bound of 0.94, the reranked 8,192-token setup is reported to close most of the gap while using approximately 40% fewer tokens than a naïve 64K-token pass, for an estimated approximately 30% cost reduction in pay-per-token billing (Han et al., 22 Oct 2025). Straightforward fields such as Document Name, Parties, and Effective Date achieve over 90% recall under the reranker, while more complex fields such as Notice to Terminate/Renewal and Governing Law remain the largest sources of false negatives; Chain-of-Thought prompting recovers approximately 5% of these misses, but those fields still trail single-sentence extractions by about 8 $x$8 points (Han et al., 22 Oct 2025).

Taken together, the later CUAD literature establishes several benchmark-interpretation points. First, larger context windows alone are not sufficient: both structure-aware segmentation and chunk reranking materially improve outcomes over naïve long-context processing (Hegel et al., 2021, Han et al., 22 Oct 2025). Second, CUAD is not a section-label dataset; when section splitting is evaluated, it is evaluated off-domain because CUAD lacks section-boundary annotations (Hegel et al., 2021). Third, high recall remains costly in precision terms even for strong systems, a difficulty already visible in the original benchmark, where Precision @ 90% Recall is below 20% for all but the largest models reported there (Hendrycks et al., 2021). Finally, evaluation remains unsettled: original CUAD metrics rely on Jaccard-based span overlap, while later prompt-based work argues that common automatic generation metrics are too coarse for legal extraction (Hendrycks et al., 2021, Klem et al., 2 Sep 2025).

CUAD therefore occupies a specific position in legal NLP. It is a high-confidence, expert-verified benchmark for extractive contract review over long commercial agreements; it is also a methodological stress test for long-context modeling, layout awareness, retrieval, and prompt design. The benchmark’s enduring significance lies not in solving legal reasoning wholesale, but in exposing how difficult it is to recover legally salient spans reliably under realistic document length, structural variation, and annotation sparsity.