CUAD-QA: Privacy-Preserving Contract QA

• CUAD-QA is a legal contract QA benchmark that redefines review from clause extraction to natural language question answering with a focus on privacy preservation.
• It includes approximately 85,000 QA pairs generated from 510 diverse contracts, covering simple, complex, and summarization queries for enterprise legal analysis.
• Key evaluation metrics demonstrate high restoration accuracy and response relevancy, ensuring effective contract review while safeguarding sensitive data.

Searching arXiv for CUAD, CUAD-QA, and related contract QA papers to ground the article in current literature. CUAD-QA is a large-scale, contract-focused question–answering corpus introduced to evaluate question answering over enterprise legal documents in settings where privacy preservation is a central requirement. It is built on the 510 real-world contract documents of CUAD, the Contract Understanding Atticus Dataset, which contains 41 expert-annotated clause categories for legal contract review (Hendrycks et al., 2021). Whereas CUAD centers on clause identification and classification, CUAD-QA converts those contracts into a QA formulation with approximately 85,000 question–answer pairs spanning simple, complex, and summarization-style queries, thereby aligning the benchmark more closely with enterprise contract analysis workflows and with privacy-preserving pipelines such as CON-QA (Singh et al., 24 Sep 2025).

1. Origins in CUAD and the shift from clause extraction to QA

CUAD-QA derives directly from CUAD, a benchmark released for legal contract review that contains 510 contracts, 13,101 labeled clause spans, and 41 issue categories across 25 contract types drawn from the SEC’s EDGAR system (Hendrycks et al., 2021). In CUAD, the task is to highlight salient portions of a contract that are important for human review, using expert-annotated spans and a SQuAD 2.0-style framing in which models must often predict “no answer” (Hendrycks et al., 2021).

CUAD-QA preserves the same contract base while changing the task formulation. The newer corpus is described as clause-grounded QA generated over those 510 CUAD contracts and aimed at realistic enterprise usage, where questions may refer to sensitive entities, clause semantics, and cross-clause obligations (Singh et al., 24 Sep 2025). The reported motivation is twofold: to provide a realistic QA benchmark for cloud LLMs operating on sensitive contracts, and to support evaluation of privacy-preserving systems that anonymize personally identifiable information and commercially sensitive entities while preserving legal fidelity (Singh et al., 24 Sep 2025).

This transition is methodologically significant. CUAD operationalizes contract review as issue-conditioned span extraction; CUAD-QA operationalizes it as question answering over full-length contracts. A plausible implication is that the benchmark moves from a primarily extractive, label-conditioned setting toward a broader retrieval-and-generation setting in which both extractive and abstractive behavior are relevant.

2. Corpus composition and contractual coverage

CUAD-QA is reported to contain approximately 85,000 QA pairs generated over 510 full-length contract documents from CUAD (Singh et al., 24 Sep 2025). The generation process was performed page-by-page, and the paper states that it yields about 10 QA pairs per contract page (Singh et al., 24 Sep 2025). Because the source corpus is CUAD, the contractual domain inherits CUAD’s coverage of 25 contract types drawn from EDGAR filings, including Distributor Agreement, License Agreement, Maintenance Agreement, and Strategic Alliance Agreement, among others (Hendrycks et al., 2021).

The corpus is explicitly positioned as contract-focused and clause-grounded. Its queries cover governing law, payment and termination obligations, exclusivity, effective dates, signatories, exhibits, and jurisdictional scope (Singh et al., 24 Sep 2025). The privacy relevance of the corpus is also emphasized: the questions and answers frequently involve personally identifiable information and commercially sensitive identifiers such as company names, signatories, jurisdictions, exhibits, thresholds, and proprietary technologies (Singh et al., 24 Sep 2025).

The available documentation does not report per-type counts for query categories, question or answer length statistics, or histograms of distributional properties (Singh et al., 24 Sep 2025). It also does not specify train/dev/test splits (Singh et al., 24 Sep 2025). Accordingly, CUAD-QA is best understood as a large evaluation-oriented QA corpus rather than a fully documented benchmark with a published split protocol analogous to conventional supervised QA datasets.

3. Query taxonomy and answer forms

CUAD-QA is organized around three query types encountered in contract review: simple, complex, and summarization (Singh et al., 24 Sep 2025). Simple questions target direct lookup of a single clause, exemplified by queries such as “What is the effective date of this agreement?” with an answer like “January 1, 2023” (Singh et al., 24 Sep 2025). Complex questions require multi-clause reasoning, as in asking whether an exclusivity clause applies to all product categories in a given geography (Singh et al., 24 Sep 2025). Summarization questions require abstractive synthesis across clauses, such as summarizing a buyer’s obligations in the event of early termination (Singh et al., 24 Sep 2025).

The answer space therefore includes both extractive and abstractive outputs. The paper states that the corpus includes extractive answers such as dates and named parties, and abstractive or summarized answers when required by the query type, but it does not quantify the extractive-to-abstractive ratio (Singh et al., 24 Sep 2025).

This query design distinguishes CUAD-QA from CUAD’s issue-as-query span extraction format. In CUAD, each issue category is converted into a prompt such as “Highlight the parts (if any) of this clause related to ‘<Label Category>’,” with evaluation based on span overlap and class-imbalanced precision–recall measures (Hendrycks et al., 2021). In CUAD-QA, the prompts are natural language questions anchored in realistic contract analysis scenarios, and the outputs are evaluated in a generation setting with privacy instrumentation (Singh et al., 24 Sep 2025).

A common misconception would be to treat CUAD-QA as merely a reformatting of CUAD’s 41 categories into natural-language prompts. The reported scope is broader: CUAD-QA includes single-clause lookup, multi-clause reasoning, and summarization, and it is embedded in a retrieval, anonymization, generation, and reconstruction pipeline rather than only span extraction (Singh et al., 24 Sep 2025).

4. Construction methodology and data schema

The QA pairs in CUAD-QA were generated by ChatGPT-4o-mini, prompted with individual contract pages to produce natural, clause-grounded questions and answers (Singh et al., 24 Sep 2025). The construction process is therefore LLM-generated rather than independently human-annotated at the QA level (Singh et al., 24 Sep 2025). Quality assessment is reported downstream through automated metrics and expert review of sampled outputs, but the paper does not describe a separate human annotation phase for the QA pairs themselves and does not report inter-annotator agreement for the corpus (Singh et al., 24 Sep 2025).

The paper does not publish a formal JSON schema, but it describes the minimal structure implicit in the pipeline. Instances include a document or page reference linked to the source CUAD contract and selected page or chunk, question text and reference answer text produced by ChatGPT-4o-mini, a query type label in the set {simple, complex, summarization}, and implicit clause grounding through page-level prompting and retrieval-augmented chunk selection (Singh et al., 24 Sep 2025). Explicit clause identifiers or answer spans are not reported (Singh et al., 24 Sep 2025). Sensitive entity metadata is detected during anonymization and deanonymization using GLiNER, but entity offsets or entity categories are not described as part of a public schema (Singh et al., 24 Sep 2025).

These design choices matter for how CUAD-QA should be interpreted. It is clause-grounded, but not span-supervised in the sense of CUAD’s start/end annotations (Hendrycks et al., 2021). It is grounded in real contracts, but the QA layer is synthetic in the sense that it is generated by an LLM rather than by contract lawyers or law students (Singh et al., 24 Sep 2025). This suggests that CUAD-QA is most appropriate for evaluating end-to-end contractual QA behavior, especially under anonymization constraints, rather than for fine-grained supervised extraction with gold span boundaries.

5. Role within the CON-QA privacy-preserving framework

CUAD-QA functions as the QA substrate for CON-QA, a hybrid privacy-preserving framework for secure question answering over enterprise contracts (Singh et al., 24 Sep 2025). Within that framework, the input query $X$ is first analyzed locally by Qwen-2.5-14B to extract structured semantic components

$$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$

with a schema-guided extracted representation

$$A' = \{a'_i \mid a'_i \in A, \text{ extracted via Qwen}\}.$$

Relevant document identifiers are then matched against metadata, and chunk retrieval is performed through semantic search in a local vector database (Singh et al., 24 Sep 2025).

Privacy protection is implemented by local anonymization of sensitive entities detected in the query and retrieved chunks. The paper formalizes entity detection through a NER operator $\mathcal{N}(\cdot)$, producing query entities $\mathbb{E}_Q$, chunk entities $\mathbb{E}_{d_j}$, and a combined set $\mathbb{E}_{total}$ (Singh et al., 24 Sep 2025). For each sensitive entity $e_i$, a one-to-many surrogate set

$S_i = \{s_{i1}, \dots, s_{iK}\}$

is generated under semantic and diversity constraints, and a session-specific randomized replacement

$$\tilde{e}_i = \phi^{(t)}(e_i) \sim \text{Uniform}(S_i)$$

is chosen (Singh et al., 24 Sep 2025). An equivalent formulation uses a session-specific one-to-many mapping $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$0 and a many-to-one reverse mapping $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$1, both deleted after the session to prevent cross-session inference (Singh et al., 24 Sep 2025).

The anonymized query and anonymized chunks are sent to GPT-4o-mini, which produces an anonymized answer

$$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$2

Deanonymization then applies the local inverse mapping,

$$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$3

to reconstruct the original answer with restored entities (Singh et al., 24 Sep 2025).

Within this architecture, CUAD-QA is not merely a benchmark collection; it is the contractual QA workload on which the privacy, retrieval, anonymization, and answer-reconstruction stages are evaluated. Its design is therefore inseparable from the operational demands of privacy-preserving enterprise legal QA.

6. Evaluation protocol and reported empirical results

The end-to-end CON-QA evaluation on CUAD-QA samples 1,000 QA pairs from the corpus (Singh et al., 24 Sep 2025). The principal metrics reported are Private Entity Restoration Accuracy, Response Relevancy under RAGAS, Answer Correctness, and Faithfulness to retrieved context (Singh et al., 24 Sep 2025). The paper gives the following results for that evaluation.

Metric	Reported value
Private Entity Restoration Accuracy	0.9880
Response Relevancy (RAGAS)	0.9778
Answer Correctness	0.8810
Faithfulness	0.9886

Private Entity Restoration Accuracy is described as the proportion of sensitive entities correctly restored in the final answer relative to the non-anonymized reference pipeline, with a formalization consistent with

$$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$4

where $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$5 is the set of sensitive entities in the gold answer and $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$6 is the set in the deanonymized output (Singh et al., 24 Sep 2025). The RAGAS-based metrics follow the formulation of Es et al. (2024), though formulas are not reproduced in the paper (Singh et al., 24 Sep 2025).

The paper also reports direct comparisons against Hide-and-Seek (HaS). For anonymization over 50 contractual queries or chunks with an average of 18 PII entities per item, CON-QA achieves 99.11% coverage, 1.34% surrogate reuse, 98.76% unique surrogates, 1.23% linkability, and 0.89% missed entities, compared with 76.15%, 46.71%, 59.12%, 46.71%, and 31.38%, respectively, for HaS Hide (Singh et al., 24 Sep 2025). For deanonymization over 50 QA items, CON-QA reaches 98.00% entity accuracy and 83.00% BERTScore, compared with 89.00% and 81.10% for HaS Seek (Singh et al., 24 Sep 2025).

Human expert evaluation is reported on 30 QA sets, with an NLP expert panel of $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$7 assigning overall accuracy of 85.83% and legal experts of $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$8 assigning 91.66% (Singh et al., 24 Sep 2025). The paper notes that summarization-style questions may show minor drops under strict surface-level evaluation because recovered answers can be more elaborative than concise references (Singh et al., 24 Sep 2025).

These results frame CUAD-QA as a benchmark in which privacy preservation and QA utility are assessed jointly. Unlike CUAD’s span-overlap evaluation based on AUPR, Precision@Recall, Jaccard similarity, and EM (Hendrycks et al., 2021), CUAD-QA is evaluated through restoration fidelity, retrieval-grounded response quality, and expert judgment (Singh et al., 24 Sep 2025). The metric shift reflects the task shift from extractive clause localization to privacy-aware generative QA.

7. Limitations, reproducibility, and place in legal NLP

Several limitations are explicitly reported. First, CUAD-QA’s QA pairs are LLM-generated, and the paper does not report separate human annotation of those pairs or inter-annotator agreement (Singh et al., 24 Sep 2025). Second, the paper does not provide formal train/dev/test splits, a public download link, or explicit license terms for CUAD-QA, although it notes that CUAD itself is publicly available (Singh et al., 24 Sep 2025). Third, explicit clause identifiers and answer spans are not reported, which limits direct use for span-supervised extractive QA (Singh et al., 24 Sep 2025). Fourth, the source contracts come from CUAD’s EDGAR-based collection, which focuses on complex, heavily negotiated agreements and may underrepresent broader legal instruments such as consumer or employment contracts across diverse jurisdictions (Hendrycks et al., 2021).

Ethical and deployment-related caveats are also central. The privacy mechanism depends on ephemerality and session isolation of the mapping functions $$A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},$$9, $$A' = \{a'_i \mid a'_i \in A, \text{ extracted via Qwen}\}.$$0, and the surrogate mapping set $$A' = \{a'_i \mid a'_i \in A, \text{ extracted via Qwen}\}.$$1; the paper states that implementers must ensure secure handling and deletion of these artifacts (Singh et al., 24 Sep 2025). It further notes that, although anonymization is strong, downstream usage should still adhere to corporate policies and regulatory frameworks such as GDPR and HIPAA (Singh et al., 24 Sep 2025). Residual risk remains because cloud answer generation uses GPT-4o-mini, and the paper recommends rigorous red-teaming (Singh et al., 24 Sep 2025).

Within legal NLP, CUAD-QA occupies a distinct position. CUAD is an expert-annotated contract review dataset oriented toward clause categorization and span extraction across 41 categories (Hendrycks et al., 2021). LawInstruct and LegalBench-Instruct are mentioned as instruction-tuning and evaluation resources across legal tasks, but not as contract-specific QA corpora with privacy instrumentation (Singh et al., 24 Sep 2025). CUAD-QA fills the narrower niche of contract QA coupled to anonymization and deanonymization evaluation for cloud LLM usage (Singh et al., 24 Sep 2025).

Its main significance lies in combining three elements that had often been treated separately: real enterprise-style contracts inherited from CUAD, natural-language question answering over those contracts, and explicit privacy-preserving evaluation. This suggests that CUAD-QA is best viewed not simply as a derivative of CUAD, but as a benchmark substrate for studying whether cloud-based contractual QA can preserve both legal utility and confidentiality under realistic operational constraints (Singh et al., 24 Sep 2025).