---
title: 'CUAD-QA: Privacy-Preserving Contract QA'
url: https://www.emergentmind.com/topics/cuad-qa
type: topic
---

# CUAD-QA: Privacy-Preserving Contract QA

Searching arXiv for CUAD, CUAD-QA, and related contract QA papers to ground the article in current literature.
CUAD-QA is a large-scale, contract-focused question–answering corpus introduced to evaluate question answering over enterprise legal documents in settings where privacy preservation is a central requirement. It is built on the 510 real-world contract documents of CUAD, the Contract Understanding Atticus Dataset, which contains 41 expert-annotated clause categories for legal contract review [2103.06268]. Whereas CUAD centers on clause identification and classification, CUAD-QA converts those contracts into a QA formulation with approximately 85,000 question–answer pairs spanning simple, complex, and summarization-style queries, thereby aligning the benchmark more closely with enterprise contract analysis workflows and with privacy-preserving pipelines such as CON-QA [2509.19925].

## 1. Origins in CUAD and the shift from clause extraction to QA

CUAD-QA derives directly from CUAD, a benchmark released for legal contract review that contains 510 contracts, 13,101 labeled clause spans, and 41 issue categories across 25 contract types drawn from the SEC’s EDGAR system [2103.06268]. In CUAD, the task is to highlight salient portions of a contract that are important for human review, using expert-annotated spans and a SQuAD 2.0-style framing in which models must often predict “no answer” [2103.06268].

CUAD-QA preserves the same contract base while changing the task formulation. The newer corpus is described as clause-grounded QA generated over those 510 CUAD contracts and aimed at realistic enterprise usage, where questions may refer to sensitive entities, clause semantics, and cross-clause obligations [2509.19925]. The reported motivation is twofold: to provide a realistic QA benchmark for cloud LLMs operating on sensitive contracts, and to support evaluation of privacy-preserving systems that anonymize personally identifiable information and commercially sensitive entities while preserving legal fidelity [2509.19925].

This transition is methodologically significant. CUAD operationalizes contract review as issue-conditioned span extraction; CUAD-QA operationalizes it as question answering over full-length contracts. A plausible implication is that the benchmark moves from a primarily extractive, label-conditioned setting toward a broader retrieval-and-generation setting in which both extractive and abstractive behavior are relevant.

## 2. Corpus composition and contractual coverage

CUAD-QA is reported to contain approximately 85,000 QA pairs generated over 510 full-length contract documents from CUAD [2509.19925]. The generation process was performed page-by-page, and the paper states that it yields about 10 QA pairs per contract page [2509.19925]. Because the source corpus is CUAD, the contractual domain inherits CUAD’s coverage of 25 contract types drawn from EDGAR filings, including Distributor Agreement, License Agreement, Maintenance Agreement, and Strategic Alliance Agreement, among others [2103.06268].

The corpus is explicitly positioned as contract-focused and clause-grounded. Its queries cover governing law, payment and termination obligations, exclusivity, effective dates, signatories, exhibits, and jurisdictional scope [2509.19925]. The privacy relevance of the corpus is also emphasized: the questions and answers frequently involve personally identifiable information and commercially sensitive identifiers such as company names, signatories, jurisdictions, exhibits, thresholds, and proprietary technologies [2509.19925].

The available documentation does not report per-type counts for query categories, question or answer length statistics, or histograms of distributional properties [2509.19925]. It also does not specify train/dev/test splits [2509.19925]. Accordingly, CUAD-QA is best understood as a large evaluation-oriented QA corpus rather than a fully documented benchmark with a published split protocol analogous to conventional supervised QA datasets.

## 3. Query taxonomy and answer forms

CUAD-QA is organized around three query types encountered in contract review: simple, complex, and summarization [2509.19925]. Simple questions target direct lookup of a single clause, exemplified by queries such as “What is the effective date of this agreement?” with an answer like “January 1, 2023” [2509.19925]. Complex questions require multi-clause reasoning, as in asking whether an exclusivity clause applies to all product categories in a given geography [2509.19925]. Summarization questions require abstractive synthesis across clauses, such as summarizing a buyer’s obligations in the event of early termination [2509.19925].

The answer space therefore includes both extractive and abstractive outputs. The paper states that the corpus includes extractive answers such as dates and named parties, and abstractive or summarized answers when required by the query type, but it does not quantify the extractive-to-abstractive ratio [2509.19925].

This query design distinguishes CUAD-QA from CUAD’s issue-as-query span extraction format. In CUAD, each issue category is converted into a prompt such as “Highlight the parts (if any) of this clause related to ‘<Label Category>’,” with evaluation based on span overlap and class-imbalanced precision–recall measures [2103.06268]. In CUAD-QA, the prompts are natural language questions anchored in realistic contract analysis scenarios, and the outputs are evaluated in a generation setting with privacy instrumentation [2509.19925].

A common misconception would be to treat CUAD-QA as merely a reformatting of CUAD’s 41 categories into natural-language prompts. The reported scope is broader: CUAD-QA includes single-clause lookup, multi-clause reasoning, and summarization, and it is embedded in a retrieval, anonymization, generation, and reconstruction pipeline rather than only span extraction [2509.19925].

## 4. Construction methodology and data schema

The QA pairs in CUAD-QA were generated by ChatGPT-4o-mini, prompted with individual contract pages to produce natural, clause-grounded questions and answers [2509.19925]. The construction process is therefore LLM-generated rather than independently human-annotated at the QA level [2509.19925]. Quality assessment is reported downstream through automated metrics and expert review of sampled outputs, but the paper does not describe a separate human annotation phase for the QA pairs themselves and does not report inter-annotator agreement for the corpus [2509.19925].

The paper does not publish a formal JSON schema, but it describes the minimal structure implicit in the pipeline. Instances include a document or page reference linked to the source CUAD contract and selected page or chunk, question text and reference answer text produced by ChatGPT-4o-mini, a query type label in the set \{simple, complex, summarization\}, and implicit clause grounding through page-level prompting and retrieval-augmented chunk selection [2509.19925]. Explicit clause identifiers or answer spans are not reported [2509.19925]. Sensitive entity metadata is detected during anonymization and deanonymization using GLiNER, but entity offsets or entity categories are not described as part of a public schema [2509.19925].

These design choices matter for how CUAD-QA should be interpreted. It is clause-grounded, but not span-supervised in the sense of CUAD’s start/end annotations [2103.06268]. It is grounded in real contracts, but the QA layer is synthetic in the sense that it is generated by an LLM rather than by contract lawyers or law students [2509.19925]. This suggests that CUAD-QA is most appropriate for evaluating end-to-end contractual QA behavior, especially under anonymization constraints, rather than for fine-grained supervised extraction with gold span boundaries.

## 5. Role within the CON-QA privacy-preserving framework

CUAD-QA functions as the QA substrate for CON-QA, a hybrid privacy-preserving framework for secure question answering over enterprise contracts [2509.19925]. Within that framework, the input query \(X\) is first analyzed locally by Qwen-2.5-14B to extract structured semantic components
$$
A = \{doc\_ids, parties, metadata\_fields, text\_search\_terms, query\_type, dates\},
$$
with a schema-guided extracted representation
$$
A' = \{a'_i \mid a'_i \in A, \text{ extracted via Qwen}\}.
$$
Relevant document identifiers are then matched against metadata, and chunk retrieval is performed through semantic search in a local vector database [2509.19925].

Privacy protection is implemented by local anonymization of sensitive entities detected in the query and retrieved chunks. The paper formalizes entity detection through a NER operator \(\mathcal{N}(\cdot)\), producing query entities \(\mathbb{E}_Q\), chunk entities \(\mathbb{E}_{d_j}\), and a combined set \(\mathbb{E}_{total}\) [2509.19925]. For each sensitive entity \(e_i\), a one-to-many surrogate set
$$
S_i = \{s_{i1}, \dots, s_{iK}\}
$$
is generated under semantic and diversity constraints, and a session-specific randomized replacement
$$
\tilde{e}_i = \phi^{(t)}(e_i) \sim \text{Uniform}(S_i)
$$
is chosen [2509.19925]. An equivalent formulation uses a session-specific one-to-many mapping \(f_s: E \to A_s\) and a many-to-one reverse mapping \(g_s: A_s \to E\), both deleted after the session to prevent cross-session inference [2509.19925].

The anonymized query and anonymized chunks are sent to GPT-4o-mini, which produces an anonymized answer
$$
\tilde{A} = f_{LLM}(\tilde{Q}, \tilde{D}).
$$
Deanonymization then applies the local inverse mapping,
$$
A = \phi^{-1}(\tilde{A}),
$$
to reconstruct the original answer with restored entities [2509.19925].

Within this architecture, CUAD-QA is not merely a benchmark collection; it is the contractual QA workload on which the privacy, retrieval, anonymization, and answer-reconstruction stages are evaluated. Its design is therefore inseparable from the operational demands of privacy-preserving enterprise legal QA.

## 6. Evaluation protocol and reported empirical results

The end-to-end CON-QA evaluation on CUAD-QA samples 1,000 QA pairs from the corpus [2509.19925]. The principal metrics reported are Private Entity Restoration Accuracy, Response Relevancy under RAGAS, Answer Correctness, and Faithfulness to retrieved context [2509.19925]. The paper gives the following results for that evaluation.

| Metric | Reported value |
|---|---:|
| Private Entity Restoration Accuracy | 0.9880 |
| Response Relevancy (RAGAS) | 0.9778 |
| Answer Correctness | 0.8810 |
| Faithfulness | 0.9886 |

Private Entity Restoration Accuracy is described as the proportion of sensitive entities correctly restored in the final answer relative to the non-anonymized reference pipeline, with a formalization consistent with
$$
\text{Accuracy} = \frac{|E^* \cap \hat{E}|}{|E^*|},
$$
where \(E^*\) is the set of sensitive entities in the gold answer and \(\hat{E}\) is the set in the deanonymized output [2509.19925]. The RAGAS-based metrics follow the formulation of Es et al. (2024), though formulas are not reproduced in the paper [2509.19925].

The paper also reports direct comparisons against Hide-and-Seek (HaS). For anonymization over 50 contractual queries or chunks with an average of 18 PII entities per item, CON-QA achieves 99.11% coverage, 1.34% surrogate reuse, 98.76% unique surrogates, 1.23% linkability, and 0.89% missed entities, compared with 76.15%, 46.71%, 59.12%, 46.71%, and 31.38%, respectively, for HaS Hide [2509.19925]. For deanonymization over 50 QA items, CON-QA reaches 98.00% entity accuracy and 83.00% BERTScore, compared with 89.00% and 81.10% for HaS Seek [2509.19925].

Human expert evaluation is reported on 30 QA sets, with an NLP expert panel of \(n=8\) assigning overall accuracy of 85.83% and legal experts of \(n=2\) assigning 91.66% [2509.19925]. The paper notes that summarization-style questions may show minor drops under strict surface-level evaluation because recovered answers can be more elaborative than concise references [2509.19925].

These results frame CUAD-QA as a benchmark in which privacy preservation and QA utility are assessed jointly. Unlike CUAD’s span-overlap evaluation based on AUPR, Precision@Recall, Jaccard similarity, and EM [2103.06268], CUAD-QA is evaluated through restoration fidelity, retrieval-grounded response quality, and expert judgment [2509.19925]. The metric shift reflects the task shift from extractive clause localization to privacy-aware generative QA.

## 7. Limitations, reproducibility, and place in legal NLP

Several limitations are explicitly reported. First, CUAD-QA’s QA pairs are LLM-generated, and the paper does not report separate human annotation of those pairs or inter-annotator agreement [2509.19925]. Second, the paper does not provide formal train/dev/test splits, a public download link, or explicit license terms for CUAD-QA, although it notes that CUAD itself is publicly available [2509.19925]. Third, explicit clause identifiers and answer spans are not reported, which limits direct use for span-supervised extractive QA [2509.19925]. Fourth, the source contracts come from CUAD’s EDGAR-based collection, which focuses on complex, heavily negotiated agreements and may underrepresent broader legal instruments such as consumer or employment contracts across diverse jurisdictions [2103.06268].

Ethical and deployment-related caveats are also central. The privacy mechanism depends on ephemerality and session isolation of the mapping functions \(f_s\), \(g_s\), and the surrogate mapping set \(\mathcal{M}\); the paper states that implementers must ensure secure handling and deletion of these artifacts [2509.19925]. It further notes that, although anonymization is strong, downstream usage should still adhere to corporate policies and regulatory frameworks such as GDPR and HIPAA [2509.19925]. Residual risk remains because cloud answer generation uses GPT-4o-mini, and the paper recommends rigorous red-teaming [2509.19925].

Within legal NLP, CUAD-QA occupies a distinct position. CUAD is an expert-annotated contract review dataset oriented toward clause categorization and span extraction across 41 categories [2103.06268]. LawInstruct and LegalBench-Instruct are mentioned as instruction-tuning and evaluation resources across legal tasks, but not as contract-specific QA corpora with privacy instrumentation [2509.19925]. CUAD-QA fills the narrower niche of contract QA coupled to anonymization and deanonymization evaluation for cloud LLM usage [2509.19925].

Its main significance lies in combining three elements that had often been treated separately: real enterprise-style contracts inherited from CUAD, natural-language question answering over those contracts, and explicit privacy-preserving evaluation. This suggests that CUAD-QA is best viewed not simply as a derivative of CUAD, but as a benchmark substrate for studying whether cloud-based contractual QA can preserve both legal utility and confidentiality under realistic operational constraints [2509.19925].

Source: https://www.emergentmind.com/topics/cuad-qa