CON-QA: Privacy-Preserving Contract QA

• CON-QA is a hybrid privacy-preserving QA system that securely handles enterprise contracts using a local LLM for query decomposition and anonymization.
• It employs a three-stage pipeline featuring semantic query decomposition, one-to-many surrogate mapping, and cloud-side answer generation with local inverse reconstruction.
• Empirical evaluations show high entity restoration (98%) and low surrogate linkability, balancing answer utility with minimized disclosure risk.

CON-QA is a hybrid privacy-preserving question-answering framework for confidential enterprise contracts that combines a locally deployed LLM with a cloud-hosted LLM in order to retain cloud-scale reasoning quality while reducing exposure of sensitive contractual content. It is designed for contract documents containing personally identifiable information and commercially sensitive clauses, and operates through three stages: local semantic query decomposition with query-aware chunk retrieval, anonymization of detected sensitive entities through a structured one-to-many mapping scheme, and cloud-side answer generation over anonymized inputs followed by local reconstruction of the original answer through a session-consistent many-to-one reverse mapping (Singh et al., 24 Sep 2025).

1. Scope and problem formulation

CON-QA addresses question answering over enterprise contracts in a setting where the underlying documents contain names of people, signatories, addresses, jurisdictions, pricing terms, exclusivity provisions, payment thresholds, obligations, and other legally meaningful or commercially sensitive content. The central problem is not generic legal retrieval, but secure use of cloud LLMs such as GPT-4o-mini, ChatGPT, and Gemini for contract QA without sending raw confidential text to external APIs (Singh et al., 24 Sep 2025).

The framework is motivated by a practical tension between utility and disclosure risk. The paper explicitly argues that standard anonymization or pure differential-privacy-style perturbation is often too destructive for legal QA because it can break clause semantics, reduce retrieval quality, and hurt answer fidelity. CON-QA therefore adopts a hybrid design in which privacy-sensitive analysis, retrieval, anonymization, and reconstruction are performed locally, while the cloud model is used only after semantic preservation has been achieved through anonymization. This suggests that the system is best understood not as a general legal assistant, but as a privacy-preserving orchestration layer for contract-domain retrieval-augmented QA (Singh et al., 24 Sep 2025).

A common simplification is to treat contract QA as ordinary document QA with named-entity masking. The reported design is more specific than that. The local model first transforms a natural-language query into a structured contract-oriented representation, then limits downstream processing to the most relevant document set and chunks, and only then applies session-specific anonymization. Privacy protection is thus coupled to retrieval and answer reconstruction rather than attached as a purely pre-processing step (Singh et al., 24 Sep 2025).

2. Hybrid architecture and local-cloud division of labor

The architecture is explicitly hybrid. A local LLM performs query understanding, query decomposition, contract-aware retrieval, entity detection, surrogate generation, and final deanonymization, while the cloud LLM performs answer generation over anonymized inputs. In the reported implementation, the local model is Qwen2.5-14B-Instruct-1M, the cloud model is GPT-4o-mini, and sensitive-entity detection is handled with GLiNER (Singh et al., 24 Sep 2025).

This division of labor is central to the framework’s privacy model. The local side processes raw confidential inputs in-house and constructs the privacy-preserving interface that the cloud model sees. The cloud side receives only an anonymized query and anonymized retrieved chunks. Because enterprises may not be able to fine-tune or deploy a proprietary legal LLM at comparable scale, the paper positions the cloud LLM as a generation component whose utility can be retained once the exposure surface has been narrowed (Singh et al., 24 Sep 2025).

The framework also includes a document-level metadata index over the CUAD corpus, stored as metadata.json, where each contract is linked to semantic attributes extracted by the local model. Query decomposition is therefore connected to metadata-guided document identification before chunk retrieval. This design differs from generic semantic search pipelines that rely only on vector similarity over raw text, because CON-QA first narrows retrieval to candidate contracts through structured contract metadata and then performs semantic chunk search within the identified documents (Singh et al., 24 Sep 2025).

A plausible implication is that the system’s privacy and utility claims depend not only on anonymization quality but also on the local model’s ability to parse contract-oriented query structure accurately. The paper formalizes the extracted query component set as

$$A = \{ \text{doc\_ids},\, \text{parties},\, \text{metadata\_fields},\, \text{text\_search\_terms},\, \text{query\_type},\, \text{dates} \},$$

and denotes the local LLM’s extracted representation by

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$

Candidate documents are then selected as

$$D = \{ d_k \mid d_k \in metadata.json,\, \text{matches}(d_k, A') \}.$$

These definitions make the retrieval stage contract-aware at the level of parties, dates, clause types, and intent categories rather than only lexical similarity (Singh et al., 24 Sep 2025).

3. Three-stage processing pipeline

The first stage is semantic query decomposition and query-aware document chunk retrieval. Given a user query, the local LLM extracts structured semantic components, searches the metadata index to identify relevant contracts, and then performs semantic similarity search over chunks of those documents using a vector database. The stated purpose is to reduce the amount of raw text that needs to be processed downstream, so that the cloud model receives only the user query after anonymization and the most relevant document chunks after anonymization, rather than entire contracts or broad text dumps (Singh et al., 24 Sep 2025).

The second stage is anonymization of sensitive entities through a structured one-to-many mapping scheme. GLiNER is applied to both the user query and the retrieved chunks. The paper denotes the detected entity sets by

$$\mathcal{E}_Q = \{e_i^Q \mid e_i^Q \in \mathcal{N}(Q),\ 1 \leq i \leq n_Q \},$$

$$\mathcal{E}_{d_j} = \{e_i^{d_j} \mid e_i^{d_j} \in \mathcal{N}(d_j),\ 1 \leq i \leq n_{d_j} \},$$

and the total sensitive-entity set by

$$\mathcal{E}_{\text{total}} = \mathcal{E}_Q \cup \left( \bigcup_{j=1}^{k} \mathcal{E}_{d_j} \right).$$

For each sensitive entity $e_i$, the system generates a surrogate set

$$\mathcal{S}_i = \{ s_{i1}, s_{i2}, \ldots, s_{iK} \},$$

with the full mapping

$$\mathcal{M} = \left\{ (e_i, \mathcal{S}_i) \,\middle|\, e_i \in \mathcal{E}_{\text{total}},\ |\mathcal{S}_i| = K,\ \text{sim}(e_i, s_{ij}) \geq \theta,\ \text{dist}(s_{im}, s_{in}) \geq \delta \right\}.$$

The paper interprets $\theta$ as enforcing semantic closeness to the original and $$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$0 as enforcing diversity among surrogates (Singh et al., 24 Sep 2025).

The third stage is cloud answer generation followed by local reconstruction. For each entity $$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$1, one surrogate is sampled uniformly from the candidate set:

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$2

and the session-specific mapping is discarded at session end:

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$3

The cloud model receives anonymized inputs and produces

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$4

The local side then applies the inverse mapping

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$5

to recover the final answer

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$6

and the reverse mapping is likewise discarded after session completion:

$$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$7

The paper characterizes this as anonymized response generation with accurate local reconstruction through a session-consistent many-to-one reverse mapping (Singh et al., 24 Sep 2025).

4. Privacy mechanism and threat model

The core privacy claim concerns resistance to cross-session entity inference attacks. A deterministic one-to-one pseudonymization scheme can become linkable across repeated interactions, because the same original entity is always mapped to the same surrogate. CON-QA instead uses one-to-many surrogate generation with session-specific randomized replacement. According to the paper, this mitigates cross-session inference by combining session-specific surrogate sampling, discarding the mapping after each session, maintaining high surrogate diversity, and minimizing surrogate reuse (Singh et al., 24 Sep 2025).

This privacy mechanism is tied to semantic preservation. The surrogates are not arbitrary placeholders; they are intended to preserve semantic class and contextual coherence so that the cloud model can still reason correctly about anonymized contracts. The paper’s claim is therefore not merely that identifiers are hidden, but that the cloud LLM can answer coherently because the anonymized entities preserve entity type, legal role, and contextual structure, after which the local system restores the original identifiers deterministically (Singh et al., 24 Sep 2025).

The empirical comparison against the Hide and Seek framework is reported in terms of anonymization quality. For the Hide model, the paper reports Coverage 76.15%, Reuse 46.71%, Unique Surrogates 59.12%, Linkability 46.71%, and Missed entities 31.38%. For CON-QA, the reported values are Coverage 99.11%, Reuse 1.34%, Unique Surrogates 98.76%, Linkability 1.23%, and Missed entities 0.89%. These values are presented as evidence that the one-to-many anonymization scheme nearly exhausts sensitive-entity coverage while sharply reducing surrogate reuse and linkability (Singh et al., 24 Sep 2025).

The reconstruction step is likewise evaluated against the Seek model. The reported deanonymization metrics are Entity Accuracy 89.00% and BERTScore 81.10% for Seek, versus Entity Accuracy 98.00% and BERTScore 83.00% for CON-QA. The paper interprets this difference as evidence that deterministic many-to-one reverse mapping is more reliable than a learned decoder for restoring original contractual entities (Singh et al., 24 Sep 2025).

A frequent misconception is that privacy is obtained simply by masking names. The reported framework is more restrictive and more operationally complex: masking is integrated with retrieval, cloud interaction is limited to anonymized query-chunk pairs, and the state required for reversal is explicitly session-bounded. At the same time, the paper does not claim that privacy is eliminated entirely; an evident limitation is that the cloud LLM still sees anonymized context, so the system reduces rather than abolishes privacy exposure (Singh et al., 24 Sep 2025).

5. CUAD-QA dataset and evaluation methodology

To evaluate the framework, the paper introduces CUAD-QA, a contract-domain corpus built over 510 real-world CUAD contract documents and comprising about 85,000 question-answer pairs. The pairs were generated with ChatGPT-4o-mini by prompting over individual pages from the contracts, yielding roughly 10 QA pairs per contract page on average. The dataset spans three query types: simple questions, complex questions, and summarization-style questions (Singh et al., 24 Sep 2025).

The simple questions are direct lookup questions from one clause; the paper gives “What is the effective date of this agreement?” with answer “January 1, 2023” as an example. The complex questions require multi-clause reasoning and interpretation; the example asks whether an exclusivity clause applies to all product categories in North America. The summarization-style questions require abstractive synthesis over multiple clauses; the example asks for a summary of the buyer’s obligations in the event of early termination. This suggests that CUAD-QA is intended to measure not only extractive contract QA but also clause fusion and legal summarization (Singh et al., 24 Sep 2025).

The reported experimental sample consists of 1,000 QA pairs from CUAD-QA. The evaluation uses utility and privacy metrics rather than only lexical match. The utility-oriented metrics are Private Entity Restoration Accuracy, Response Relevancy, Answer Correctness, and Faithfulness. The reported CON-QA scores are 0.9880 for Private Entity Restoration Accuracy, 0.9778 for Response Relevancy, 0.8810 for Answer Correctness, and 0.9886 for Faithfulness. The paper notes that Answer Correctness is slightly lower mainly because summarization answers are often longer and more elaborative than compact references (Singh et al., 24 Sep 2025).

The study also includes human evaluation. Two evaluator groups were used: eight NLP domain experts or data scientists, and two legal domain experts from academia and industry. The evaluation covered 30 QA sets. NLP experts rated five parameters on a five-point scale, while legal experts rated four domain-specific criteria. The reported overall accuracies are 85.83% for the NLP expert group and 91.66% for the legal expert group. These figures are presented to support the claim that automatic metrics understate quality, particularly for summarization-style legal answers where exact lexical agreement is too strict (Singh et al., 24 Sep 2025).

6. Performance characteristics, practical significance, and limitations

The reported quantitative results frame CON-QA as a system that preserves both privacy and utility. The entity restoration score of 0.9880 and faithfulness score of 0.9886 are used to argue that legal clause semantics remain intact across anonymization, cloud generation, and deanonymization. The comparison against Hide and Seek further positions the framework as strong on both privacy-centric and answer-reconstruction metrics (Singh et al., 24 Sep 2025).

The practical deployment pattern implied by the paper is straightforward: keep sensitive preprocessing local, expose only anonymized text to the cloud LLM, reconstruct outputs locally, and discard mappings after each session. The paper explicitly identifies legal departments, compliance teams, procurement, merger-and-acquisition due diligence, and regulated industries such as healthcare and finance as plausible application settings. This suggests that CON-QA is meant for enterprise document workflows in which confidential contracts must remain usable without being transmitted in raw form to third-party LLM services (Singh et al., 24 Sep 2025).

Several limitations are either explicit or directly implied. The framework depends on NER quality, so missed entities may remain exposed if GLiNER fails to detect them. The answer-correctness metric is sensitive to the mismatch between compact references and longer summarization answers. Session-based reconstruction requires the mapping to be maintained during the session and safely discarded afterward. The cloud model still sees anonymized context, so privacy is reduced rather than eliminated. The dataset itself is generated by GPT-4o-mini over real contracts, which means the QA pairs may reflect model-specific biases in question formulation and answer style (Singh et al., 24 Sep 2025).

A further source of ambiguity is terminological rather than methodological. “CON-QA” is not a unique label across the broader QA literature. “CONDAQA,” a negation-reasoning reading-comprehension benchmark, is described in its paper as “CONDAQA” or “CON-QA” (Ravichander et al., 2022). By contrast, the framework discussed here is specifically the privacy-preserving contract-domain system titled “CON-QA: Privacy-Preserving QA using cloud LLMs in Contract Domain” (Singh et al., 24 Sep 2025). The overlap in naming should not be mistaken for overlap in task definition.

7. Relation to adjacent QA research

Within the broader QA landscape, CON-QA belongs to a class of systems that adapt QA pipelines to domain-specific operational constraints rather than treating question answering as a purely end-to-end language modeling problem. COREQQA, for example, is a compliance-requirements understanding tool for legal documents that uses a two-stage retrieval-plus-answering pipeline over regulations such as GDPR and returns ranked passages with highlighted answers, but it is framed as support for requirements engineering rather than privacy-preserving cloud interaction (Abualhaija et al., 2022). CON-QA shares the legal-document setting and retrieval emphasis, but its defining concern is secure outsourcing of answer generation through anonymization and local reconstruction.

The framework is also distinct from query-rewriting systems such as CONQRR, which rewrites conversational questions into standalone queries for off-the-shelf retrievers in open-domain conversational QA (Wu et al., 2021). CONQRR optimizes retrieval compatibility under dialogue context, whereas CON-QA uses local query decomposition to identify contract metadata, parties, dates, and clause-related search terms before retrieval. The resemblance lies in the reliance on a preprocessing stage that reshapes the query to suit downstream retrieval, but the objectives are different: conversational context resolution in one case and privacy-aware contract retrieval in the other.

Similarly, CON-QA differs from QA-testing frameworks such as $$A' = \{ a'_i \mid a'_i \in A,\, \text{extracted via Qwen},\, 1 \leq i \leq m \}.$$8, which generate answer-targeted questions from context to test QA software, and from ambiguity or conflict benchmarks such as CondAmbigQA and NATCONFQA, which focus on condition-aware disambiguation and conflict-aware multi-answer reasoning rather than privacy preservation (Liu et al., 11 Nov 2025, Li et al., 3 Feb 2025, Nachshoni et al., 17 Aug 2025). These neighboring efforts indicate a broader shift in QA research toward structurally constrained evaluation and deployment. CON-QA occupies the privacy-and-governance end of that spectrum: it is less about improving generic answer accuracy and more about enabling contract-domain QA under enterprise confidentiality constraints (Singh et al., 24 Sep 2025).