---
title: ClariQ Dataset for Clarification Research
url: https://www.emergentmind.com/topics/clariq-dataset
type: topic
---

# ClariQ Dataset for Clarification Research

ClariQ is a benchmark dataset for research on proactive clarification in open-domain conversational information retrieval and search. It provides richly annotated dialogues and supports empirical evaluation of both clarification-need detection and clarification-question selection, fostering the development of systems that can diagnose and resolve user ambiguity through effective question-asking. ClariQ extends earlier TREC-based datasets by applying rigorous human annotation and crowdsourcing pipelines, achieving medium scale and high facet coverage while enabling comparisons with classic and neural baselines [2305.15933], [2009.11352], [2109.05794], [2410.19692].

## 1. Dataset Genesis, Construction, and Annotation

ClariQ was constructed to overcome limitations in prior clarification datasets, specifically narrow topical scope, inconsistent splits, and under-specified annotation criteria. Its sources are the 300 topics from the TREC Web Track (2009–2014), selected for their mix of ambiguous and faceted information needs [2305.15933], [2109.05794].

Annotation followed a strict pipeline:
- Each topic was rewritten as a natural conversational request.
- For every "query → facet" pair (facet = sub-topic), one or more clarification questions (CQs) were authored by crowd workers.
- Each CQ was paired with a definitive answer resolving that facet.
- The schema formalizes every example as a triple $(q, cq, a)$, where $q$ is the user’s request, $cq$ is a single clarification question, and $a$ is the answer fully resolving the CQ.
- Quality guidelines for CQs required grammatical correctness, clarity, and single-facet focus; answers had to be precise and facet-grounded.

Quality control involved expert review, redundancy removal, and mapping of questions to facets. Clarity scoring (1–4 scale) was introduced during annotation, with substantial inter-annotator agreement ($\kappa = 0.78$) [2109.05794].

## 2. Quantitative Properties, Example Formats, and Splits

ClariQ’s scale and structure are summarized as follows:

| Statistic                   | Single-turn (Human)  | Multi-turn (Synthetic) | Clarifying Questions |
|-----------------------------|----------------------|------------------------|----------------------|
| Dialogues / Triplets        | 18,277               | ~2,000,000             | ~4,000 unique human  |
| Avg. Turns per Dialogue     | 1                    | 3–4                    |                      |
| Total Dialogue Turns        | 18,277               | >5,000,000             |                      |

- No official train/validation/test split is prescribed; a common practice is to randomly split the verified single-turn portion 80%/10%/10% (train/val/test), yielding approximately 14,600/1,800/1,800 examples.
- Each single-turn sample follows the format: 
  - User query $q$
  - System (CQ) $cq$
  - User answer $a$
- Example:
  - $q$: "Things to do in London?"  
    $cq$: "Are you looking for sights, restaurants, or shows?"  
    $a$: "I’m mostly interested in historical landmarks."  
[2305.15933], [2109.05794].

## 3. Supported Tasks and Benchmark Results

ClariQ supports two canonical tasks, aligned with the open research problems of knowing when to clarify and what to ask [2305.15933], [2410.19692]:

1. **Clarification-Need Prediction** (Binary or graded classification: does the user request require clarification?):
   - Baseline models include RandomForest (Precision = 0.3540, Recall = 0.3717, $F_1 = 0.3249$) and BERT (Precision = 0.3806, Recall = 0.3804, $F_1 = 0.3344$) on the binary classification task [2305.15933].
   - Systems also predict a 1–4 clarification-need score as in the ConvAI3 challenge [2009.11352].

2. **Clarification Question Selection** (Ranking/selection of the most helpful CQ from a candidate pool):
   - BM25 achieves Mean Average Precision (MAP) = 0.6360, nDCG = 0.7211; Doc2Query + BM25 boosts MAP to 0.6705, nDCG to 0.7501 [2305.15933].
   - Neural rankers (e.g., BERT/Hybrid) further improve relevance, though ClariQ's controlled candidate pool simplifies the task relative to looser datasets [2109.05794].

Key evaluation metrics include:
- Precision, Recall, $F_1$ (for classification)
- MAP, MRR, nDCG@k, P@k, R@k (for ranking/selection)
Formulas are provided in the cited works [2305.15933], [2109.05794], [2410.19692].

## 4. Comparative Analysis With Other Datasets

ClariQ sits at an intermediate point in the search clarification dataset landscape:

| Dataset      | Dialogues   | Unique CQs | Topics      | Annotation Detail                   |
|--------------|-------------|------------|-------------|-------------------------------------|
| ClariQ       | ~2M         | ~4K        | 300 TREC    | Facet-driven, crowd-verified        |
| Qulac        | 10K         | 3K         | 198 TREC    | Facet-grounded, closely related     |
| TavakoliCQ   | 170K        | 7K         | 3 SE sites  | User-intent tagged per turn         |
| MIMICS       | >460K       | N/A        | Bing logs   | Click/engagement/graded QA labels   |

- ClariQ and Qulac share the most annotation schema and source overlap.
- TavakoliCQ and MANtIS annotate broader user intent but with looser topical coverage.
- MIMICS emphasizes production-scale search log signal and is the largest in scale [2305.15933], [2006.10174].
- BM25 ranks higher on ClariQ than on more diverse datasets (MAP $\approx 0.64–0.67$ on ClariQ).

A *plausible implication* is that models validated solely on ClariQ may overestimate their generalizability relative to settings with larger lexical or domain diversity.

## 5. Downstream Impact and Utilization

ClariQ data and benchmarks have enabled several lines of research:
- Neural QG and ranking architectures for proactively resolving user ambiguity [2109.05794]
- Offline evaluation protocols for single- and multi-turn clarification
- Mixed-initiative conversational agents that integrate clarification-need detection with CQ retrieval or generation pipelines
- Innovations in automatic CQ evaluation via LLMs, e.g., AGENT-CQ, which demonstrates improvements in question quality and retrieval effectiveness when leveraging LLM-generated clarifications on ClariQ [2410.19692]

A consistent finding across retrieval studies is that supplementing the original user query with a clarifying question and its answer yields measurable improvements in MRR and nDCG for document ranking tasks [2109.05794], [2410.19692].

## 6. Current Limitations and Open Challenges

Known limitations highlighted in the literature include:
- Heavily TREC-centric source and high overlap with Qulac, limiting topical novelty [2305.15933].
- Synthetic multi-turn dialogues, while large, may not fully capture natural conversational dynamics.
- The fixed candidate question pool makes the ranking task easier than in more open-domain, log-based datasets (e.g., MIMICS).
- No official data splits; common splits risk non-comparability across studies.
- Precision-recall tradeoffs in answer annotation (high precision, moderate recall) and single-facet focus [2410.19692].

Major future challenges are:
- Modeling cross-turn reference and anaphora in multi-turn clarifications [2109.05794].
- Enhancing generalization to open-domain or unseen facets (as tested in TavakoliCQ/MIMICS).
- Incorporating explicit user satisfaction and engagement signals for end-to-end, user-aligned systems.

## 7. Availability and Recommended Usage

The full ClariQ dataset—including single-turn and synthetic multi-turn conversations, gold question banks, and evaluation scripts—is publicly released under an open research license at [https://github.com/aliannejadi/ClariQ] [2009.11352]. Recommended usage includes:
- Training and benchmarking models for both clarification-need detection and clarification question selection.
- Comparative evaluation with neural rankers, generative models, or user-interactive clarification agents.
- As a controlled, facet-grounded validation set before moving to more diverse, noisy, or user-interaction-driven datasets.


---

**Key references**:  
- "A Survey on Asking Clarification Questions Datasets in Conversational Systems" [2305.15933]  
- "ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)" [2009.11352]  
- "Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions" [2109.05794]  
- "AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs" [2410.19692]

Source: https://www.emergentmind.com/topics/clariq-dataset