ClariQ Dataset for Clarification Research

• ClariQ is a benchmark dataset for proactive clarification that offers richly annotated dialogues to resolve user ambiguities in conversational search.
• It standardizes query-to-clarification pairs through a rigorous human annotation and crowdsourcing pipeline, ensuring high facet coverage and quality.
• The dataset supports key tasks like clarification-need prediction and question selection, enabling evaluation with both classical and neural ranking models.

ClariQ is a benchmark dataset for research on proactive clarification in open-domain conversational information retrieval and search. It provides richly annotated dialogues and supports empirical evaluation of both clarification-need detection and clarification-question selection, fostering the development of systems that can diagnose and resolve user ambiguity through effective question-asking. ClariQ extends earlier TREC-based datasets by applying rigorous human annotation and crowdsourcing pipelines, achieving medium scale and high facet coverage while enabling comparisons with classic and neural baselines (Rahmani et al., 2023, Aliannejadi et al., 2020, Aliannejadi et al., 2021, Siro et al., 2024).

1. Dataset Genesis, Construction, and Annotation

ClariQ was constructed to overcome limitations in prior clarification datasets, specifically narrow topical scope, inconsistent splits, and under-specified annotation criteria. Its sources are the 300 topics from the TREC Web Track (2009–2014), selected for their mix of ambiguous and faceted information needs (Rahmani et al., 2023, Aliannejadi et al., 2021).

Annotation followed a strict pipeline:

• Each topic was rewritten as a natural conversational request.
• For every "query → facet" pair (facet = sub-topic), one or more clarification questions (CQs) were authored by crowd workers.
• Each CQ was paired with a definitive answer resolving that facet.
• The schema formalizes every example as a triple $(q, cq, a)$, where $q$ is the user’s request, $cq$ is a single clarification question, and $a$ is the answer fully resolving the CQ.
• Quality guidelines for CQs required grammatical correctness, clarity, and single-facet focus; answers had to be precise and facet-grounded.

Quality control involved expert review, redundancy removal, and mapping of questions to facets. Clarity scoring (1–4 scale) was introduced during annotation, with substantial inter-annotator agreement ($\kappa = 0.78$) (Aliannejadi et al., 2021).

2. Quantitative Properties, Example Formats, and Splits

ClariQ’s scale and structure are summarized as follows:

Statistic	Single-turn (Human)	Multi-turn (Synthetic)	Clarifying Questions
Dialogues / Triplets	18,277	~2,000,000	~4,000 unique human
Avg. Turns per Dialogue	1	3–4
Total Dialogue Turns	18,277	>5,000,000

• No official train/validation/test split is prescribed; a common practice is to randomly split the verified single-turn portion 80%/10%/10% (train/val/test), yielding approximately 14,600/1,800/1,800 examples.
• Each single-turn sample follows the format:
  • User query $q$
  • System (CQ) $cq$
  • User answer $a$
• Example:
  • $q$: "Things to do in London?"
  • $cq$: "Are you looking for sights, restaurants, or shows?"
  • $q$0: "I’m mostly interested in historical landmarks."
  • (Rahmani et al., 2023, Aliannejadi et al., 2021).

3. Supported Tasks and Benchmark Results

ClariQ supports two canonical tasks, aligned with the open research problems of knowing when to clarify and what to ask (Rahmani et al., 2023, Siro et al., 2024):

1. Clarification-Need Prediction (Binary or graded classification: does the user request require clarification?):
   • Baseline models include RandomForest (Precision = 0.3540, Recall = 0.3717, $q$1) and BERT (Precision = 0.3806, Recall = 0.3804, $q$2) on the binary classification task (Rahmani et al., 2023).
   • Systems also predict a 1–4 clarification-need score as in the ConvAI3 challenge (Aliannejadi et al., 2020).
2. Clarification Question Selection (Ranking/selection of the most helpful CQ from a candidate pool):
   • BM25 achieves Mean Average Precision (MAP) = 0.6360, nDCG = 0.7211; Doc2Query + BM25 boosts MAP to 0.6705, nDCG to 0.7501 (Rahmani et al., 2023).
   • Neural rankers (e.g., BERT/Hybrid) further improve relevance, though ClariQ's controlled candidate pool simplifies the task relative to looser datasets (Aliannejadi et al., 2021).

Key evaluation metrics include:

• Precision, Recall, $q$3 (for classification)
• MAP, MRR, nDCG@k, P@k, R@k (for ranking/selection) Formulas are provided in the cited works (Rahmani et al., 2023, Aliannejadi et al., 2021, Siro et al., 2024).

4. Comparative Analysis With Other Datasets

ClariQ sits at an intermediate point in the search clarification dataset landscape:

Dataset	Dialogues	Unique CQs	Topics	Annotation Detail
ClariQ	~2M	~4K	300 TREC	Facet-driven, crowd-verified
Qulac	10K	3K	198 TREC	Facet-grounded, closely related
TavakoliCQ	170K	7K	3 SE sites	User-intent tagged per turn
MIMICS	>460K	N/A	Bing logs	Click/engagement/graded QA labels

• ClariQ and Qulac share the most annotation schema and source overlap.
• TavakoliCQ and MANtIS annotate broader user intent but with looser topical coverage.
• MIMICS emphasizes production-scale search log signal and is the largest in scale (Rahmani et al., 2023, Zamani et al., 2020).
• BM25 ranks higher on ClariQ than on more diverse datasets (MAP $q$4 on ClariQ).

A plausible implication is that models validated solely on ClariQ may overestimate their generalizability relative to settings with larger lexical or domain diversity.

5. Downstream Impact and Utilization

ClariQ data and benchmarks have enabled several lines of research:

• Neural QG and ranking architectures for proactively resolving user ambiguity (Aliannejadi et al., 2021)
• Offline evaluation protocols for single- and multi-turn clarification
• Mixed-initiative conversational agents that integrate clarification-need detection with CQ retrieval or generation pipelines
• Innovations in automatic CQ evaluation via LLMs, e.g., AGENT-CQ, which demonstrates improvements in question quality and retrieval effectiveness when leveraging LLM-generated clarifications on ClariQ (Siro et al., 2024)

A consistent finding across retrieval studies is that supplementing the original user query with a clarifying question and its answer yields measurable improvements in MRR and nDCG for document ranking tasks (Aliannejadi et al., 2021, Siro et al., 2024).

6. Current Limitations and Open Challenges

Known limitations highlighted in the literature include:

• Heavily TREC-centric source and high overlap with Qulac, limiting topical novelty (Rahmani et al., 2023).
• Synthetic multi-turn dialogues, while large, may not fully capture natural conversational dynamics.
• The fixed candidate question pool makes the ranking task easier than in more open-domain, log-based datasets (e.g., MIMICS).
• No official data splits; common splits risk non-comparability across studies.
• Precision-recall tradeoffs in answer annotation (high precision, moderate recall) and single-facet focus (Siro et al., 2024).

Major future challenges are:

• Modeling cross-turn reference and anaphora in multi-turn clarifications (Aliannejadi et al., 2021).
• Enhancing generalization to open-domain or unseen facets (as tested in TavakoliCQ/MIMICS).
• Incorporating explicit user satisfaction and engagement signals for end-to-end, user-aligned systems.

7. Availability and Recommended Usage

The full ClariQ dataset—including single-turn and synthetic multi-turn conversations, gold question banks, and evaluation scripts—is publicly released under an open research license at https://github.com/aliannejadi/ClariQ. Recommended usage includes:

• Training and benchmarking models for both clarification-need detection and clarification question selection.
• Comparative evaluation with neural rankers, generative models, or user-interactive clarification agents.
• As a controlled, facet-grounded validation set before moving to more diverse, noisy, or user-interaction-driven datasets.

---

Key references:

• "A Survey on Asking Clarification Questions Datasets in Conversational Systems" (Rahmani et al., 2023)
• "ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)" (Aliannejadi et al., 2020)
• "Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions" (Aliannejadi et al., 2021)
• "AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs" (Siro et al., 2024)