AGENT-CQ: LLM-Driven Clarifying Questions

• AGENT-CQ is an end-to-end framework that generates diverse, context-specific clarifying questions and simulated user answers using dual-stage LLM prompting.
• It employs facet-based and temperature-variation strategies to enhance specificity and diversity in question generation for ambiguous, underspecified queries.
• A novel multi-LLM evaluation stage (CrowdLLM) reliably assesses CQ quality, yielding superior retrieval effectiveness compared to template and human-curated approaches.

AGENT-CQ (Automatic GENeration and evaluaTion of Clarifying Questions) is an end-to-end LLM framework for scalable, high-quality clarifying question (CQ) generation and judgment in open-domain conversational search. AGENT-CQ introduces a dual-stage pipeline: it generates diverse, context-specific clarifying questions and simulates user answers using advanced prompting strategies, then evaluates generated artifacts with a novel multi-LLM crowd evaluator (“CrowdLLM”) designed to replace costly and inconsistent human annotation. Empirical results on TREC ClariQ demonstrate that AGENT-CQ-generated clarifications yield superior retrieval effectiveness compared to both template and human-curated approaches, and that CrowdLLM provides reliable, fine-grained quality assessment aligned with human judgments (Siro et al., 2024).

1. System Overview and Motivation

AGENT-CQ addresses two interlocking challenges in conversational search (CS): (1) scalable, adaptable CQ generation for ambiguous or underspecified queries, and (2) robust, objective evaluation of CQ and answer quality without human annotation bottlenecks. Traditional approaches rely on manual curation or template-based question generation, which are non-scalable and exhibit low diversity. LLM-based synthetic data methods have shown promise for generating dialogic content in other domains, but systematic, high-precision CQ generation and evaluation for CS remained unsolved prior to this work.

The AGENT-CQ architecture consists of:

• A generation stage that produces diverse clarifying questions and simulated user answers using LLM prompting (facet-based and diversity-optimized temperature scheduling).
• A CrowdLLM evaluation stage that employs multiple independent LLM judge instances as proxy crowd-workers, scoring CQs and answers on multidimensional Likert/ranking scales.

2. Generation Stage: Methods and Algorithms

The generation stage has three phases:

2.1 Clarifying Question Generation

For each query $q$, AGENT-CQ generates a set $\{C_{q,1}, \ldots, C_{q,m}\}$ of clarifying questions either through:

• Facet-Based Generation: The LLM first identifies ~40 latent topical “facets” $F = \{f_1, ..., f_r\}$ for $q$, then generates a specific CQ for each facet, e.g., “Which home remedies have you tried for angular cheilitis?” for the medical query “How to cure angular cheilitis?”
• Temperature-Variation Generation: The LLM is prompted at multiple temperatures $T$ (e.g., 0.5, 0.6, ..., 0.9) to sample diverse candidate CQs per query, e.g., “Are you looking for over-the-counter creams or natural treatments?”

Pseudocode for facet-based CQ generation (Algorithm 1): $\{C_{q,1}, \ldots, C_{q,m}\}$7

2.2 Question Filtering

Generated CQs are scored using a convex combination: $$\mathrm{Score}(q, cq) = \alpha R(q, cq) + (1-\alpha) L(q, cq),$$ where $R$ quantifies topical relevance and $L$ the likelihood that the CQ elicits the missing intent. Top-10 CQs per query are retained ($\alpha = 0.4$).

2.3 User Answer Simulation

For each ($q$, CQ) pair, the system simulates a user answer by prompting the LLM with explicit “verbosity” and “reveal probability” parameters. These govern answer length and signal completeness.

Pseudocode for answer simulation (Algorithm 3): $\{C_{q,1}, \ldots, C_{q,m}\}$8

3. Evaluation Stage: CrowdLLM and Quality Measurement

3.1 Automated Evaluation Protocol

CrowdLLM instantiates three independent GPT-4o (or equivalent) LLM judge agents per artifact, each at different temperatures (0.2, 0.5, 0.7), to simulate a proxy crowd. Each judge scores:

• CQs: 7 metrics (clarification potential, relevance, specificity, usefulness, clarity, complexity, overall quality)
• Answers: 4 metrics (relevance, usefulness, naturalness, overall quality)

Aggregated scores: $\{C_{q,1}, \ldots, C_{q,m}\}$0 where $\{C_{q,1}, \ldots, C_{q,m}\}$1 is the set of judges.

3.2 Validation of Evaluation

Inter-rater reliability is assessed via intraclass correlation (ICC), Cohen’s $\{C_{q,1}, \ldots, C_{q,m}\}$2, and Fleiss’ K for pairwise judgments. Human-LLM correlation is measured via Spearman’s $\{C_{q,1}, \ldots, C_{q,m}\}$3 and Kendall’s $\{C_{q,1}, \ldots, C_{q,m}\}$4. Statistical significance is tested via ANOVA/Tukey for CQs and trinomial test for answers.

4. Experimental Setup and Dataset

• Dataset: ClariQ (198 queries from TREC Web Track 2009–2012; 891 human facets; >8,000 human CQs).
• Retrieval Backbone: Pyserini BM25; Sentence-BERT cross-encoder re-ranker.
• Pipeline: For each query: generate top-1 CQ, simulate answer, form expanded query ($\{C_{q,1}, \ldots, C_{q,m}\}$5), retrieve/rerank documents.
• All 198 queries used for end-to-end experiments—no train/dev/test split required due to LLM-only generation.

5. Empirical Results and Observations

Retrieval Effectiveness:

Backbone	Condition	nDCG@1	nDCG@5	nDCG@10
BM25	Baseline	0.180	0.187	0.209
BM25	Human CQ+Ans	0.201	0.221	0.246
BM25	GPT-Baseline	0.173	0.193	0.215
BM25	GPT-Temp (AGENT-CQ)	0.225	0.199	0.214
CrossEnc	Baseline	0.283	0.294	0.303
CrossEnc	Human CQ+Ans	0.307	0.288	0.301
CrossEnc	GPT-Baseline	0.267	0.259	0.277
CrossEnc	GPT-Temp (AGENT-CQ)	0.312	0.296	0.301

• GPT-Temp (diverse temperature) generation yields statistically significant ($\{C_{q,1}, \ldots, C_{q,m}\}$6) nDCG@1 improvements over the BM25 and cross-encoder baselines and matches or exceeds human CQ+answer with neural reranking.
• Facet-based generation results in higher specificity and complexity; temperature-variation favors overall diversity and effectiveness.
• AGENT-CQ-generated answers are rated by CrowdLLM as matching or outperforming human answers in overall quality.

6. Insights, Limitations, and Further Work

• Quality Scaling: The LLM-driven approach achieves higher or competitive retrieval effectiveness compared to manual curation and surpasses template-based methods in quality and diversity metrics.
• Robustness: Multi-temperature sampling ensures diversity; crowd LLMs provide multidimensional evaluations not restricted by human annotation cost or variance.
• Limitations: The ClariQ dataset does not capture multi-turn clarification, and “LLM-as-judge” risks biasing evaluation toward LLM-generated content (narcissistic evaluation). Prompt and model selection significantly affect outcomes.
• Extensions: Prospective work includes developing domain-fine-tuned models, bias calibration in evaluation, support for multi-turn dialogue pipelines, and hybrid retrieval architectures that combine AGENT-CQ clarifications with traditional IR data (Siro et al., 2024).