---
title: 'AGENT-CQ: LLM-Driven Clarifying Questions'
url: https://www.emergentmind.com/topics/agent-cq
type: topic
---

# AGENT-CQ: LLM-Driven Clarifying Questions

AGENT-CQ (Automatic GENeration and evaluaTion of Clarifying Questions) is an end-to-end large language model (LLM) framework for scalable, high-quality clarifying question (CQ) generation and judgment in open-domain conversational search. AGENT-CQ introduces a dual-stage pipeline: it generates diverse, context-specific clarifying questions and simulates user answers using advanced prompting strategies, then evaluates generated artifacts with a novel multi-LLM crowd evaluator (“CrowdLLM”) designed to replace costly and inconsistent human annotation. Empirical results on TREC ClariQ demonstrate that AGENT-CQ-generated clarifications yield superior retrieval effectiveness compared to both template and human-curated approaches, and that CrowdLLM provides reliable, fine-grained quality assessment aligned with human judgments [2410.19692].

## 1. System Overview and Motivation

AGENT-CQ addresses two interlocking challenges in conversational search (CS): (1) scalable, adaptable CQ generation for ambiguous or underspecified queries, and (2) robust, objective evaluation of CQ and answer quality without human annotation bottlenecks. Traditional approaches rely on manual curation or template-based question generation, which are non-scalable and exhibit low diversity. LLM-based synthetic data methods have shown promise for generating dialogic content in other domains, but systematic, high-precision CQ generation and evaluation for CS remained unsolved prior to this work.

The AGENT-CQ architecture consists of:
- A **generation stage** that produces diverse clarifying questions and simulated user answers using LLM prompting (facet-based and diversity-optimized temperature scheduling).
- A **CrowdLLM evaluation stage** that employs multiple independent LLM judge instances as proxy crowd-workers, scoring CQs and answers on multidimensional Likert/ranking scales.

## 2. Generation Stage: Methods and Algorithms

The generation stage has three phases:

**2.1 Clarifying Question Generation**

For each query \(q\), AGENT-CQ generates a set \(\{C_{q,1}, \ldots, C_{q,m}\}\) of clarifying questions either through:
- **Facet-Based Generation**: The LLM first identifies ~40 latent topical “facets” \(F = \{f_1, ..., f_r\}\) for \(q\), then generates a specific CQ for each facet, e.g., “Which home remedies have you tried for angular cheilitis?” for the medical query “How to cure angular cheilitis?”
- **Temperature-Variation Generation**: The LLM is prompted at multiple temperatures \(T\) (e.g., 0.5, 0.6, ..., 0.9) to sample diverse candidate CQs per query, e.g., “Are you looking for over-the-counter creams or natural treatments?”

Pseudocode for facet-based CQ generation (Algorithm 1):
```python
F = LLM_generate_facets(q)  # ~40 facets
C = []
for f in F:
    C.append(LLM_generate_question(q, f))
return C
```

**2.2 Question Filtering**

Generated CQs are scored using a convex combination:
\[
\mathrm{Score}(q, cq) = \alpha R(q, cq) + (1-\alpha) L(q, cq),
\]
where \(R\) quantifies topical relevance and \(L\) the likelihood that the CQ elicits the missing intent. Top-10 CQs per query are retained (\(\alpha = 0.4\)).

**2.3 User Answer Simulation**

For each (\(q\), CQ) pair, the system simulates a user answer by prompting the LLM with explicit “verbosity” and “reveal probability” parameters. These govern answer length and signal completeness.

Pseudocode for answer simulation (Algorithm 3):
```python
A = []
for cq in C:
    verbosity, reveal_prob = sample_user_params()
    prompt = build_prompt(q, cq, verbosity, reveal_prob)
    a = LLM_generate_answer(prompt)
    A.append(a)
return A
```

## 3. Evaluation Stage: CrowdLLM and Quality Measurement

**3.1 Automated Evaluation Protocol**

CrowdLLM instantiates three independent GPT-4o (or equivalent) LLM judge agents per artifact, each at different temperatures (0.2, 0.5, 0.7), to simulate a proxy crowd. Each judge scores:
- CQs: 7 metrics (clarification potential, relevance, specificity, usefulness, clarity, complexity, overall quality)
- Answers: 4 metrics (relevance, usefulness, naturalness, overall quality)

Aggregated scores:
\[
\mathrm{Score}(x) = \frac{1}{|M|} \sum_{m \in M} s_m(x),
\]
where \(M\) is the set of judges.

**3.2 Validation of Evaluation**

Inter-rater reliability is assessed via intraclass correlation (ICC), Cohen’s \(\kappa\), and Fleiss’ K for pairwise judgments. Human-LLM correlation is measured via Spearman’s \(\rho\) and Kendall’s \(\tau\). Statistical significance is tested via ANOVA/Tukey for CQs and trinomial test for answers.

## 4. Experimental Setup and Dataset

- **Dataset:** ClariQ (198 queries from TREC Web Track 2009–2012; 891 human facets; >8,000 human CQs).
- **Retrieval Backbone:** Pyserini BM25; Sentence-BERT cross-encoder re-ranker.
- **Pipeline:** For each query: generate top-1 CQ, simulate answer, form expanded query (\(\tilde{q}=q + a^*\)), retrieve/rerank documents.
- **All 198 queries** used for end-to-end experiments—no train/dev/test split required due to LLM-only generation.

## 5. Empirical Results and Observations

**Retrieval Effectiveness:**

| Backbone | Condition                 | nDCG@1 | nDCG@5 | nDCG@10 |
|----------|---------------------------|--------|--------|---------|
| BM25     | Baseline                  | 0.180  | 0.187  | 0.209   |
| BM25     | Human CQ+Ans              | 0.201  | 0.221  | 0.246   |
| BM25     | GPT-Baseline              | 0.173  | 0.193  | 0.215   |
| BM25     | GPT-Temp (AGENT-CQ)       | 0.225  | 0.199  | 0.214   |
| CrossEnc | Baseline                  | 0.283  | 0.294  | 0.303   |
| CrossEnc | Human CQ+Ans              | 0.307  | 0.288  | 0.301   |
| CrossEnc | GPT-Baseline              | 0.267  | 0.259  | 0.277   |
| CrossEnc | GPT-Temp (AGENT-CQ)       | 0.312  | 0.296  | 0.301   |

- GPT-Temp (diverse temperature) generation yields statistically significant (\(p < 0.05\)) nDCG@1 improvements over the BM25 and cross-encoder baselines and matches or exceeds human CQ+answer with neural reranking.
- Facet-based generation results in higher specificity and complexity; temperature-variation favors overall diversity and effectiveness.
- AGENT-CQ-generated answers are rated by CrowdLLM as matching or outperforming human answers in overall quality.

## 6. Insights, Limitations, and Further Work

- **Quality Scaling:** The LLM-driven approach achieves higher or competitive retrieval effectiveness compared to manual curation and surpasses template-based methods in quality and diversity metrics.
- **Robustness:** Multi-temperature sampling ensures diversity; crowd LLMs provide multidimensional evaluations not restricted by human annotation cost or variance.
- **Limitations:** The ClariQ dataset does not capture multi-turn clarification, and “LLM-as-judge” risks biasing evaluation toward LLM-generated content (narcissistic evaluation). Prompt and model selection significantly affect outcomes.
- **Extensions:** Prospective work includes developing domain-fine-tuned models, bias calibration in evaluation, support for multi-turn dialogue pipelines, and hybrid retrieval architectures that combine AGENT-CQ clarifications with traditional IR data [2410.19692].

Source: https://www.emergentmind.com/topics/agent-cq