---
title: Ask-to-Clarify Framework
url: https://www.emergentmind.com/topics/ask-to-clarify-framework
type: topic
---

# Ask-to-Clarify Framework

An ask-to-clarify framework is an interactive design for language or multimodal agents that do not immediately commit to an answer or action when the available information is insufficient, ambiguous, or misleading, but instead ask a targeted follow-up question, incorporate the reply, and then proceed. In the task-oriented information-seeking formulation of "Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues" [2305.13690], the central problem is that a system may be unable to provide an accurate and personalized answer because users cannot fully express complex information needs in a single request and because the system may lack or be uncertain about user profile information. Related work generalizes the same pattern to conversational recommendation [2105.04774], tool use [2503.01940], open-domain QA [2410.13788], visually grounded dialogue [2110.09779], VQA under context under-specification [2601.16400], embodied agents [2509.15061], and prompt-based LLM agents that must decide whether a request itself is underspecified [2606.19559]. Across these settings, clarification functions as an uncertainty-reduction step embedded in a larger decision process rather than as an isolated question-generation task.

## 1. Problem formulation and sources of underspecification

A recurring formulation represents the interaction as an initial request plus iterative clarification turns followed by a terminal output. In task-oriented information seeking, the conversation is written as
\[
R, U, T \mid Q_1, A_1, Q_2, A_2, \dots, Q_K, A_K \mid Y,
\]
where \(R\) is the initial user request, \(U\) the user profile, \(T\) task knowledge, \(Q_k\) and \(A_k\) the clarification questions and answers, and \(Y\) the final answer [2305.13690]. The associated objectives are
\[
P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})
\]
for next-question generation and
\[
P(Y \mid R, U, T, Q_1, A_1, \dots, Q_K, A_K)
\]
for answer prediction [2305.13690]. This decomposition makes clarification an explicit intermediate stage between initial request interpretation and final task completion.

The literature identifies several distinct sources of clarification need. In task-oriented dialogue, the two emphasized sources are under-specified information needs and missing user-profile information [2305.13690]. In conversational recommendation, the missing information is dynamic user preference, and the paper argues that asking at the level of attribute types or relations can be more natural and efficient than repeatedly asking about specific attribute values [2105.04774]. In tool use, missing user intent is operationalized as missing API parameters; AskToAct starts from a fully specified query \(q\) and tool invocation solution
\[
S = \{(f_i, P_i) \mid i = 1,\dots,n\},
\]
then removes selected parameters to construct an unspecified query \(q'\) while preserving exact supervision over the removed arguments [2503.01940]. In open-domain QA, the ambiguity is represented by multiple user interpretations \(u_1,\dots,u_k\), each with an expected answer \(y_i\), and a clarifying question is valuable only if it leads to better user-specific answers in future turns [2410.13788].

Other domains preserve the same structure while changing the latent variable that is missing. In visually grounded reference games, the system is uncertain over a target image \(y \in Y\), and clarification reduces that uncertainty through yes/no questions [2110.09779]. In VQA under context under-specification, the image-question pair \((I,q)\) is insufficient because the answer depends on an unobserved contextual variable \(c\); CoA therefore frames the control decision as
\[
u \in \{Answer,\ Clarify\}
\]
before either answering directly or asking one clarification question [2601.16400]. In embodied instruction following, the missing information is often referential, such as which fruit, which colored cup, or which block ordering is intended, and Ask-to-Clarify resolves that ambiguity through multi-turn dialogue before producing low-level robot actions [2509.15061]. In open-domain dialogue, underspecification also includes facetedness: a request may not be linguistically ambiguous yet still be too broad for a satisfying answer, motivating a separate User Request Understanding component that predicts whether clarification is needed [2109.05794].

## 2. Decision policies for when to ask

A central issue is not only whether a request is ambiguous in the abstract, but whether clarification is worth an additional turn. "Clarify When Necessary" [2311.09469] formalizes this as the problem of assigning each input a utility score \(u(x)\) that predicts how much clarification would improve performance. Given feasible outputs
\[
Y = \{y_i\}_{i=1}^{k},
\]
one intended output \(y^* \in Y\), and corresponding clarifying answers
\[
A = \{a_i\}_{i=1}^{k},
\]
the paper argues that useful clarification depends on uncertainty over user intents rather than generic uncertainty over model outputs [2311.09469]. Its INTENT-SIM procedure greedily generates a clarifying question, samples possible user answers, clusters semantically equivalent answers with an NLI model, and computes entropy over the resulting candidate-intent distribution. Under a fixed clarification budget, the system clarifies the highest-entropy examples; the paper reports that when only allowed to ask for clarification on 10% of examples, its system can double the performance gains over randomly selecting examples to clarify [2311.09469].

Several later systems separate the ask decision from the content-selection decision. KBQG in conversational recommendation explicitly reduces the policy action space to only two actions—ask a clarifying question or provide recommendations—and lets a separate knowledge-based preference module decide which relation is most worth clarifying next [2105.04774]. Its policy is
\[
\pi_{\theta}(a_t|s_t) = \sigma(W_1 \cdot \tanh(W_2 s_t+\mathbf{b}_1) + \mathbf{b}_2),
\]
with dialogue state
\[
s_t = b_t \oplus q_t \oplus c_t,
\]
where \(b_t\) is a latent belief state, \(q_t\) marks which relations have already been clarified, and \(c_t\) indicates the ratio of candidate items whose affinity score exceeds threshold \(M\) [2105.04774]. The same paper treats ask-more versus recommend-sooner as a discounted RL tradeoff, with damping factor \(\eta\) encouraging fewer conversation turns [2105.04774].

Other work uses explicit classifiers or routers. CoA trains a controller with cross-entropy,
\[
\mathcal{L}_{\text{ctrl}} = -\log \pi_{\text{ctrl}}(u^\ast \mid I,q),
\]
to choose between answer and clarify before any question is generated [2601.16400]. Ask-to-Clarify for embodied agents uses four signal tokens—`<AMBG>`, `<NOT_AMBG>`, `<ACT>`, and `<REJ>`—and a training-free signal detector that routes between dialogue and action execution according to the token at the end of the VLM output [2509.15061]. In selective QA, CLAM uses a prompted ambiguity classifier whose score is the log probability of the next token being `True` in a prompt ending with “This question is ambiguous:”; the threshold is tuned on a holdout set and the paper uses \(T=-0.3\) in the remaining experiments [2212.07769].

Prompt-based agent work has also made the ask trigger itself an uncertainty representation. "Uncertainty Decomposition for Clarification Seeking in LLM Agents" [2606.19559] argues that a single confidence scalar collapses together uncertainty about the next action and uncertainty about whether the goal is sufficiently specified. It therefore decomposes output into action confidence \(c_t\) and request uncertainty \(u_t\), where \(u_t\) estimates the degree to which the user’s goal is underspecified [2606.19559]. The decision rule is deterministic:
\[
u_t \ge \theta \;\Rightarrow\; a_t^\star \leftarrow \text{request},
\]
with \(\theta=0.5\) reported as a balanced default, though the paper also shows that \(\theta=0.25\) gives the highest mean clarification F1 on its clarification benchmarks averaged across five backbones [2606.19559]. By contrast, work on CoDraw finds that human clarification behavior is only weakly aligned with model uncertainty, and proposes a threshold policy
\[
\text{ask if } H(\text{size}) \ge \theta
\]
based on the model’s own uncertainty rather than imitation of human asking points [2402.06509]. This suggests that clarification policies optimized for downstream task success need not mirror human turn-taking preferences.

## 3. Choosing what to ask

Once the decision to clarify has been made, the framework must identify the missing variable or the most informative distinction among competing interpretations. The design space spans free-form generation, template-based content selection, and expected-outcome optimization.

In task-oriented information seeking, MAS2S is a BART-based multi-source seq2seq model with separate encodings of dialogue, user profile, and task knowledge [2305.13690]. The dialogue encoder outputs token-level dialogue embeddings \(S_d\), the user-profile encoder outputs \(S_u\), and task knowledge is represented by a single embedding \(s_t\) taken from the final \([cls]\) state [2305.13690]. Task knowledge is then used as the query in bilinear attention:
\[
A_{d} = \text{softmax}(\text{exp}(S_d^\mathsf{T} W_d s_t)), \qquad
A_{u} = \text{softmax}(\text{exp}(S_u^\mathsf{T} W_u s_t)),
\]
yielding knowledge-attended dialogue and profile vectors
\[
d = S_d^\mathsf{T} A_{d}, \qquad u = S_u^\mathsf{T} A_{u},
\]
which are fused into an “answer confidence embedding”
\[
c = \text{MLP}([d;u]).
\]
The decoder, initialized from \(c\), then generates either a clarification question or a final answer, with
\[
h_t = \text{BART}(w_{t-1}, h_{t-1}), \qquad
p_t = \text{softmax}(V W_v h_t + b_v)
\]
over vocabulary and candidate answers [2305.13690]. The paper interprets this embedding as indicating whether enough evidence has been gathered.

In KBQG, question generation is primarily content selection rather than neural surface realization. The system computes relation-attention weights
\[
\alpha_{r,u}=\frac{\exp(a(r,u))}{\sum_{r'\in R} \exp(a(r',u))}
\]
over knowledge-graph relations and sequentially picks an unused relation with highest weight; a hand-designed template then turns that relation into a generic question such as “What is your preference on the genre of the movie?” [2105.04774]. The important modeling choice is that the system asks at the relation level but stores the answer at the value level:
\[
\mathbf{b}_t=\sum_{t'=1}^{t}\mathbf{z}_{t'},
\]
where \(z_t\) are specific attribute values extracted from user responses [2105.04774].

Several papers optimize question utility more directly. ClarifyDelphi treats a good clarification question as one whose plausible answers most alter the downstream moral judgment [2212.10409]. For a generated question \(q\), an answer model simulates a weakener answer \(a_w\) and a strengthener answer \(a_s\), Delphi predicts judgment distributions \(P_{j_w}\) and \(P_{j_s}\), and the reward is the Jensen–Shannon divergence
\[
r(s, q, a_w, a_s) = \mathrm{JSD}(P_{j_w} \,\|\, P_{j_s}),
\]
so questions are rewarded when their plausible answers induce divergent moral judgments [2212.10409]. In visually grounded open-domain clarification, the system generates candidate yes/no questions by decomposing image captions into noun phrases and then selects the one with highest expected information gain:
\[
\arg\min_{q\in Q} \mathbb{E}_{p(y \mid x^{t})}\mathbb{E}_{p(a\mid q,y)}\left[-\ln P(y \mid x^{t}, q, a)\right].
\]
Here the best clarification is the one expected to most reduce residual uncertainty over the latent target image [2110.09779].

Future-turn modeling in open-domain QA uses a different criterion. A candidate first-turn clarifying question \(q\) is scored by rolling out simulated user responses \(a_i\) for each interpretation and averaging exact match over the corresponding final answers:
\[
\frac{1}{k}\sum_{i=1}^k \text{EM}(\hat y_i, y_i).
\]
Direct answers are scored the same way across all interpretations, and ties are broken in favor of the direct answer to avoid gratuitous clarification [2410.13788]. This suggests that “what to ask” can be trained from future conversational utility rather than from local next-turn preference.

## 4. State representation, memory, and self-correction

Ask-to-clarify systems depend on how unresolved information is represented and updated. Some architectures use explicit symbolic or semi-symbolic state; others use latent embeddings or prompt-encoded uncertainty explanations.

Visual Curiosity is an early example of a structured memory-centered design [1810.00912]. The agent maintains a scene graph \(G=(V,E)\) whose nodes store object locations \(\mathbf{l}_k\) and categorical distributions \(\mathbf{p}_k^a\) over attributes, with uncertainty measured by entropy
\[
e_k^a = Entropy(\mathbf{p}_k^a).
\]
These entropy values are explicit inputs to the recurrent question policy, which selects a target attribute and object to ask about and optionally a reference object to disambiguate the referent [1810.00912]. After a valid Oracle answer, the top-down update sets the relevant attribute distribution to a one-hot vector, turning dialogue into structured supervision for subsequent perception and further questioning [1810.00912]. A closely related interpretability view appears in “Interpreting Models by Allowing to Ask,” where an asking network recursively predicts, asks, receives a restricted answer from an oracle, and refines its output:
\[
\hat{y}_t = f(x, a_{t-1}, \hat{y}_{t-1}, q_1, q_2, \cdots, q_{t-1}), \qquad
q_t = g(x, \hat{y}_{t-1}, q_{t-1}), \qquad
a_t = h(q_t, y).
\]
In that framing, the question itself is an interpretable disclosure of what the model finds tricky to predict [1811.05106].

Task and agent settings often rely on latent state, but still separate what is known from what remains unresolved. AskToAct stores per-parameter records with fields such as `removed`, `original`, and `current`, effectively tracking whether each tool argument is specified, unspecified, clarified, or still missing [2503.01940]. The self-correcting version augments clarification dialogues with five explicit error types—Clearly Stated Intent Clarification, Imprecise Clarification, Irrelevant Clarification, Redundant Clarification, and Incomplete Clarification—and inserts erroneous spans delimited by `<SOE> ... <EOE>` together with corrective utterances [2503.01940]. Selective masking then trains the model only on non-error tokens, which the paper describes as enabling dynamic error detection during clarification interactions [2503.01940]. This suggests that clarification state should include not just slot completeness but also whether the interaction itself has gone off course.

Prompt-based agent work encodes state directly in the emitted trajectory. The proposed decomposition in [2606.19559] extends history from
\[
H_t^{\text{UAM}} = \{(o_i, r_i, a_i, c_i, e_i)\}_{i=1}^{t-1}
\]
to
\[
H_t^{\text{proposed}} = \{(o_i, r_i, u_i, x_i, a_i, c_i, e_i)\}_{i=1}^{t-1},
\]
adding request uncertainty \(u_i\) and a natural-language explanation \(x_i\) alongside action confidence \(c_i\) and its explanation \(e_i\) [2606.19559]. ClarQ-LLM takes a different route by representing provider-held information as a dependency-aware response tree: each node corresponds to information the provider may reveal, edges encode logical dependency and permissible order of revelation, and the provider agent only exposes task-critical Response I items when the seeker’s question licenses them [2409.06097]. This suggests that a clarification framework can be grounded either in explicit task trees or in decomposed uncertainty traces, provided that the system can remember which uncertainties have already been resolved and which remain active.

## 5. Data resources, benchmarks, and evaluation

Evaluation methods differ sharply depending on whether the goal is open-domain dialogue quality, task completion, or recovery of explicit missing variables. In task-oriented dialogue, ClariT extends ShARC by sampling 1–5 clarification turns from existing dialogues, rewriting those question–answer pairs into declarative profile sentences such as “I am a family farmer,” and removing the corresponding clarification turns from the dialogue context [2305.13690]. The resulting dataset contains 108,599 dialogues, 1,742 task knowledge items, 85,749 user profiles, and 260,924 turns, split into 76,019/10,860/21,720 dialogues for training, validation, and testing [2305.13690]. The paper reports a quality check over 100 sampled dialogues with satisfaction ratio 1.0 for fluency, usefulness, relevancy, and clarification, and approximate parity with original ShARC on naturalness [2305.13690]. MAS2S is evaluated with BLEU and ROUGE for clarification-question generation and Success for answer prediction; on ClariT it reaches Success 0.412, BLEU 0.315, and ROUGE 0.296, outperforming GAN-Utility, SOLOIST, and UBAR, with gains over UBAR of +0.015 Success, +0.033 BLEU, and +0.023 ROUGE [2305.13690].

Open-domain and interactive evaluation tends to move away from surface-form overlap. The single- and multi-turn dialogue corpus in [2109.05794] builds triples \((U,Q,A)\) from TREC Web Track topics, labels clarification need on a 1–4 scale, and evaluates clarification need prediction with Precision, Recall, F1, and MSE while evaluating question selection both by question relevance and by whether appending \(Q\) and \(A\) to the original request improves downstream document retrieval [2109.05794]. ClarQ-LLM goes further by embedding seeker agents in interactive bilingual English–Chinese tasks with a provider agent and reporting Success Rate, Average Query Discrepancy,
\[
AQD = \frac{1}{N} \sum_{i=1}^{N} (Q_{actual,i} - A_{provider,i}),
\]
and Average Query Length,
\[
AQL = \frac{1}{N} \sum_{i=1}^{N} L_{seeker,i},
\]
thereby penalizing both under-questioning and over-questioning [2409.06097]. In its main L2L setting on 260 test tasks, the best reported success rate is 0.605 for Llama3.1-405B-Instruct in Completion mode on English, while human seekers reach 0.80 in Chinese and 0.85 in English in H2L evaluation, leaving a substantial gap [2409.06097].

Benchmarks tailored to agentic clarification make the ask/no-ask decision itself measurable. AskBench transforms QA pairs into interactive scenarios with explicit rubrics \(\mathcal{K}=\{k_1,\dots,k_m\}\) and reports final-answer accuracy, checkpoint coverage, and redundant-questioning rate across AskMind and AskOverconfidence settings [2602.11199]. Its RLVR training uses turn-level rewards
\[
r_t =
\begin{cases}
-2.0, & a_t = 1,\\
-0.8, & a_t = 0 \text{ and } c_t = 0,\\
0.8, & 0 < c_t < m,\\
1.0, & c_t = m,
\end{cases}
\]
plus terminal rewards
\[
r_T =
\begin{cases}
1.0, & d_T = \text{correct},\\
-1.0, & d_T = \text{wrong},\\
-2.0, & d_T = \text{still asking},
\end{cases}
\]
to reward targeted clarification and penalize premature answers and useless questions [2602.11199]. In visual ask-or-answer evaluation, CoA introduces ContextClarify with ambiguous and non-ambiguous contrast instances and reports accuracy, precision, recall, and F1 for the controller, ambiguity-resolution reward and human 0/1/2 scores for clarification quality, and final VQA accuracy [2601.16400]. In tool use, AskToAct evaluates clarification with Intent Coverage Rate, Clarification Efficiency, Clarification Performance Score, and Interaction Rounds, and separately evaluates tool execution with Solution Completion Rate, Tool Selection Score, and Parameter Resolution Score [2503.01940]. This suggests that mature evaluation must separately measure whether the system asked when necessary, whether it asked about the right thing, whether it stopped at the right time, and whether the final downstream task succeeded.

## 6. Applications, limitations, and emerging design principles

Across domains, the most stable design principle is the separation of “when to ask” from “what to ask.” KBQG makes this decomposition explicit in conversational recommendation [2105.04774], CoA does so for VQA under context under-specification [2601.16400], and Ask-to-Clarify in embodied agents uses a VLM for collaboration plus a diffusion model for action, connected through a FiLM-based connection module and routed by explicit signal tokens [2509.15061]. This suggests that clarification is best treated as a policy layer over downstream task modules rather than as a side effect of generic response generation.

A second design principle is that clarification should be grounded in the structure of the downstream task. In tool use, missing API parameters are explicit operationalizations of user intent [2503.01940]. In task-oriented dialogue, task knowledge acts as the query that identifies which aspects of the dialogue and user profile matter most [2305.13690]. In ClarifyDelphi, the downstream judgment model defines which missing contexts are consequential [2212.10409]. In visually grounded disambiguation, expected information gain over candidate targets drives question selection [2110.09779]. A plausible implication is that clarification quality improves when the agent can represent what information is decision-critical, rather than only whether the input is lexically incomplete.

A third recurring lesson is that asking more is not the objective. MAS2S is closer than UBAR to the oracle number of clarification turns on ClariT, with average number of clarification questions 2.97 versus oracle 2.36 and absolute difference 0.94 versus 1.68 [2305.13690]. AskBench reports strong checkpoint coverage together with nontrivial redundant-questioning rates, showing that high coverage can coexist with inefficient over-asking [2602.11199]. ClarQ-LLM’s AQD and AQL expose the same issue at task level: strong models are more verbose and ask more than humans even when they succeed [2409.06097]. This suggests that an ask-to-clarify framework should optimize for necessary clarification, not maximal interaction.

The main limitations in the literature are also consistent. Many datasets use synthetic or transformed ambiguity: ClariT rewrites sampled clarification turns into user profiles [2305.13690], AskToAct removes or abstracts tool parameters [2503.01940], ContextClarify focuses on single-factor ambiguity [2601.16400], and the clarification-augmented benchmarks in [2606.19559] strip attributes, objects, or receptacles procedurally. Several systems rely on simulated users or oracles rather than real human clarifications [2212.07769], [2410.13788], [2503.01940]. Others solve only one half of the problem: [2411.05831] focuses on when to ask in navigation but not what to ask, while [2110.09779] focuses on what question would reduce uncertainty but remains restricted to visually grounded yes/no interaction. Prompt-based uncertainty methods are also reported to be systematically overconfident, with reliability diagrams below the diagonal and ECE for GPT-5.1 ranging from 0.24 to 0.66 depending on method and benchmark [2606.19559].

Taken together, the literature defines ask-to-clarify not as a single model class but as an architectural pattern: maintain an explicit or implicit representation of unresolved information; estimate whether that unresolved information is consequential enough to justify interaction; formulate a targeted question that reduces uncertainty over the latent intent, parameter, referent, or task state; update state from the reply; and terminate clarification only when downstream action or answering is sufficiently supported. MAS2S and ClariT [2305.13690] provide a concrete task-oriented formulation with dialogue, user profile, and task knowledge; later work extends the same logic to recommendation [2105.04774], QA [2410.13788], tool use [2503.01940], multimodal decision making [2601.16400], black-box LLM agents [2606.19559], and embodied collaboration [2509.15061]. The shared technical lesson is that clarification is most effective when driven by a representation of task-conditioned insufficiency rather than by generic confidence or imitation of superficial question-asking behavior.

Source: https://www.emergentmind.com/topics/ask-to-clarify-framework