Ask-to-Clarify Framework

• Ask-to-Clarify Framework is an interactive model that defers immediate responses by asking targeted follow-up questions when user inputs are insufficient or ambiguous.
• It employs uncertainty quantification and dialog state updates to optimally decide when a clarification turn will improve task accuracy and decision-making.
• The framework is applied across task-oriented dialogues, conversational recommendations, and tool use, with evaluations focusing on efficiency, reduced ambiguity, and improved downstream performance.

An ask-to-clarify framework is an interactive design for language or multimodal agents that do not immediately commit to an answer or action when the available information is insufficient, ambiguous, or misleading, but instead ask a targeted follow-up question, incorporate the reply, and then proceed. In the task-oriented information-seeking formulation of "Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues" (Feng et al., 2023), the central problem is that a system may be unable to provide an accurate and personalized answer because users cannot fully express complex information needs in a single request and because the system may lack or be uncertain about user profile information. Related work generalizes the same pattern to conversational recommendation (Ren et al., 2021), tool use (Zhang et al., 3 Mar 2025), open-domain QA (Zhang et al., 2024), visually grounded dialogue (White et al., 2021), VQA under context under-specification (Cao et al., 23 Jan 2026), embodied agents (Lin et al., 18 Sep 2025), and prompt-based LLM agents that must decide whether a request itself is underspecified (Matsnev, 17 Jun 2026). Across these settings, clarification functions as an uncertainty-reduction step embedded in a larger decision process rather than as an isolated question-generation task.

1. Problem formulation and sources of underspecification

A recurring formulation represents the interaction as an initial request plus iterative clarification turns followed by a terminal output. In task-oriented information seeking, the conversation is written as

$$R, U, T \mid Q_1, A_1, Q_2, A_2, \dots, Q_K, A_K \mid Y,$$

where $R$ is the initial user request, $U$ the user profile, $T$ task knowledge, $Q_k$ and $A_k$ the clarification questions and answers, and $Y$ the final answer (Feng et al., 2023). The associated objectives are

$$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$

for next-question generation and

$P(Y \mid R, U, T, Q_1, A_1, \dots, Q_K, A_K)$

for answer prediction (Feng et al., 2023). This decomposition makes clarification an explicit intermediate stage between initial request interpretation and final task completion.

The literature identifies several distinct sources of clarification need. In task-oriented dialogue, the two emphasized sources are under-specified information needs and missing user-profile information (Feng et al., 2023). In conversational recommendation, the missing information is dynamic user preference, and the paper argues that asking at the level of attribute types or relations can be more natural and efficient than repeatedly asking about specific attribute values (Ren et al., 2021). In tool use, missing user intent is operationalized as missing API parameters; AskToAct starts from a fully specified query $q$ and tool invocation solution

$R$0

then removes selected parameters to construct an unspecified query $R$1 while preserving exact supervision over the removed arguments (Zhang et al., 3 Mar 2025). In open-domain QA, the ambiguity is represented by multiple user interpretations $R$2, each with an expected answer $R$3, and a clarifying question is valuable only if it leads to better user-specific answers in future turns (Zhang et al., 2024).

Other domains preserve the same structure while changing the latent variable that is missing. In visually grounded reference games, the system is uncertain over a target image $R$4, and clarification reduces that uncertainty through yes/no questions (White et al., 2021). In VQA under context under-specification, the image-question pair $R$5 is insufficient because the answer depends on an unobserved contextual variable $R$6; CoA therefore frames the control decision as

$R$7

before either answering directly or asking one clarification question (Cao et al., 23 Jan 2026). In embodied instruction following, the missing information is often referential, such as which fruit, which colored cup, or which block ordering is intended, and Ask-to-Clarify resolves that ambiguity through multi-turn dialogue before producing low-level robot actions (Lin et al., 18 Sep 2025). In open-domain dialogue, underspecification also includes facetedness: a request may not be linguistically ambiguous yet still be too broad for a satisfying answer, motivating a separate User Request Understanding component that predicts whether clarification is needed (Aliannejadi et al., 2021).

2. Decision policies for when to ask

A central issue is not only whether a request is ambiguous in the abstract, but whether clarification is worth an additional turn. "Clarify When Necessary" (Zhang et al., 2023) formalizes this as the problem of assigning each input a utility score $R$8 that predicts how much clarification would improve performance. Given feasible outputs

$R$9

one intended output $U$0, and corresponding clarifying answers

$U$1

the paper argues that useful clarification depends on uncertainty over user intents rather than generic uncertainty over model outputs (Zhang et al., 2023). Its INTENT-SIM procedure greedily generates a clarifying question, samples possible user answers, clusters semantically equivalent answers with an NLI model, and computes entropy over the resulting candidate-intent distribution. Under a fixed clarification budget, the system clarifies the highest-entropy examples; the paper reports that when only allowed to ask for clarification on 10% of examples, its system can double the performance gains over randomly selecting examples to clarify (Zhang et al., 2023).

Several later systems separate the ask decision from the content-selection decision. KBQG in conversational recommendation explicitly reduces the policy action space to only two actions—ask a clarifying question or provide recommendations—and lets a separate knowledge-based preference module decide which relation is most worth clarifying next (Ren et al., 2021). Its policy is

$U$2

with dialogue state

$U$3

where $U$4 is a latent belief state, $U$5 marks which relations have already been clarified, and $U$6 indicates the ratio of candidate items whose affinity score exceeds threshold $U$7 (Ren et al., 2021). The same paper treats ask-more versus recommend-sooner as a discounted RL tradeoff, with damping factor $U$8 encouraging fewer conversation turns (Ren et al., 2021).

Other work uses explicit classifiers or routers. CoA trains a controller with cross-entropy,

$U$9

to choose between answer and clarify before any question is generated (Cao et al., 23 Jan 2026). Ask-to-Clarify for embodied agents uses four signal tokens—<AMBG>, <NOT_AMBG>, <ACT>, and <REJ>—and a training-free signal detector that routes between dialogue and action execution according to the token at the end of the VLM output (Lin et al., 18 Sep 2025). In selective QA, CLAM uses a prompted ambiguity classifier whose score is the log probability of the next token being True in a prompt ending with “This question is ambiguous:”; the threshold is tuned on a holdout set and the paper uses $T$0 in the remaining experiments (Kuhn et al., 2022).

Prompt-based agent work has also made the ask trigger itself an uncertainty representation. "Uncertainty Decomposition for Clarification Seeking in LLM Agents" (Matsnev, 17 Jun 2026) argues that a single confidence scalar collapses together uncertainty about the next action and uncertainty about whether the goal is sufficiently specified. It therefore decomposes output into action confidence $T$1 and request uncertainty $T$2, where $T$3 estimates the degree to which the user’s goal is underspecified (Matsnev, 17 Jun 2026). The decision rule is deterministic: $T$4 with $T$5 reported as a balanced default, though the paper also shows that $T$6 gives the highest mean clarification F1 on its clarification benchmarks averaged across five backbones (Matsnev, 17 Jun 2026). By contrast, work on CoDraw finds that human clarification behavior is only weakly aligned with model uncertainty, and proposes a threshold policy

$T$7

based on the model’s own uncertainty rather than imitation of human asking points (Testoni et al., 2024). This suggests that clarification policies optimized for downstream task success need not mirror human turn-taking preferences.

3. Choosing what to ask

Once the decision to clarify has been made, the framework must identify the missing variable or the most informative distinction among competing interpretations. The design space spans free-form generation, template-based content selection, and expected-outcome optimization.

In task-oriented information seeking, MAS2S is a BART-based multi-source seq2seq model with separate encodings of dialogue, user profile, and task knowledge (Feng et al., 2023). The dialogue encoder outputs token-level dialogue embeddings $T$8, the user-profile encoder outputs $T$9, and task knowledge is represented by a single embedding $Q_k$0 taken from the final $Q_k$1 state (Feng et al., 2023). Task knowledge is then used as the query in bilinear attention: $Q_k$2 yielding knowledge-attended dialogue and profile vectors

$Q_k$3

which are fused into an “answer confidence embedding”

$Q_k$4

The decoder, initialized from $Q_k$5, then generates either a clarification question or a final answer, with

$Q_k$6

over vocabulary and candidate answers (Feng et al., 2023). The paper interprets this embedding as indicating whether enough evidence has been gathered.

In KBQG, question generation is primarily content selection rather than neural surface realization. The system computes relation-attention weights

$Q_k$7

over knowledge-graph relations and sequentially picks an unused relation with highest weight; a hand-designed template then turns that relation into a generic question such as “What is your preference on the genre of the movie?” (Ren et al., 2021). The important modeling choice is that the system asks at the relation level but stores the answer at the value level: $Q_k$8 where $Q_k$9 are specific attribute values extracted from user responses (Ren et al., 2021).

Several papers optimize question utility more directly. ClarifyDelphi treats a good clarification question as one whose plausible answers most alter the downstream moral judgment (Pyatkin et al., 2022). For a generated question $A_k$0, an answer model simulates a weakener answer $A_k$1 and a strengthener answer $A_k$2, Delphi predicts judgment distributions $A_k$3 and $A_k$4, and the reward is the Jensen–Shannon divergence

$A_k$5

so questions are rewarded when their plausible answers induce divergent moral judgments (Pyatkin et al., 2022). In visually grounded open-domain clarification, the system generates candidate yes/no questions by decomposing image captions into noun phrases and then selects the one with highest expected information gain: $A_k$6 Here the best clarification is the one expected to most reduce residual uncertainty over the latent target image (White et al., 2021).

Future-turn modeling in open-domain QA uses a different criterion. A candidate first-turn clarifying question $A_k$7 is scored by rolling out simulated user responses $A_k$8 for each interpretation and averaging exact match over the corresponding final answers: $A_k$9 Direct answers are scored the same way across all interpretations, and ties are broken in favor of the direct answer to avoid gratuitous clarification (Zhang et al., 2024). This suggests that “what to ask” can be trained from future conversational utility rather than from local next-turn preference.

4. State representation, memory, and self-correction

Ask-to-clarify systems depend on how unresolved information is represented and updated. Some architectures use explicit symbolic or semi-symbolic state; others use latent embeddings or prompt-encoded uncertainty explanations.

Visual Curiosity is an early example of a structured memory-centered design (Yang et al., 2018). The agent maintains a scene graph $Y$0 whose nodes store object locations $Y$1 and categorical distributions $Y$2 over attributes, with uncertainty measured by entropy

$Y$3

These entropy values are explicit inputs to the recurrent question policy, which selects a target attribute and object to ask about and optionally a reference object to disambiguate the referent (Yang et al., 2018). After a valid Oracle answer, the top-down update sets the relevant attribute distribution to a one-hot vector, turning dialogue into structured supervision for subsequent perception and further questioning (Yang et al., 2018). A closely related interpretability view appears in “Interpreting Models by Allowing to Ask,” where an asking network recursively predicts, asks, receives a restricted answer from an oracle, and refines its output: $Y$4 In that framing, the question itself is an interpretable disclosure of what the model finds tricky to predict (Kang et al., 2018).

Task and agent settings often rely on latent state, but still separate what is known from what remains unresolved. AskToAct stores per-parameter records with fields such as removed, original, and current, effectively tracking whether each tool argument is specified, unspecified, clarified, or still missing (Zhang et al., 3 Mar 2025). The self-correcting version augments clarification dialogues with five explicit error types—Clearly Stated Intent Clarification, Imprecise Clarification, Irrelevant Clarification, Redundant Clarification, and Incomplete Clarification—and inserts erroneous spans delimited by <SOE> ... <EOE> together with corrective utterances (Zhang et al., 3 Mar 2025). Selective masking then trains the model only on non-error tokens, which the paper describes as enabling dynamic error detection during clarification interactions (Zhang et al., 3 Mar 2025). This suggests that clarification state should include not just slot completeness but also whether the interaction itself has gone off course.

Prompt-based agent work encodes state directly in the emitted trajectory. The proposed decomposition in (Matsnev, 17 Jun 2026) extends history from

$Y$5

to

$Y$6

adding request uncertainty $Y$7 and a natural-language explanation $Y$8 alongside action confidence $Y$9 and its explanation $$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$0 (Matsnev, 17 Jun 2026). ClarQ-LLM takes a different route by representing provider-held information as a dependency-aware response tree: each node corresponds to information the provider may reveal, edges encode logical dependency and permissible order of revelation, and the provider agent only exposes task-critical Response I items when the seeker’s question licenses them (Gan et al., 2024). This suggests that a clarification framework can be grounded either in explicit task trees or in decomposed uncertainty traces, provided that the system can remember which uncertainties have already been resolved and which remain active.

5. Data resources, benchmarks, and evaluation

Evaluation methods differ sharply depending on whether the goal is open-domain dialogue quality, task completion, or recovery of explicit missing variables. In task-oriented dialogue, ClariT extends ShARC by sampling 1–5 clarification turns from existing dialogues, rewriting those question–answer pairs into declarative profile sentences such as “I am a family farmer,” and removing the corresponding clarification turns from the dialogue context (Feng et al., 2023). The resulting dataset contains 108,599 dialogues, 1,742 task knowledge items, 85,749 user profiles, and 260,924 turns, split into 76,019/10,860/21,720 dialogues for training, validation, and testing (Feng et al., 2023). The paper reports a quality check over 100 sampled dialogues with satisfaction ratio 1.0 for fluency, usefulness, relevancy, and clarification, and approximate parity with original ShARC on naturalness (Feng et al., 2023). MAS2S is evaluated with BLEU and ROUGE for clarification-question generation and Success for answer prediction; on ClariT it reaches Success 0.412, BLEU 0.315, and ROUGE 0.296, outperforming GAN-Utility, SOLOIST, and UBAR, with gains over UBAR of +0.015 Success, +0.033 BLEU, and +0.023 ROUGE (Feng et al., 2023).

Open-domain and interactive evaluation tends to move away from surface-form overlap. The single- and multi-turn dialogue corpus in (Aliannejadi et al., 2021) builds triples $$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$1 from TREC Web Track topics, labels clarification need on a 1–4 scale, and evaluates clarification need prediction with Precision, Recall, F1, and MSE while evaluating question selection both by question relevance and by whether appending $$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$2 and $$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$3 to the original request improves downstream document retrieval (Aliannejadi et al., 2021). ClarQ-LLM goes further by embedding seeker agents in interactive bilingual English–Chinese tasks with a provider agent and reporting Success Rate, Average Query Discrepancy,

$$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$4

and Average Query Length,

$$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$5

thereby penalizing both under-questioning and over-questioning (Gan et al., 2024). In its main L2L setting on 260 test tasks, the best reported success rate is 0.605 for Llama3.1-405B-Instruct in Completion mode on English, while human seekers reach 0.80 in Chinese and 0.85 in English in H2L evaluation, leaving a substantial gap (Gan et al., 2024).

Benchmarks tailored to agentic clarification make the ask/no-ask decision itself measurable. AskBench transforms QA pairs into interactive scenarios with explicit rubrics $$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$6 and reports final-answer accuracy, checkpoint coverage, and redundant-questioning rate across AskMind and AskOverconfidence settings (Zhao et al., 4 Feb 2026). Its RLVR training uses turn-level rewards

$$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$7

plus terminal rewards

$$P(Q_{k} \mid R, U, T, Q_1, A_1, \dots, Q_{k-1}, A_{k-1})$$8

to reward targeted clarification and penalize premature answers and useless questions (Zhao et al., 4 Feb 2026). In visual ask-or-answer evaluation, CoA introduces ContextClarify with ambiguous and non-ambiguous contrast instances and reports accuracy, precision, recall, and F1 for the controller, ambiguity-resolution reward and human 0/1/2 scores for clarification quality, and final VQA accuracy (Cao et al., 23 Jan 2026). In tool use, AskToAct evaluates clarification with Intent Coverage Rate, Clarification Efficiency, Clarification Performance Score, and Interaction Rounds, and separately evaluates tool execution with Solution Completion Rate, Tool Selection Score, and Parameter Resolution Score (Zhang et al., 3 Mar 2025). This suggests that mature evaluation must separately measure whether the system asked when necessary, whether it asked about the right thing, whether it stopped at the right time, and whether the final downstream task succeeded.

6. Applications, limitations, and emerging design principles

Across domains, the most stable design principle is the separation of “when to ask” from “what to ask.” KBQG makes this decomposition explicit in conversational recommendation (Ren et al., 2021), CoA does so for VQA under context under-specification (Cao et al., 23 Jan 2026), and Ask-to-Clarify in embodied agents uses a VLM for collaboration plus a diffusion model for action, connected through a FiLM-based connection module and routed by explicit signal tokens (Lin et al., 18 Sep 2025). This suggests that clarification is best treated as a policy layer over downstream task modules rather than as a side effect of generic response generation.

A second design principle is that clarification should be grounded in the structure of the downstream task. In tool use, missing API parameters are explicit operationalizations of user intent (Zhang et al., 3 Mar 2025). In task-oriented dialogue, task knowledge acts as the query that identifies which aspects of the dialogue and user profile matter most (Feng et al., 2023). In ClarifyDelphi, the downstream judgment model defines which missing contexts are consequential (Pyatkin et al., 2022). In visually grounded disambiguation, expected information gain over candidate targets drives question selection (White et al., 2021). A plausible implication is that clarification quality improves when the agent can represent what information is decision-critical, rather than only whether the input is lexically incomplete.

A third recurring lesson is that asking more is not the objective. MAS2S is closer than UBAR to the oracle number of clarification turns on ClariT, with average number of clarification questions 2.97 versus oracle 2.36 and absolute difference 0.94 versus 1.68 (Feng et al., 2023). AskBench reports strong checkpoint coverage together with nontrivial redundant-questioning rates, showing that high coverage can coexist with inefficient over-asking (Zhao et al., 4 Feb 2026). ClarQ-LLM’s AQD and AQL expose the same issue at task level: strong models are more verbose and ask more than humans even when they succeed (Gan et al., 2024). This suggests that an ask-to-clarify framework should optimize for necessary clarification, not maximal interaction.

The main limitations in the literature are also consistent. Many datasets use synthetic or transformed ambiguity: ClariT rewrites sampled clarification turns into user profiles (Feng et al., 2023), AskToAct removes or abstracts tool parameters (Zhang et al., 3 Mar 2025), ContextClarify focuses on single-factor ambiguity (Cao et al., 23 Jan 2026), and the clarification-augmented benchmarks in (Matsnev, 17 Jun 2026) strip attributes, objects, or receptacles procedurally. Several systems rely on simulated users or oracles rather than real human clarifications (Kuhn et al., 2022, Zhang et al., 2024, Zhang et al., 3 Mar 2025). Others solve only one half of the problem: (Abraham et al., 2024) focuses on when to ask in navigation but not what to ask, while (White et al., 2021) focuses on what question would reduce uncertainty but remains restricted to visually grounded yes/no interaction. Prompt-based uncertainty methods are also reported to be systematically overconfident, with reliability diagrams below the diagonal and ECE for GPT-5.1 ranging from 0.24 to 0.66 depending on method and benchmark (Matsnev, 17 Jun 2026).

Taken together, the literature defines ask-to-clarify not as a single model class but as an architectural pattern: maintain an explicit or implicit representation of unresolved information; estimate whether that unresolved information is consequential enough to justify interaction; formulate a targeted question that reduces uncertainty over the latent intent, parameter, referent, or task state; update state from the reply; and terminate clarification only when downstream action or answering is sufficiently supported. MAS2S and ClariT (Feng et al., 2023) provide a concrete task-oriented formulation with dialogue, user profile, and task knowledge; later work extends the same logic to recommendation (Ren et al., 2021), QA (Zhang et al., 2024), tool use (Zhang et al., 3 Mar 2025), multimodal decision making (Cao et al., 23 Jan 2026), black-box LLM agents (Matsnev, 17 Jun 2026), and embodied collaboration (Lin et al., 18 Sep 2025). The shared technical lesson is that clarification is most effective when driven by a representation of task-conditioned insufficiency rather than by generic confidence or imitation of superficial question-asking behavior.