When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search

Abstract: Search agents powered by LLMs are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.

• The paper introduces the DiscoBench benchmark, modeling ambiguity as a dynamic property in multi-step search to assess clarification behavior.
• It demonstrates that current LLM-based search agents struggle with proactive ambiguity detection, with top models achieving less than 45% end-to-end accuracy.
• Empirical results reveal that increased retrieval and reasoning effort marginally improve detection, highlighting the need for novel architectures in interactive disambiguation.

Clarification-Aware Deep Search: An Analysis of the DiscoBench Benchmark

Motivation and Problem Definition

Modern web search agents driven by LLMs increasingly engage in multi-step reasoning and retrieval to fulfill complex information needs. However, most evaluation protocols and benchmarks posit that user queries are explicit, precise, and complete. In real settings, user requests are typically incomplete, vague, underspecified, or factually erroneous, and in deep (multi-hop) search scenarios, such ambiguity propagates, resulting in error cascades that derail reasoning chains and waste computational resources. This is exemplified by critical failure modes where ambiguity is not resolved at an early stage, later poisoning the downstream search trajectory (Figure 1).

Figure 1: A motivating example of ambiguity propagation in interactive deep search.

Methodological Innovations: The DiscoBench Benchmark

DiscoBench is introduced as a comprehensive benchmark targeting proactive ambiguity detection, clarification-question-asking, and disambiguation capabilities in multi-step search. The design explicitly models ambiguity as a dynamic, context-dependent property that arises and propagates across checkpoints in sequential reasoning, departing from conventional static ambiguity definitions.

DiscoBench comprises:

• 211 complex multi-hop queries, systematically constructed across 11 real-world domains and 463 intentionally injected ambiguity instances.
• Four ambiguity types: Entity (shared attributes, multiple candidates), Version (temporal or state-related ambiguity), Criteria (differing standards/rankings), and Factual Inaccuracy (inconsistent or erroneous information).
• A user simulator for multi-turn interaction, supplying discriminative clues only upon appropriate clarification requests.

  Figure 2: Overview of the proposed interactive retrieval framework and evaluation protocol.

The dataset construction pipeline integrates LLM-assisted expansion, manual multi-hop chain design, targeted ambiguity injection (with human review for solvability and realism), and the generation of discriminative facts for interactive clarification, enabling objective, factual, and fine-grained evaluation of both interaction and search.

Figure 3: Overview of the two-phase dataset construction pipeline, including seed multi-hop QA construction, ambiguity injection, discriminative fact generation, and quality control.

Experimental Protocol and Evaluation Metrics

Evaluation is performed along four axes: (1) task utility (end-to-end accuracy, checkpoint pass rate), (2) ambiguity detection (accuracy, F1), (3) interaction quality (quality and efficacy of clarification questions), and (4) cost efficiency (number of clarification turns, tool usage, and token consumption).

Two prompting conditions are considered:

• Neutral Prompts: No explicit ambiguity warning.
• Guided Prompts: Agent is told ambiguity exists and is encouraged to clarify.

SOTA LLM-based search agents (e.g., Doubao-Seed-2.0-Pro, Gemini-3.1-Pro, Claude-Opus-4.7, DeepSeek-V4-Pro, MiniMax-M2.7, etc.) are evaluated using a standardized agent framework interfaced with the Tavily black-box search backend.

Key Findings

Ambiguity Handling Remains a Core Challenge

Despite substantial advancement in open-domain retrieval and agentic reasoning, all agents show poor clarification-aware deep search capabilities. Under neutral prompting, the top-performing agent (Doubao-Seed-2.0-Pro) achieves only 43.1% end-to-end accuracy; the majority of models fall below 40%. Notably, checkpoint pass rates (intermediate task progress) substantially exceed end-to-end completion, identifying ambiguity resolution as a persistent bottleneck.

Guided Prompting and Reasoning Effort Yield Only Marginal Gains

Explicitly instructing agents to watch for ambiguity (guided prompting) increases both detection and clarification rates but does not close the performance gap—clarification behavior improves (detection F1 rises from 45.3% to 64.9%), but final solution rates remain low, indicating that prompt engineering alone is insufficient.

Increasing the agent's 'reasoning effort' also provides improvement, particularly in ambiguity detection and recall, but task accuracy remains capped even at high effort levels, signifying deeper deficiencies in ambiguity localization and interactive planning.

Figure 4: Reasoning-effort comparison for Doubao-Seed-2.0-Pro under neutral prompting.

Ambiguity Detection and Clarification Are Orthogonal Skills

Detection (knowing when to ask for clarification) and the ability to formulate effective clarifying questions are not strongly correlated. For instance, Qwen3.6-Max rarely initiates clarification but produces high-quality clarification questions when forced, whereas other agents over-ask with low utility.

Naïve Search-Heavy Behavior Is Suboptimal

Frequent or repeated use of web search tools (retrieval intensity) does not predict higher accuracy. Agents that continue searching when ambiguity is unresolved (search-heavy guessing) systematically underperform agents that promptly clarify, and sometimes even score worse than direct guessing. This reveals a critical failure mode where retrieval uncertainty is not properly escalated to interactive disambiguation.

Figure 5: Detection performance across different ambiguity types.

Ambiguity Type and Complexity Present Stratified Difficulty

Factual inaccuracies are most readily detected due to explicit conflicts with retrieved evidence, while entity and criteria ambiguities—requiring discrimination among plausible alternatives—are the hardest. Increasing the number of ambiguity checkpoints (complexity) amplifies failure, as ambiguity propagation compounds error probabilities.

Figure 6: Performance across ambiguity-complexity levels under neutral prompting.

Ablation: Retrieval and Ambiguity Are Fundamental

Removal of external search tools devastates performance (e.g., Doubao-Seed-2.0-Pro drops from 43.1% to 2.4%), confirming that parametric memory alone is grossly insufficient. Removing ambiguity from tasks raises accuracy by as much as 40 percentage points, indicating that ambiguity handling is the unique and principal challenge posed by DiscoBench.

Practical and Theoretical Implications

DiscoBench demonstrates that general-purpose LLMs remain severely limited in their ability to:

• Proactively identify ambiguous states without explicit supervision.
• Formulate and sequence context-aware clarification questions.
• Integrate disambiguating information into search trajectories in real time.
• Balance retrieval operations with intelligent interactive interventions.

For practical search agent deployment, these results underscore the necessity of explicit uncertainty modeling and user interaction policies beyond prompt engineering or scaling.

Theoretically, the findings call for research into new architectures and training paradigms that expose agents to multi-turn disambiguation scenarios, possibly incorporating meta-reasoning over uncertainty, model-based ambiguity prediction, and interface co-adaptation with real (not simulated) users for robust user-agent grounding.

Future Directions

Key directions include:

• Expanding ambiguity types to encompass preference, intent, and contextually subjective ambiguity.
• Leveraging reinforcement learning with human-in-the-loop clarification signals, which may address current limitations of LLM-based simulators.
• Developing agent policies that explicitly track and escalate retrieval uncertainty, learning when not to trust parametric knowledge or multi-step evidence chains in the presence of ambiguity.

Conclusion

DiscoBench offers a scientifically rigorous, interaction-driven protocol for benchmarking clarification-aware deep search. It exposes a critical capability gap in current LLM-based agents, which is not resolved through more compute, more searching, or prompt reweighting. Closing this gap is imperative for robust autonomous information-seeking systems and future agent architectures that must operate under realistic, ambiguous user requests.