Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

Abstract: Long-horizon AI agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreversible errors. When instructions are incomplete, the agent must decide not only whether to ask for clarification but when, and no prior work measures how clarification value changes over the course of execution. We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the agent's trajectory across four information dimensions (goal, input, constraint, context), three agent benchmarks, and four frontier models (three per benchmark; one on a single benchmark only; 84 task variants; 6,000+ runs). Counter to the common intuition that "earlier is always better," we find that the value of clarification depends sharply on what information is missing: goal clarification loses nearly all value after 10% of execution (pass@3 drops from 0.78 to baseline), while input clarification retains value through roughly 50%. Deferring any clarification type past mid-trajectory degrades performance below never asking at all. Cross-model Kendall tau correlations (0.78-0.87 among models sharing identical task coverage; 0.34-0.67 across the full 4-model panel) confirm these timing profiles are substantially task-intrinsic. A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window, with strategies ranging from over-asking (52% of sessions) to never asking at all. These empirical demand curves provide the quantitative foundation that existing theoretical frameworks require but have lacked, and establish concrete design targets for timing-aware clarification policies. Code and data will be publicly released.

• The paper demonstrates that clarification timing critically impacts task success, with early queries for goal dimensions yielding higher pass@3 scores.
• It employs a forced-injection framework across diverse benchmarks to construct empirical value-of-information curves for different information dimensions.
• Findings reveal that delayed clarifications increase wasted compute and expose a misalignment between optimal inquiry timing and current SOTA LLM behaviors.

Clarification Timing in Long-Horizon LLM Agents: Empirical Value-of-Information Analysis

Motivation and Background

Clarification in agentic workflows is a critical facet of robust LLM agent deployment in domains with significant underspecification and long sequential horizons. While the benefit of clarification is established in agent benchmarks, prior research primarily models clarification as a binary property—ignoring temporal placement as an independent variable. The paper "Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?" (2605.07937) systematically analyzes the value of clarification at different execution points, introducing a forced-injection protocol to supply ground-truth clarifications at controlled times across diverse tasks, information dimensions, and LLM architectures.

Figure 1: Overview of the forced-injection experimental framework, injecting clarifications at controlled points to measure pass@3 across information dimensions.

The experimental approach uncouples the detection of ambiguity from the timing of information delivery, isolating the "demand curve" for clarification by constructing value-of-information (VOI) curves as a function of injection timing. This setting allows the authors to empirically ground theoretical VOI and metareasoning predictions [howard1966information, russell1991right]—addressing a notable lacuna in the agentic reasoning literature.

Experimental Methodology

Task Construction and Protocols

Tasks are drawn from 84 underspecified variants within three benchmark suites covering enterprise workflows (TheAgentCompany), procedural code repair (SWE-Bench Pro), and tool-centric manipulation (MCP-Atlas). Variants manipulate four information dimensions: goal, input, constraint, and context. The core intervention—forced-injection—delivers missing ground-truth segments at selected fractions of an oracle-calibrated action budget (10%, 30%, 50%, 70%, 90%).

Performance is evaluated with pass@3 (task-level success in three attempts), and the cost due to late clarification is captured via "wasted compute"—actions performed pre-injection that do not appear in the oracle trace. Cross-model correlation and "point of no return" analysis further decompose the structure of timing effects.

Dimension-Dependent Clarification Value

Timing Effects on Success Rates

Pass@3 VOI curves show strong dimension-dependence. For goal information, the utility of clarification is highly front-loaded—clarification after 10% of the trajectory recovers almost no additional task success. In MCP-Atlas, goal dimension pass@3 drops from 0.78 (injection at 10%) to 0.44 (at 50%) and aligns with the no-clarification baseline (0.40) after 70%. In contrast, input clarification exhibits a gradual decline, with value persisting through approximately 50% of the workflow (from 0.46 at 10% down to 0.25 at 90%). Constraints and context display less pronounced, highly task-dependent patterns.

Figure 2: MCP-Atlas VOI curves by information dimension reveal front-loaded value for goal, gradual decline for input, and marginal or flat returns for constraint and context dimensions.

Wasted Compute as a Timing Cost

Across all benchmarks, wasted compute increases monotonically with delayed clarification. For TheAgentCompany, waste grows linearly from 0% (injection at 10%) to 21.7% (at 90%); in MCP-Atlas, high waste even at early injections is due to short, irreversible trajectories.

Figure 3: Later clarification consistently produces more wasted work across all benchmarks, with the effect size and pattern influenced by task horizon and commitment structure.

Task-Intrinsic Effects and Consistency

Kendall's $\tau$ rank correlations across models range from 0.78 to 0.87 on the same variant set, and 0.34 to 0.67 when comparing all tasks and models, demonstrating that timing effects arise primarily from intrinsic task structure rather than model-specific factors.

Natural Clarification Behavior in SOTA LLMs

A natural-ask protocol (enabled via the ask_user tool) uncovers substantial misalignment between theoretically optimal and empirically observed clarification policies in current frontier LLMs. GPT-5.2 demonstrates a tendency toward over-clarification (asking in 52% of sessions, mean timing 43%), Claude Sonnet 4.5 asks selectively and late (23% of sessions, mean timing 50%), and Gemini 3 Flash does not ask at all. Notably, models fail to ask during the empirically optimal window, especially for the goal dimension.

Figure 4: Natural-ask protocol results; "over-clarifier" (GPT-5.2), "selective" (Claude), and "under-clarifier" (Gemini 3 Flash) archetypes are evident in both frequency and timing.

Overlaying empirical ask distributions onto VOI curves highlights this misalignment. For instance, GPT-5.2 and Claude both systematically ask too late to realize the available utility of goal clarification.

Figure 5: Natural ask timing superimposed on VOI curves shows large behavioral gaps—frontier models do not request clarification in the empirically optimal window for goal or input.

Theoretical Alignment, Anomalies, and Policy Implications

The results directly corroborate value-of-information theoretical predictions: front-loaded, irreversible commitment for goal/context, gradual decay for input, and rapid attenuation for constraint only in the presence of "oracle gaps." Contradicting naive intuition, clarification delay can outperform late, or even any, clarification—highlighting the disruptive potential of mid-trajectory injection for certain dimension/task pairs.

Figure 6: Aggregate VOI curves across all benchmarks reinforce dimension-dependent, non-monotonic clarification value profiles.

There is no absolute "point of no return" except for outcome-critical goal and constraint cases, where significant recovery ceases after 30% of the execution path.

Limitations and Future Directions

The analysis delineates the demand curve—the causal value of receiving clarification at each point—but leaves open the supply problem of ambiguity detection and optimal ask-timing. Forced injection disables the model's own ask behavior, ensuring upper bounds on achievable performance. Additional noise is present in some sub-benchmark/dimension cells—especially for SWE-Bench Pro and context dimension—due to limited variant coverage.

Future research directions include development of timing-sensitive clarification policies, mediation between ask volume and selectivity, and the integration of supply- and demand-side estimation for meta-reasoned ask-or-act routines. This work provides a foundation for such policy optimization and for curriculum design in agent training.

Conclusion

This paper constructs the first empirical, dimension-sensitive value-of-information curves for clarification timing in long-horizon LLM agents, revealing sharp task and information-dimensional structure. The value of clarification is not universally "earlier is better": only goal-dimension queries demand near-immediate resolution; input-dimension queries can be delayed with little penalty; late clarification can be worse than no clarification at all for constraints. Existing SOTA models routinely fail to ask within effective time windows, establishing a concrete policy deficit. Practically, these results supply actionable targets for next-generation agentic policies and architecturally-informed ambiguity management. The forced-injection methodology and resulting VOI curves are likely to become core benchmarks for the empirical study of meta-reasoning and dynamic information acquisition in high-stakes, long-horizon AI systems.