Fail Fast or Ask in AI Systems

• Fail Fast or Ask is a decision challenge where systems must detect underspecification and balance rapid execution against the need for timely clarification.
• The work proposes models that quantify the value of clarifications based on missing information and empirical timing studies using forced-injection frameworks.
• Insights span AI agents, coding systems, and human-in-the-loop designs, emphasizing calibrated uncertainty, selective escalation, and optimal control policies.

Searching arXiv for the primary paper and closely related work on clarification-seeking and fail-fast behavior in agents. “Fail fast, or ask” denotes a recurring decision problem in AI systems and adjacent engineering workflows: whether to proceed under uncertainty and accept the risk of rapid downstream failure, or to pause for clarification, escalation, or additional evidence. In long-horizon agents, this problem is framed as a timing-sensitive clarification decision over incomplete instructions (Gulati et al., 8 May 2026). In coding agents, it appears as the choice between autonomous execution on underspecified issues and proactive question-asking (Edwards et al., 27 Mar 2026). In human-in-the-loop systems, it becomes a selective deferral policy that trades model speed against human oversight (Zellinger et al., 18 Jul 2025). Across these settings, recent work converges on a common result: the central issue is not merely whether an agent can ask, but whether it can recognize unresolvable uncertainty, ask at the right time, and avoid both silent guessing and low-value over-asking (Gulati et al., 8 May 2026, Elfeki et al., 10 Apr 2026).

1. Long-horizon agents and the timing of clarification

In long-horizon AI agents, a task unfolds as a trajectory of “complex multi-step workflows spanning tens to hundreds of sequential actions,” and a single wrong assumption can “cascade into irreversible errors” (Gulati et al., 8 May 2026). Clarification timing is therefore defined as when during execution the agent receives missing information, relative to continuing under uncertainty. The problem is not only whether to ask, but when to ask (Gulati et al., 8 May 2026).

The paper “Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?” formalizes this using a trajectory commitment model (Gulati et al., 8 May 2026). A task is specified by a parameter vector over four information dimensions—goal, input, constraint, and context—and the agent executes a trajectory while one component is missing. Commitment to a dimension at time $t$ is the fraction of actions so far that causally depend on that missing component, and the value of clarification is upper-bounded by the recoverable portion of the trajectory:

$$\text{VOI}_d(t) \leq \text{VOI}_d(0)\cdot (1-C_d(t)).$$

This framework implies that different missing information types induce different value-of-information decay curves (Gulati et al., 8 May 2026).

To isolate timing from the model’s natural asking behavior, the authors introduce a forced-injection framework in which the ask_user tool is disabled and a ground-truth clarification is injected as a synthetic user message at 10%, 30%, 50%, 70%, or 90% of an oracle-calibrated trajectory length (Gulati et al., 8 May 2026). They evaluate four information dimensions—goal, input, constraint, and context—across 84 underspecified task variants from MCP-Atlas, TheAgentCompany, and SWE-Bench Pro, using frontier models including GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, and DeepSeek V3.2, for 6,048 forced-injection trials plus 300 natural-ask sessions (Gulati et al., 8 May 2026).

The central empirical result is that the value of clarification depends sharply on what information is missing (Gulati et al., 8 May 2026). Goal clarification is “heavily front-loaded”: on MCP-Atlas, pass@3 is 0.78 at 10% injection against 0.80 oracle and 0.40 no-clarification, but by 70% it is 0.39, effectively at baseline (Gulati et al., 8 May 2026). Input clarification decays more gradually: on MCP-Atlas it remains above no-clarification through 50% of execution, falls to baseline at 70%, and becomes worse than never asking at 90% (Gulati et al., 8 May 2026). Constraint clarification helps most when obtained “before execution” or very early, especially when Oracle exceeds no-clarification, whereas context appears “more like goal,” though evidence is weaker because context variants are sparse (Gulati et al., 8 May 2026).

These results directly reject the simple heuristic that earlier is always better in the same way for all missing information. Instead, they support dimension-specific timing windows: goal information should be asked within 0–10% of the trajectory, input information within roughly 0–50%, constraints before execution or very early, and context as early as possible (Gulati et al., 8 May 2026). A further quantitative result is that “deferring any clarification type past mid-trajectory degrades performance below never asking at all” in the aggregate across benchmarks, especially on MCP-Atlas (Gulati et al., 8 May 2026). This establishes a concrete interpretation of “fail fast, or ask” for long-horizon agents: ask early about what the user wants, allow limited exploration for missing inputs, and avoid late pivots once substantial commitment has accumulated (Gulati et al., 8 May 2026).

2. Empirical regularities: task-intrinsic timing and natural asking failures

A major question is whether clarification timing profiles are model-specific or task-intrinsic. The same study computes cross-model Kendall’s $\tau$ correlations over per-variant pass@3 timing patterns and finds strong consistency (Gulati et al., 8 May 2026). On TheAgentCompany, where models share identical coverage, Claude–Gemini, Claude–GPT, and Gemini–GPT correlations are 0.87, 0.82, and 0.78, all significant at $p<10^{-3}$ or better (Gulati et al., 8 May 2026). Across the full four-model panel and all benchmarks, correlations remain positive and significant, ranging from 0.34 to 0.67 (Gulati et al., 8 May 2026). This indicates that clarification timing is “substantially task-intrinsic,” even though architecture and training still matter (Gulati et al., 8 May 2026).

The complementary natural-ask study on TheAgentCompany shows that current frontier models do not naturally ask within the empirically optimal windows (Gulati et al., 8 May 2026). GPT-5.2 asks in 52% of sessions, with mean first-ask timing at 43% of the trajectory and median 50%, making it an “over-clarifier” that often asks too late for goals (Gulati et al., 8 May 2026). Claude Sonnet 4.5 asks in 23% of sessions, almost always once, around the midpoint, making it “balanced/selective” but still late for goal information (Gulati et al., 8 May 2026). Gemini 3 Flash asks in 0% of sessions and is therefore an “under-clarifier” (Gulati et al., 8 May 2026). The paper concludes that no current frontier model matches the empirically optimal timing: GPT-5.2 over-asks and asks late for goals, Claude under-asks but sometimes lands within the useful input window, and Gemini never asks at all (Gulati et al., 8 May 2026).

A related benchmark, HiL-Bench, reframes the same issue as “selective escalation” (Elfeki et al., 10 Apr 2026). It introduces tasks in SWE and text-to-SQL with 3–5 human-validated blockers per task, where blockers can be missing information, ambiguous requests, or contradictory information and “surface only through progressive exploration, not upfront inspection” (Elfeki et al., 10 Apr 2026). Its primary metric, Ask-F1, is the harmonic mean of question precision and blocker recall:

$$\text{Precision}=\frac{|Q_{\text{rel}}|}{|Q|},\qquad \text{Recall}=\frac{|B_{\text{addr}}|}{|B|},\qquad \text{Ask-F1}=\frac{2\cdot \text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}.$$

By design, this metric prevents gaming through question spam (Elfeki et al., 10 Apr 2026).

HiL-Bench finds a “large universal judgment gap” (Elfeki et al., 10 Apr 2026). Full-information pass@3 is 86–91% on text-to-SQL and 64–88% on SWE, but with blocked tasks plus ask_human, best results are only 38% and 12%, respectively (Elfeki et al., 10 Apr 2026). Failure analysis identifies three recurring help-seeking patterns: overconfident wrong beliefs with no gap detection, high uncertainty detection yet persistent errors, and broad, imprecise escalation without self-correction (Elfeki et al., 10 Apr 2026). These observations closely parallel the natural-ask findings in long-horizon agents: both silent guessing and poorly targeted questioning are systematic deficiencies rather than isolated mistakes (Gulati et al., 8 May 2026, Elfeki et al., 10 Apr 2026).

3. Coding agents: uncertainty-aware clarification and the value of asking

The same fail-fast versus ask problem appears in software engineering agents operating on underspecified issues. “Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents” studies an underspecified variant of SWE-bench Verified in which concrete details such as file paths, reproducer code, exact error messages, and bug IDs are deliberately removed while preserving high-level terminology (Edwards et al., 27 Mar 2026). The core decision is whether to proceed directly to navigation, editing, and testing, or to spend a turn querying the user for more context (Edwards et al., 27 Mar 2026).

The paper compares five setups: fully specified without interaction, underspecified without interaction, an interactive baseline that is forced to ask once, a single-agent uncertainty-aware setup, and a multi-agent uncertainty-aware scaffold that separates coding from ambiguity detection (Edwards et al., 27 Mar 2026). All systems use OpenHands and Claude Sonnet 4.5 as the base model, with a user simulator that returns the ground-truth missing information when asked (Edwards et al., 27 Mar 2026).

The main result is that question-asking substantially closes the gap caused by underspecification, but only when asking is well calibrated (Edwards et al., 27 Mar 2026). Resolve rates on 500 tasks are 70.80% for the fully specified condition, 54.80% for the underspecified no-interaction baseline, 61.20% for the uncertainty-aware single agent, 69.40% for the uncertainty-aware multi-agent scaffold, and 70.40% for the interactive baseline (Edwards et al., 27 Mar 2026). The multi-agent system therefore nearly matches the fully specified setting without fine-tuning, indicating that the key barrier is not the coding capability itself but the ability to recognize missing context and trigger clarification (Edwards et al., 27 Mar 2026).

The uncertainty-aware multi-agent scaffold is important because it decouples “understanding whether I can act” from “actually writing code” (Edwards et al., 27 Mar 2026). A dedicated Intent Agent observes the conversation history and outputs a JSON decision with needs_clarification: true/false, while the Main Agent continues the normal solve loop (Edwards et al., 27 Mar 2026). Conditional analyses show that when the multi-agent system asks at least once, its resolve rate is 65.99%, compared to 44.48% for the hidden no-asking baseline on the same subset, and nearly equal to the fully specified and forced-interactive setups at 66.57% (Edwards et al., 27 Mar 2026). When it does not ask, its resolve rate is 76.92%, essentially identical to the no-asking baseline on that subset, implying that its selective asking is well aligned with actual task difficulty (Edwards et al., 27 Mar 2026).

The paper also reports that ask rates increase with estimated human task difficulty, and that the multi-agent system asks more in early and mid trajectory than the single-agent system, which often asks too late (Edwards et al., 27 Mar 2026). This resonates with the timing results from long-horizon agents: clarification is useful only within certain windows, and asking late after speculative execution wastes both tokens and trajectory steps (Gulati et al., 8 May 2026, Edwards et al., 27 Mar 2026). A plausible implication is that coding agents require both a timing policy and a trigger policy: they must detect underspecification and convert that detection into early, targeted questions.

A complementary line of work focuses not only on when to ask, but on how to ask well. “Aligning LLMs to Ask Good Questions: A Case Study in Clinical Reasoning” proposes ALFA, a framework that decomposes question quality into six attributes—clarity, focus, answerability, medical accuracy, diagnostic relevance, and avoiding differential-diagnosis bias—and aligns models through preference optimization on attribute-specific question pairs (Li et al., 20 Feb 2025). Although the domain is clinical reasoning rather than coding, the underlying lesson is general: asking is not just a binary choice; the information gain depends on question quality. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDocs relative to instruction-tuned baselines and achieve a 64.4% question-level win rate (Li et al., 20 Feb 2025). This suggests that a complete treatment of “fail fast, or ask” must include the timing, trigger, and content of the question.

4. Human-in-the-loop systems and selective deferral

A different formulation replaces clarification with direct escalation to a human expert. “Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering” studies a setting where a reasoning model can answer or defer to a human when it cannot confidently answer (Zellinger et al., 18 Jul 2025). On the hardest MATH problems, baseline error rates are 2.8% for Qwen3 235B-A22B, 5.8% for DeepSeek R1, and 3.8% for OpenAI o3, while latencies range from 67.6 s to 125.9 s per query (Zellinger et al., 18 Jul 2025). The goal is to reduce error toward zero without incurring prohibitive latency and cost.

The paper’s key uncertainty proxy is the length of the reasoning trace (Zellinger et al., 18 Jul 2025). For Qwen3 235B-A22B and DeepSeek R1, accuracy remains roughly flat over short traces and then declines sharply as the number of output tokens grows, so selective prediction based on reasoning length becomes effective (Zellinger et al., 18 Jul 2025). Deferring only 7.5% of queries cuts Qwen3’s error on difficult MATH from 2.8% to below 1%, and similar absolute gains appear for DeepSeek R1 (Zellinger et al., 18 Jul 2025). The paper summarizes performance across rejection rates using AU-ARC, the area under the accuracy-rejection curve over 0–20% rejection (Zellinger et al., 18 Jul 2025).

To reduce latency and cost, the authors then front the reasoning model with a large non-reasoning model using a three-way policy:

$$\pi_{nr}= \begin{cases} \text{fail fast} & \text{if } p_{\text{true}}\le c_{\text{fail fast}},\ \text{pass} & \text{if } c_{\text{fail fast}}<p_{\text{true}}\le c_{\text{pass}},\ \text{respond} & \text{if } p_{\text{true}}>c_{\text{pass}}. \end{cases}$$

Here the fast model can answer easy queries, pass medium-confidence queries to the reasoning model, or defer hard queries directly to the human (Zellinger et al., 18 Jul 2025). For DeepSeek R1, using this “Fail Fast, or Ask” pipeline with 60% utilization of the non-reasoning model preserves AU-ARC at 0.91 while reducing latency by 36.5% and cost by 45.8% (Zellinger et al., 18 Jul 2025). The paper also identifies “latency drag,” the phenomenon that easy queries handled by the fast model leave the reasoning model with a harder conditional distribution, increasing its average latency (Zellinger et al., 18 Jul 2025). This shows that fail-fast policies can save cost, but their benefit depends on how uncertainty correlates with runtime.

This selective-deferral perspective clarifies a common misconception. “Ask” need not mean an open-ended clarification dialogue; in human-in-the-loop systems it can mean escalation to a human oracle or specialist. Conversely, “fail fast” does not mean failing irrecoverably; it can mean declining to invoke a slow reasoning pipeline when a query is obviously too difficult and directly deferring to a human (Zellinger et al., 18 Jul 2025). A plausible implication is that clarification-seeking and selective deferral are two variants of the same control problem: minimize error by redirecting only those cases where the value of further autonomous processing is lower than the value of human input.

5. Multi-agent systems and edge-level clarification

In multi-agent systems, the fail-fast versus ask trade-off shifts from user–agent interaction to agent–agent handoffs. “AgentAsk: Multi-Agent Systems Need to Ask” argues that many multi-agent systems underperform strong single-agent baselines because of “edge-level error cascades,” where minor inaccuracies at one message handoff propagate across the chain (Lin et al., 8 Oct 2025). Rather than restarting the whole system or using a heavy evaluator everywhere, AgentAsk inserts minimally necessary clarifications between agents (Lin et al., 8 Oct 2025).

The paper models a multi-agent system as a graph of agents and message edges, with each edge treated as a potential failure point (Lin et al., 8 Oct 2025). AgentAsk observes an edge-local state consisting of the original user query or subgoal, the candidate message, and short-horizon history, and chooses an action $a_t=(z_t,\tilde v_t,q_t)$, where $z_t$ is a binary ask gate, $\tilde v_t$ chooses whether to ask the sender or receiver, and $q_t$ is the clarification question (Lin et al., 8 Oct 2025). The policy factorization is:

$$\text{VOI}_d(t) \leq \text{VOI}_d(0)\cdot (1-C_d(t)).$$0

If $$\text{VOI}_d(t) \leq \text{VOI}_d(0)\cdot (1-C_d(t)).$$1, the message passes through unchanged; if $$\text{VOI}_d(t) \leq \text{VOI}_d(0)\cdot (1-C_d(t)).$$2, the clarifier asks a targeted question, inserts the reply, and only then forwards the message (Lin et al., 8 Oct 2025).

A key contribution is a taxonomy of edge-level failures: Data Gap, Signal Corruption, Referential Drift, and Capability Gap (Lin et al., 8 Oct 2025). Data Gap and Signal Corruption dominate the observed errors, and both are often fixable by a single local question (Lin et al., 8 Oct 2025). AgentAsk is trained in three stages: distilling edge-level judgments from curated failure traces, supervising a compact policy to determine when, what, whom, and how to ask, and optimizing online with E-GRPO to balance accuracy, latency, and cost (Lin et al., 8 Oct 2025). The RL objective combines local edge rewards for effective questions with a terminal correctness reward, while penalizing excessive asking and enforcing a KL constraint to remain close to the distilled policy (Lin et al., 8 Oct 2025).

Empirically, AgentAsk consistently improves accuracy and robustness over public multi-agent implementations across math, reasoning, and coding benchmarks while keeping latency and extra cost under 5%, and approaching the performance of a strong evaluator (Lin et al., 8 Oct 2025). This yields a distinct systems interpretation of “fail fast, or ask”: in multi-agent systems, local asking can dominate global fail-fast mechanisms when most failures are edge-local and one-shot clarifications suffice. A plausible implication is that the same principle may transfer to single-agent toolchains: treat tool outputs, memory writes, or retrieved documents as handoff edges and insert clarifications only where they are likely to arrest an error cascade.

6. Related formulations beyond autonomous agents

The same decision structure appears in several adjacent domains, though with different observables and costs. In automated failure testing for microservices, “Feedback-based, Automated Failure Testing of Microservice-based Applications” proposes IntelliFT, a feedback-based fault-injection framework that uses fault propagation and reduction rules to expose defects in fault-handling logic quickly (Cui et al., 2019). Here “fail fast” refers to surfacing failure behavior early rather than letting silent propagation continue, while “ask” is realized as targeted fault injection guided by prior observations rather than brute-force exploration (Cui et al., 2019). On the TrainTicket benchmark, IntelliFT with reduction rules reduces fault injections from 321 to 39 compared with native LDFI while exploring the relevant failure space (Cui et al., 2019).

In collaborative environments, the problem becomes whether to execute an instruction or ask for clarification. “Learning to Execute Actions or Ask Clarification Questions” extends the Minecraft Dialogue Corpus by annotating all builder utterances into eight categories, including instruction-level and task-level clarification questions (Shi et al., 2022). Builder utterances include 914 instruction-level questions and 252 task-level questions, together about 23.78% of all builder utterances, showing that asking is central to human collaborative building (Shi et al., 2022). The paper defines a “Learning to Ask” task in which the agent predicts whether the next builder action should be Execution, Ask for clarifications, or Others, as well as a joint task that combines action-type prediction with low-level placement and removal actions (Shi et al., 2022). This is an early supervised version of the same selective escalation problem later benchmarked more directly in HiL-Bench (Elfeki et al., 10 Apr 2026).

In CI systems, the phrase “fail fast or ask” has an operational meaning around flaky tests. “230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers” studies whether failure de-duplication and log-based classifiers can distinguish flaky from true failures quickly enough to decide whether to stop the pipeline or rerun and investigate (Alshammari et al., 2024). Text-based matching is nearly perfect in some projects and dangerously ineffective in others, while TF–IDF classification performs best overall, but still with strong project dependence (Alshammari et al., 2024). A plausible implication is that fail-fast policies are safe only when the uncertainty estimation mechanism is well calibrated to the specific environment; otherwise the cost of a wrong early decision is high.

An industry-centered study, “Good and Bad Failures in Industrial CI/CD — Balancing Cost and Quality Assurance,” generalizes this point to pipeline design (Sun et al., 16 Apr 2025). It distinguishes “good” failures, which occur early and cheaply in pre-merge stages, from “bad” failures, which occur late and have wider organizational impact (Sun et al., 16 Apr 2025). Across four embedded-software companies, pre-merge and post-merge failure rates are approximately 5:3 while pre-merge check volume dominates at roughly 15:1, leading the authors to argue that pre-merge optimization offers “more impacts and less risks” (Sun et al., 16 Apr 2025). This industrial evidence echoes the agent findings: early, high-signal clarification or failure detection is useful, whereas late noisy intervention is costly (Sun et al., 16 Apr 2025, Gulati et al., 8 May 2026).

A broader conceptual analogue appears in “Ask Again, Then Fail: LLMs’ Vacillations in Judgment,” which studies how conversational LLMs change correct answers to wrong ones under follow-up questions (Xie et al., 2023). There the challenge is not whether to ask, but whether a follow-up should trigger principled reconsideration or socially induced vacillation. The paper introduces a Follow-up Questioning Mechanism and metrics for Modification and Modification Rate, showing that current models often treat generic user doubt as if it were substantive evidence (Xie et al., 2023). Although the interaction setting differs, the underlying lesson is similar: the value of asking depends on whether the resulting information is epistemically relevant, not merely interactional.

7. Synthesis and open directions

Across long-horizon agents, coding systems, human-in-the-loop reasoning, multi-agent orchestration, and adjacent engineering settings, “fail fast, or ask” has converged into a problem of selective escalation under uncertainty. Three variables recur.

The first is recognition: the system must detect that its current information state is inadequate. HiL-Bench makes this “blocker recall” (Elfeki et al., 10 Apr 2026). The coding-agent work operationalizes it as calibrated uncertainty about resolvability under the current context (Edwards et al., 27 Mar 2026). In human-in-the-loop reasoning, it appears as a measurable relation between reasoning-trace length and answer accuracy (Zellinger et al., 18 Jul 2025).

The second is timing: asking too late can destroy value. The long-horizon forced-injection study is explicit that goal clarification loses nearly all value after 10% of execution, while input clarification remains useful only through roughly 50%, and deferring clarification past mid-trajectory can be worse than never asking at all (Gulati et al., 8 May 2026). This timing dependence is not a minor implementation detail; it changes the optimal control policy by information type.

The third is question quality: over-asking, vague questions, or poorly targeted escalations waste budget and may not resolve uncertainty. Ask-F1 penalizes question spam (Elfeki et al., 10 Apr 2026). ALFA shows that fine-grained attributes such as clarity, focus, answerability, and bias avoidance matter for downstream success (Li et al., 20 Feb 2025). AgentAsk formalizes what to ask, whom to ask, and how to ask at the edge level (Lin et al., 8 Oct 2025).

Several misconceptions therefore become untenable. One is that a capable model will automatically know when to ask; multiple papers show that raw task competence under full information does not imply good help-seeking judgment under missing information (Edwards et al., 27 Mar 2026, Elfeki et al., 10 Apr 2026). Another is that asking more is automatically safer; both Ask-F1 and natural-ask analyses show that over-asking, asking broadly, or asking late can degrade performance or usability (Gulati et al., 8 May 2026, Elfeki et al., 10 Apr 2026). A third is that the best remedy is always global fail-fast behavior; edge-local or blocker-specific clarification can often recover most of the benefit at lower cost than restarting or escalating everything (Lin et al., 8 Oct 2025, Zellinger et al., 18 Jul 2025).

The most plausible general conclusion is that “fail fast, or ask” is best understood as a systems-level policy design problem. It requires calibrated uncertainty detection, dimension-specific or context-specific timing, a notion of the marginal value of information, and explicit optimization under latency, cost, and human-burden constraints. Recent work has supplied missing pieces of that stack: empirical clarification demand curves (Gulati et al., 8 May 2026), trainable help-seeking metrics such as Ask-F1 (Elfeki et al., 10 Apr 2026), attribute-level question alignment (Li et al., 20 Feb 2025), and practical selective-deferral architectures (Zellinger et al., 18 Jul 2025, Lin et al., 8 Oct 2025). What remains open is how to unify these components into robust deployment policies across domains where the cost of silent guessing is high and the human attention budget is finite.