---
title: Fail Fast or Ask in AI Systems
url: https://www.emergentmind.com/topics/fail-fast-or-ask
type: topic
---

# Fail Fast or Ask in AI Systems

Searching arXiv for the primary paper and closely related work on clarification-seeking and fail-fast behavior in agents.
“Fail fast, or ask” denotes a recurring decision problem in AI systems and adjacent engineering workflows: whether to proceed under uncertainty and accept the risk of rapid downstream failure, or to pause for clarification, escalation, or additional evidence. In long-horizon agents, this problem is framed as a timing-sensitive clarification decision over incomplete instructions [2605.07937]. In coding agents, it appears as the choice between autonomous execution on underspecified issues and proactive question-asking [2603.26233]. In human-in-the-loop systems, it becomes a selective deferral policy that trades model speed against human oversight [2507.14406]. Across these settings, recent work converges on a common result: the central issue is not merely whether an agent can ask, but whether it can recognize unresolvable uncertainty, ask at the right time, and avoid both silent guessing and low-value over-asking [2605.07937], [2604.09408].

## 1. Long-horizon agents and the timing of clarification

In long-horizon AI agents, a task unfolds as a trajectory of “complex multi-step workflows spanning tens to hundreds of sequential actions,” and a single wrong assumption can “cascade into irreversible errors” [2605.07937]. Clarification timing is therefore defined as *when* during execution the agent receives missing information, relative to continuing under uncertainty. The problem is not only *whether* to ask, but *when* to ask [2605.07937].

The paper “Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?” formalizes this using a trajectory commitment model [2605.07937]. A task is specified by a parameter vector over four information dimensions—goal, input, constraint, and context—and the agent executes a trajectory while one component is missing. Commitment to a dimension at time $t$ is the fraction of actions so far that causally depend on that missing component, and the value of clarification is upper-bounded by the recoverable portion of the trajectory:
$$
\text{VOI}_d(t) \leq \text{VOI}_d(0)\cdot (1-C_d(t)).
$$
This framework implies that different missing information types induce different value-of-information decay curves [2605.07937].

To isolate timing from the model’s natural asking behavior, the authors introduce a forced-injection framework in which the `ask_user` tool is disabled and a ground-truth clarification is injected as a synthetic user message at 10%, 30%, 50%, 70%, or 90% of an oracle-calibrated trajectory length [2605.07937]. They evaluate four information dimensions—goal, input, constraint, and context—across 84 underspecified task variants from MCP-Atlas, TheAgentCompany, and SWE-Bench Pro, using frontier models including GPT-5.2, Claude Sonnet 4.5, Gemini 3 Flash, and DeepSeek V3.2, for 6,048 forced-injection trials plus 300 natural-ask sessions [2605.07937].

The central empirical result is that the value of clarification depends sharply on what information is missing [2605.07937]. Goal clarification is “heavily front-loaded”: on MCP-Atlas, pass@3 is 0.78 at 10% injection against 0.80 oracle and 0.40 no-clarification, but by 70% it is 0.39, effectively at baseline [2605.07937]. Input clarification decays more gradually: on MCP-Atlas it remains above no-clarification through 50% of execution, falls to baseline at 70%, and becomes worse than never asking at 90% [2605.07937]. Constraint clarification helps most when obtained “before execution” or very early, especially when Oracle exceeds no-clarification, whereas context appears “more like goal,” though evidence is weaker because context variants are sparse [2605.07937].

These results directly reject the simple heuristic that earlier is always better in the same way for all missing information. Instead, they support dimension-specific timing windows: goal information should be asked within 0–10% of the trajectory, input information within roughly 0–50%, constraints before execution or very early, and context as early as possible [2605.07937]. A further quantitative result is that “deferring any clarification type past mid-trajectory degrades performance below never asking at all” in the aggregate across benchmarks, especially on MCP-Atlas [2605.07937]. This establishes a concrete interpretation of “fail fast, or ask” for long-horizon agents: ask early about what the user wants, allow limited exploration for missing inputs, and avoid late pivots once substantial commitment has accumulated [2605.07937].

## 2. Empirical regularities: task-intrinsic timing and natural asking failures

A major question is whether clarification timing profiles are model-specific or task-intrinsic. The same study computes cross-model Kendall’s $\tau$ correlations over per-variant pass@3 timing patterns and finds strong consistency [2605.07937]. On TheAgentCompany, where models share identical coverage, Claude–Gemini, Claude–GPT, and Gemini–GPT correlations are 0.87, 0.82, and 0.78, all significant at $p<10^{-3}$ or better [2605.07937]. Across the full four-model panel and all benchmarks, correlations remain positive and significant, ranging from 0.34 to 0.67 [2605.07937]. This indicates that clarification timing is “substantially task-intrinsic,” even though architecture and training still matter [2605.07937].

The complementary natural-ask study on TheAgentCompany shows that current frontier models do not naturally ask within the empirically optimal windows [2605.07937]. GPT-5.2 asks in 52% of sessions, with mean first-ask timing at 43% of the trajectory and median 50%, making it an “over-clarifier” that often asks too late for goals [2605.07937]. Claude Sonnet 4.5 asks in 23% of sessions, almost always once, around the midpoint, making it “balanced/selective” but still late for goal information [2605.07937]. Gemini 3 Flash asks in 0% of sessions and is therefore an “under-clarifier” [2605.07937]. The paper concludes that no current frontier model matches the empirically optimal timing: GPT-5.2 over-asks and asks late for goals, Claude under-asks but sometimes lands within the useful input window, and Gemini never asks at all [2605.07937].

A related benchmark, HiL-Bench, reframes the same issue as “selective escalation” [2604.09408]. It introduces tasks in SWE and text-to-SQL with 3–5 human-validated blockers per task, where blockers can be missing information, ambiguous requests, or contradictory information and “surface only through progressive exploration, not upfront inspection” [2604.09408]. Its primary metric, Ask-F1, is the harmonic mean of question precision and blocker recall:
$$
\text{Precision}=\frac{|Q_{\text{rel}}|}{|Q|},\qquad
\text{Recall}=\frac{|B_{\text{addr}}|}{|B|},\qquad
\text{Ask-F1}=\frac{2\cdot \text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}.
$$
By design, this metric prevents gaming through question spam [2604.09408].

HiL-Bench finds a “large universal judgment gap” [2604.09408]. Full-information pass@3 is 86–91% on text-to-SQL and 64–88% on SWE, but with blocked tasks plus `ask_human`, best results are only 38% and 12%, respectively [2604.09408]. Failure analysis identifies three recurring help-seeking patterns: overconfident wrong beliefs with no gap detection, high uncertainty detection yet persistent errors, and broad, imprecise escalation without self-correction [2604.09408]. These observations closely parallel the natural-ask findings in long-horizon agents: both silent guessing and poorly targeted questioning are systematic deficiencies rather than isolated mistakes [2605.07937], [2604.09408].

## 3. Coding agents: uncertainty-aware clarification and the value of asking

The same fail-fast versus ask problem appears in software engineering agents operating on underspecified issues. “Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents” studies an underspecified variant of SWE-bench Verified in which concrete details such as file paths, reproducer code, exact error messages, and bug IDs are deliberately removed while preserving high-level terminology [2603.26233]. The core decision is whether to proceed directly to navigation, editing, and testing, or to spend a turn querying the user for more context [2603.26233].

The paper compares five setups: fully specified without interaction, underspecified without interaction, an interactive baseline that is forced to ask once, a single-agent uncertainty-aware setup, and a multi-agent uncertainty-aware scaffold that separates coding from ambiguity detection [2603.26233]. All systems use OpenHands and Claude Sonnet 4.5 as the base model, with a user simulator that returns the ground-truth missing information when asked [2603.26233].

The main result is that question-asking substantially closes the gap caused by underspecification, but only when asking is well calibrated [2603.26233]. Resolve rates on 500 tasks are 70.80% for the fully specified condition, 54.80% for the underspecified no-interaction baseline, 61.20% for the uncertainty-aware single agent, 69.40% for the uncertainty-aware multi-agent scaffold, and 70.40% for the interactive baseline [2603.26233]. The multi-agent system therefore nearly matches the fully specified setting without fine-tuning, indicating that the key barrier is not the coding capability itself but the ability to recognize missing context and trigger clarification [2603.26233].

The uncertainty-aware multi-agent scaffold is important because it decouples “understanding whether I can act” from “actually writing code” [2603.26233]. A dedicated Intent Agent observes the conversation history and outputs a JSON decision with `needs_clarification: true/false`, while the Main Agent continues the normal solve loop [2603.26233]. Conditional analyses show that when the multi-agent system asks at least once, its resolve rate is 65.99%, compared to 44.48% for the hidden no-asking baseline on the same subset, and nearly equal to the fully specified and forced-interactive setups at 66.57% [2603.26233]. When it does not ask, its resolve rate is 76.92%, essentially identical to the no-asking baseline on that subset, implying that its selective asking is well aligned with actual task difficulty [2603.26233].

The paper also reports that ask rates increase with estimated human task difficulty, and that the multi-agent system asks more in early and mid trajectory than the single-agent system, which often asks too late [2603.26233]. This resonates with the timing results from long-horizon agents: clarification is useful only within certain windows, and asking late after speculative execution wastes both tokens and trajectory steps [2605.07937], [2603.26233]. A plausible implication is that coding agents require both a timing policy and a trigger policy: they must detect underspecification and convert that detection into early, targeted questions.

A complementary line of work focuses not only on when to ask, but on how to ask well. “Aligning LLMs to Ask Good Questions: A Case Study in Clinical Reasoning” proposes ALFA, a framework that decomposes question quality into six attributes—clarity, focus, answerability, medical accuracy, diagnostic relevance, and avoiding differential-diagnosis bias—and aligns models through preference optimization on attribute-specific question pairs [2502.14860]. Although the domain is clinical reasoning rather than coding, the underlying lesson is general: asking is not just a binary choice; the information gain depends on question quality. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDocs relative to instruction-tuned baselines and achieve a 64.4% question-level win rate [2502.14860]. This suggests that a complete treatment of “fail fast, or ask” must include the timing, trigger, and content of the question.

## 4. Human-in-the-loop systems and selective deferral

A different formulation replaces clarification with direct escalation to a human expert. “Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering” studies a setting where a reasoning model can answer or defer to a human when it cannot confidently answer [2507.14406]. On the hardest MATH problems, baseline error rates are 2.8% for Qwen3 235B-A22B, 5.8% for DeepSeek R1, and 3.8% for OpenAI o3, while latencies range from 67.6 s to 125.9 s per query [2507.14406]. The goal is to reduce error toward zero without incurring prohibitive latency and cost.

The paper’s key uncertainty proxy is the length of the reasoning trace [2507.14406]. For Qwen3 235B-A22B and DeepSeek R1, accuracy remains roughly flat over short traces and then declines sharply as the number of output tokens grows, so selective prediction based on reasoning length becomes effective [2507.14406]. Deferring only 7.5% of queries cuts Qwen3’s error on difficult MATH from 2.8% to below 1%, and similar absolute gains appear for DeepSeek R1 [2507.14406]. The paper summarizes performance across rejection rates using AU-ARC, the area under the accuracy-rejection curve over 0–20% rejection [2507.14406].

To reduce latency and cost, the authors then front the reasoning model with a large non-reasoning model using a three-way policy:
$$
\pi_{nr}=
\begin{cases}
\text{fail fast} & \text{if } p_{\text{true}}\le c_{\text{fail fast}},\\
\text{pass} & \text{if } c_{\text{fail fast}}<p_{\text{true}}\le c_{\text{pass}},\\
\text{respond} & \text{if } p_{\text{true}}>c_{\text{pass}}.
\end{cases}
$$
Here the fast model can answer easy queries, pass medium-confidence queries to the reasoning model, or defer hard queries directly to the human [2507.14406]. For DeepSeek R1, using this “Fail Fast, or Ask” pipeline with 60% utilization of the non-reasoning model preserves AU-ARC at 0.91 while reducing latency by 36.5% and cost by 45.8% [2507.14406]. The paper also identifies “latency drag,” the phenomenon that easy queries handled by the fast model leave the reasoning model with a harder conditional distribution, increasing its average latency [2507.14406]. This shows that fail-fast policies can save cost, but their benefit depends on how uncertainty correlates with runtime.

This selective-deferral perspective clarifies a common misconception. “Ask” need not mean an open-ended clarification dialogue; in human-in-the-loop systems it can mean escalation to a human oracle or specialist. Conversely, “fail fast” does not mean failing irrecoverably; it can mean declining to invoke a slow reasoning pipeline when a query is obviously too difficult and directly deferring to a human [2507.14406]. A plausible implication is that clarification-seeking and selective deferral are two variants of the same control problem: minimize error by redirecting only those cases where the value of further autonomous processing is lower than the value of human input.

## 5. Multi-agent systems and edge-level clarification

In multi-agent systems, the fail-fast versus ask trade-off shifts from user–agent interaction to agent–agent handoffs. “AgentAsk: Multi-Agent Systems Need to Ask” argues that many multi-agent systems underperform strong single-agent baselines because of “edge-level error cascades,” where minor inaccuracies at one message handoff propagate across the chain [2510.07593]. Rather than restarting the whole system or using a heavy evaluator everywhere, AgentAsk inserts minimally necessary clarifications between agents [2510.07593].

The paper models a multi-agent system as a graph of agents and message edges, with each edge treated as a potential failure point [2510.07593]. AgentAsk observes an edge-local state consisting of the original user query or subgoal, the candidate message, and short-horizon history, and chooses an action $a_t=(z_t,\tilde v_t,q_t)$, where $z_t$ is a binary ask gate, $\tilde v_t$ chooses whether to ask the sender or receiver, and $q_t$ is the clarification question [2510.07593]. The policy factorization is:
$$
\pi_\theta(a_t\mid x_t)=\pi_\theta(z_t\mid x_t)\Big[\pi_\theta(\tilde v_t\mid x_t)\pi_\theta(q_t\mid x_t)\Big]^{z_t}.
$$
If $z_t=0$, the message passes through unchanged; if $z_t=1$, the clarifier asks a targeted question, inserts the reply, and only then forwards the message [2510.07593].

A key contribution is a taxonomy of edge-level failures: Data Gap, Signal Corruption, Referential Drift, and Capability Gap [2510.07593]. Data Gap and Signal Corruption dominate the observed errors, and both are often fixable by a single local question [2510.07593]. AgentAsk is trained in three stages: distilling edge-level judgments from curated failure traces, supervising a compact policy to determine when, what, whom, and how to ask, and optimizing online with E-GRPO to balance accuracy, latency, and cost [2510.07593]. The RL objective combines local edge rewards for effective questions with a terminal correctness reward, while penalizing excessive asking and enforcing a KL constraint to remain close to the distilled policy [2510.07593].

Empirically, AgentAsk consistently improves accuracy and robustness over public multi-agent implementations across math, reasoning, and coding benchmarks while keeping latency and extra cost under 5%, and approaching the performance of a strong evaluator [2510.07593]. This yields a distinct systems interpretation of “fail fast, or ask”: in multi-agent systems, local asking can dominate global fail-fast mechanisms when most failures are edge-local and one-shot clarifications suffice. A plausible implication is that the same principle may transfer to single-agent toolchains: treat tool outputs, memory writes, or retrieved documents as handoff edges and insert clarifications only where they are likely to arrest an error cascade.

## 6. Related formulations beyond autonomous agents

The same decision structure appears in several adjacent domains, though with different observables and costs. In automated failure testing for microservices, “Feedback-based, Automated Failure Testing of Microservice-based Applications” proposes IntelliFT, a feedback-based fault-injection framework that uses fault propagation and reduction rules to expose defects in fault-handling logic quickly [1908.06466]. Here “fail fast” refers to surfacing failure behavior early rather than letting silent propagation continue, while “ask” is realized as targeted fault injection guided by prior observations rather than brute-force exploration [1908.06466]. On the TrainTicket benchmark, IntelliFT with reduction rules reduces fault injections from 321 to 39 compared with native LDFI while exploring the relevant failure space [1908.06466].

In collaborative environments, the problem becomes whether to execute an instruction or ask for clarification. “Learning to Execute Actions or Ask Clarification Questions” extends the Minecraft Dialogue Corpus by annotating all builder utterances into eight categories, including instruction-level and task-level clarification questions [2204.08373]. Builder utterances include 914 instruction-level questions and 252 task-level questions, together about 23.78% of all builder utterances, showing that asking is central to human collaborative building [2204.08373]. The paper defines a “Learning to Ask” task in which the agent predicts whether the next builder action should be Execution, Ask for clarifications, or Others, as well as a joint task that combines action-type prediction with low-level placement and removal actions [2204.08373]. This is an early supervised version of the same selective escalation problem later benchmarked more directly in HiL-Bench [2604.09408].

In CI systems, the phrase “fail fast or ask” has an operational meaning around flaky tests. “230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers” studies whether failure de-duplication and log-based classifiers can distinguish flaky from true failures quickly enough to decide whether to stop the pipeline or rerun and investigate [2401.15788]. Text-based matching is nearly perfect in some projects and dangerously ineffective in others, while TF–IDF classification performs best overall, but still with strong project dependence [2401.15788]. A plausible implication is that fail-fast policies are safe only when the uncertainty estimation mechanism is well calibrated to the specific environment; otherwise the cost of a wrong early decision is high.

An industry-centered study, “Good and Bad Failures in Industrial CI/CD — Balancing Cost and Quality Assurance,” generalizes this point to pipeline design [2504.11839]. It distinguishes “good” failures, which occur early and cheaply in pre-merge stages, from “bad” failures, which occur late and have wider organizational impact [2504.11839]. Across four embedded-software companies, pre-merge and post-merge failure rates are approximately 5:3 while pre-merge check volume dominates at roughly 15:1, leading the authors to argue that pre-merge optimization offers “more impacts and less risks” [2504.11839]. This industrial evidence echoes the agent findings: early, high-signal clarification or failure detection is useful, whereas late noisy intervention is costly [2504.11839], [2605.07937].

A broader conceptual analogue appears in “Ask Again, Then Fail: Large Language Models’ Vacillations in Judgment,” which studies how conversational LLMs change correct answers to wrong ones under follow-up questions [2310.02174]. There the challenge is not whether to ask, but whether a follow-up should trigger principled reconsideration or socially induced vacillation. The paper introduces a Follow-up Questioning Mechanism and metrics for Modification and Modification Rate, showing that current models often treat generic user doubt as if it were substantive evidence [2310.02174]. Although the interaction setting differs, the underlying lesson is similar: the value of asking depends on whether the resulting information is epistemically relevant, not merely interactional.

## 7. Synthesis and open directions

Across long-horizon agents, coding systems, human-in-the-loop reasoning, multi-agent orchestration, and adjacent engineering settings, “fail fast, or ask” has converged into a problem of selective escalation under uncertainty. Three variables recur.

The first is **recognition**: the system must detect that its current information state is inadequate. HiL-Bench makes this “blocker recall” [2604.09408]. The coding-agent work operationalizes it as calibrated uncertainty about resolvability under the current context [2603.26233]. In human-in-the-loop reasoning, it appears as a measurable relation between reasoning-trace length and answer accuracy [2507.14406].

The second is **timing**: asking too late can destroy value. The long-horizon forced-injection study is explicit that goal clarification loses nearly all value after 10% of execution, while input clarification remains useful only through roughly 50%, and deferring clarification past mid-trajectory can be worse than never asking at all [2605.07937]. This timing dependence is not a minor implementation detail; it changes the optimal control policy by information type.

The third is **question quality**: over-asking, vague questions, or poorly targeted escalations waste budget and may not resolve uncertainty. Ask-F1 penalizes question spam [2604.09408]. ALFA shows that fine-grained attributes such as clarity, focus, answerability, and bias avoidance matter for downstream success [2502.14860]. AgentAsk formalizes what to ask, whom to ask, and how to ask at the edge level [2510.07593].

Several misconceptions therefore become untenable. One is that a capable model will automatically know when to ask; multiple papers show that raw task competence under full information does not imply good help-seeking judgment under missing information [2603.26233], [2604.09408]. Another is that asking more is automatically safer; both Ask-F1 and natural-ask analyses show that over-asking, asking broadly, or asking late can degrade performance or usability [2605.07937], [2604.09408]. A third is that the best remedy is always global fail-fast behavior; edge-local or blocker-specific clarification can often recover most of the benefit at lower cost than restarting or escalating everything [2510.07593], [2507.14406].

The most plausible general conclusion is that “fail fast, or ask” is best understood as a **systems-level policy design problem**. It requires calibrated uncertainty detection, dimension-specific or context-specific timing, a notion of the marginal value of information, and explicit optimization under latency, cost, and human-burden constraints. Recent work has supplied missing pieces of that stack: empirical clarification demand curves [2605.07937], trainable help-seeking metrics such as Ask-F1 [2604.09408], attribute-level question alignment [2502.14860], and practical selective-deferral architectures [2507.14406], [2510.07593]. What remains open is how to unify these components into robust deployment policies across domains where the cost of silent guessing is high and the human attention budget is finite.

Source: https://www.emergentmind.com/topics/fail-fast-or-ask