Ask-F1: Help-Seeking Evaluation Metric

• Ask-F1 is a metric that quantifies selective help-seeking in agents by balancing the trade-off between over-asking and under-asking.
• It employs a harmonic mean of question precision and blocker recall to penalize both irrelevant escalations and missed ambiguities.
• Empirical results on HiL-Bench show that Ask-F1 improves safety and decision-making by ensuring agents ask for help only when necessary.

Ask-F1 is a process-oriented evaluation metric designed to quantify an agent's selective help-seeking judgment: specifically, its ability to decide when to escalate ambiguous or incomplete information to a human, and to do so with discriminative precision. Developed for the HiL-Bench (Human-in-the-Loop Benchmark), Ask-F1 operationalizes the balance between over-asking (spamming the human for every uncertainty) and under-asking (making overconfident guesses in the face of ambiguity), using the harmonic mean of question precision and blocker recall. The metric is designed to be immune to superficial attacks (such as question spam) and is constructed to reveal and penalize both failure modes symmetrically (Elfeki et al., 10 Apr 2026).

1. Formal Definition and Metric Structure

In the HiL-Bench, each task contains a set of human-validated “blockers” $B = \{b_1, \ldots, b_k\}$: these are missing values, ambiguities, or contradictions engineered to be discoverable only through progressive exploration. An agent can issue questions $Q = \{q_1, \ldots, q_n\}$ via an ask_human() tool. The primary evaluation pipeline involves:

• Question Precision (Pₐₛₖ): Fraction of issued questions judged to be “relevant” (i.e., actually targeting a real blocker):

$P_{ask} = \frac{|Q_{rel}|}{|Q|}$

where $Q_{rel} \subset Q$ contains questions matched to some blocker trigger.

• Blocker Recall (R_bₗₒcₖ): Fraction of all blockers that are actually addressed by at least one relevant question:

$R_{block} = \frac{|B_{addr}|}{|B|}$

with $B_{addr} \subset B$ the set of blockers resolved by any $q \in Q_{rel}$.

• Ask-F1 Score: Harmonic mean of these two quantities:

$$\text{Ask-F1} = \frac{2 \cdot P_{ask} \cdot R_{block}}{P_{ask} + R_{block}}$$

This metric structurally prevents trivial maximization by “questioning everything” (which would drive precision to zero) or “never asking” (which would drive recall to zero) (Elfeki et al., 10 Apr 2026).

2. Motivation, Intuition, and Anti-Gaming Guarantees

Ask-F1 is motivated by the persistent failure of execution-only benchmarks to differentiate between agents that correctly recognize knowledge gaps and escalate versus those that make fortunate guesses. The metric's harmonic mean formulation ensures that both axes—precision and recall—must be high for a good score. If an agent asks very often but rarely with relevance (e.g., 50 questions to resolve 4 out of 5 blockers), $P_{ask} \approx 0.08$, $R_{block} \approx 0.8$, so $Q = \{q_1, \ldots, q_n\}$0. If an agent never asks, precision is undefined ($Q = \{q_1, \ldots, q_n\}$1) or considered perfect but recall is 0, so Ask-F1 is 0.

Unlike pure recall or coverage metrics, Ask-F1 is robust to spam; it “collapses” if the agent overwhelms the human with irrelevant questions, since precision dominates the denominator in the harmonic mean. The design closes the “spam loophole” present in pure-coverage evaluation designs and directly penalizes both “silent” overconfidence and indiscriminate escalation (Elfeki et al., 10 Apr 2026).

3. Benchmark Protocol and Implementation in HiL-Bench

Each HiL-Bench task contains domain-expert-injected blockers validated against strict criteria: realism, criticality, objectivity, independence, vastness of search space, no contamination, and non-contrivance. Blockers are concealed such that they only surface under agent exploration (e.g., by encountering missing information in code, or responding to ambiguous schema elements in text-to-SQL).

A registry of “trigger questions” exists for each blocker, with judgements performed by a frozen ask_human() judge LLM via phrase matching. The agent’s questions are compared to triggers; matches yield both a resolution and a count towards $Q = \{q_1, \ldots, q_n\}$2/$Q = \{q_1, \ldots, q_n\}$3, while non-matches are labeled “irrelevant question”.

This creates a highly controlled measurement regime where the agent's ask/guess tradeoff is precisely observable and manipulable via the Ask-F1 metric (Elfeki et al., 10 Apr 2026).

4. Empirical Results and Characteristic Failure Modes

Empirical analysis across state-of-the-art agents in both text-to-SQL and software engineering (SWE) tasks demonstrates the judgment gap:

Model	R_block (SQL)	P_ask (SQL)	Ask-F1 (SQL)	Pass@3 Full→Ask	R_block (SWE)	P_ask (SWE)	Ask-F1 (SWE)	Pass@3 Full→Ask
GPT-5.3-Codex	15.0%	25.0%	18.8%	87%→5%	25.5%	69.2%	35.7%	72%→3%
GPT-5.4-Pro	23.0%	38.0%	28.7%	86%→17%	30.8%	60.6%	37.9%	64%→2%
Gemini 3.1 Pro	60.0%	47.0%	52.7%	89%→34%	38.2%	56.6%	41.6%	88%→7%
Claude Opus 4.6	62.0%	61.8%	62.0%	91%→38%	35.8%	27.7%	28.2%	69%→12%

Characteristic “fingerprints” appear in precision–recall space:

• GPT-family: High precision, low recall (overconfident; rarely asks, silent errors).
• Gemini (SQL): High recall, moderate precision (broad escalation, many irrelevant asks).
• Claude (SQL): Balanced high/high (well-calibrated judgment). On SWE, Claude’s precision drops sharply; Gemini is consistently mid-quality, and GPT variants under-ask (Elfeki et al., 10 Apr 2026).

5. Shaped RL Reward and Transfer Experiments

Direct reinforcement of Ask-F1 as a reward is too sparse for learning, so HiL-Bench uses a shaped setup:

• Per-step reward: +0.3 for each relevant question; −0.1 for irrelevant/duplicate.
• Terminal reward: $Q = \{q_1, \ldots, q_n\}$4 if at least one blocker is found.

A Qwen3 32B model trained in this regime improves from Ask-F1 ≈ 20–30% to 50–60% in both SWE and SQL domains, with process gains translating into +15–20 pass@3 points. Cross-domain training confirms that the learned skill of “knowing when to ask” transfers, with +5–10 Ask-F1 points when the agent is evaluated in a new domain (Elfeki et al., 10 Apr 2026).

6. Interpretation, Usage Guidelines, and Limitations

Interpreting Ask-F1:

• Near-zero values indicate either persistent silent guessing (recall failure) or indiscriminate question spam (precision failure).
• Judicious human-in-the-loop agents typically reach Ask-F1 of 50–70% on both axes in these controlled scenarios.

Guidelines:

• Always report precision and recall of help-seeking alongside overall task success.
• Optimize for Ask-F1—not only pass@k—so as to diagnose hidden failure modes.
• Use progressive-discovery tasks to elicit and measure true judgment capacity.

Limitations:

• HiL-Bench focuses strictly on coding/text-to-SQL; other domains (dialogue, planning, physical robotics, scientific workflows) may present different blocker dynamics.
• The ask_human() judge is fixed and may not capture subtler nuances of question phrasing quality or real human efficiency.
• High Ask-F1 in these controlled benchmarks does not imply perfect escalation in arbitrary real-world settings, where blockers can be fuzzier or less discretely measurable (Elfeki et al., 10 Apr 2026).

7. Significance and Future Directions

Ask-F1 provides a direct, gamed-resistant view into model-level help-seeking judgment, revealing a large qualitative gap in current LLM-based agents between acting autonomously and knowing when to defer to human input. The shaped reward design enables reinforcement learning of this meta-cognitive skill, and cross-domain transfer underscores its potential as a generalizable foundation for scalable, safety-aware agent deployment.

Open research questions include extending Ask-F1-style process metrics to more complex, less machine-verifiable domains, integrating human time or annoyance into the evaluation, and developing end-to-end agents capable of nuanced, adaptive escalation policies beyond discrete question events (Elfeki et al., 10 Apr 2026).