Robust and calibrated stop/continue criteria across retrievers, corpora, and LLM backbones

Develop stop/continue criteria for multi-hop question answering retrieval–reasoning procedures that remain calibrated across different retrievers, corpora, and large language model backbones, and evaluate these criteria under controlled variations of hop depth and retrieval noise to assess their reliability and transferability.

Background

The survey highlights that most multi-hop QA systems rely on static hop, token, or latency budgets to decide when to stop, while adaptive stopping based on confidence or sufficiency is evaluated only on limited benchmarks and is often poorly calibrated under distribution shift. This lack of robust stop/continue criteria leads to both over-search and under-search, harming reliability and efficiency.

The authors call for methods whose stop/continue decisions generalize across different retrievers, corpora, and LLM backbones, together with evaluation protocols that deliberately vary hop depth and retrieval noise to stress-test calibration. This problem sits within the broader axis of procedural design choices for retrieval–reasoning agents and directly impacts effectiveness, efficiency, and faithfulness.

References

An important open problem is to develop stop/continue criteria that remain calibrated across retrievers, corpora, and LLM backbones, and to evaluate them under controlled variations of hop depth and retrieval noise.

— Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends  (2601.00536 - Ji et al., 2 Jan 2026) in Section RQ4: Open Problems and Future Directions, Challenge 4

The routing ceiling is a finding about this setting, and the value of confidence-based routing where retrieval is weak or unavailable remains an open question we do not answer.

— Intrinsic Sequence-Likelihood Confidence in Retrieval-Dominated Extractive QA: Two Pre-Specified Negatives, and What They Do and Do Not Attribute  (2609.19942 - Lee et al., 17 Sep 2026) in Section 1, Introduction

Our evaluation draws on a single benchmark family (Cochrane-derived queries). While we mitigate the sample-size concern with a larger disjoint held-out union ($n=280$; extended version) and report macro-F1 with bootstrap intervals throughout, generalisation to other evidence sources (drug-label databases, trial registries such as ClinicalTrials.gov, or non-Cochrane systematic reviews) remains to be shown.

— When More Evidence Hurts: Publication-Bias Drift and Principled Stopping for Biomedical Causal Search  (2609.24101 - Sun et al., 21 Sep 2026) in Section 6, Discussion and Limitations

Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval.

— Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay  (2608.26846 - Chhikara, 27 Aug 2026) in Abstract; Section 5, RQ5: Is another evidence step useful?

What continuing would recover remains unmeasured.

— ABF-Bench: A Verifiable Benchmark for Agentic Blockchain Forensics  (2610.23656 - Yu et al., 16 Oct 2026) in Section 7.1, paragraph discussing early stopping and unused budget