TASR: Training-Free Adaptive Stopping for Iterative Retrieval

Abstract: Iterative retrieval-augmented generation agents commonly overspend by continuing to retrieve after the model has converged on an answer, incurring calls that change neither the prediction nor the supporting evidence. Existing remedies learn a stopping policy from labeled trajectories, tying the decision to a trained component that requires retraining for each new model or task. We propose TASR (Training-Free Adaptive Stopping Rule), a one-line predicate that fires when the model repeats its previous-round normalized answer and the isotonically calibrated logit margin exceeds 0.25. No classifier or value head is learned; the threshold is fixed across all twenty-four (model, retriever, corpus) configurations we evaluate. On a 3-model x 2-dataset distractor grid, TASR retains 94.8% of fixed-k=5's macro F1 at 62.6% of its calls and exceeds fixed-k=3 by +3.42 F1. The pattern holds on nine open-domain BM25 cells (55.01 F1 at 2.98 calls vs. 54.33 at 3.00 for fixed-k=3) and, with calibration locked from the distractor split, on nine dense-retrieval cells across two retriever families, with zero significant regressions in either extension. The rule was selected from an exhaustive enumeration of 381 candidate stopping rules; no alternative Pareto-dominates it on any evaluated configuration. A signal-quality analysis shows that verbalized 1-5 confidence collapses on RLHF-tuned models (96.5% of values equal 5, entropy 0.182 nats), while the logit margin achieves 44x better class-conditional separation, grounding the design in a measurable model pathology. TASR is an auditable, training-free Pareto baseline against which learned stopping controllers can be compared. Code is publicly available.

• The paper introduces TASR, a training-free adaptive stopping rule that halts iterative retrieval when observed answer repetition and a calibrated logit margin exceed a fixed threshold.
• The method applies per-round isotonic calibration to differentiate correct from incorrect answers, achieving over 3 nats of separation and 44× improvement over verbalized confidence.
• Empirical results demonstrate that TASR improves macro F1 scores and reduces retrieval calls compared to fixed-budget baselines, especially benefiting weaker single-pass models.

TASR: A Training-Free Adaptive Stopping Rule for Iterative Retrieval

Problem Motivation and Context

Retrieval-Augmented Generation (RAG) systems alternate evidence retrieval and answer generation via LLMs, incurring computation and monetary costs for each retrieval and LLM call. Iterative RAG agents frequently "overspend" by continuing to retrieve after their answers have stabilized, leading to redundant computations that do not improve performance but still incur significant cost. Current approaches to mitigating this inefficiency involve learned stopping policies—often trained classifiers or value heads—which demand labeled trajectories and retraining for every new model or retrieval setting. These methods increase complexity and reduce auditability, motivating the development of a robust, training-free, and transparent alternative.

TASR Rule: Design and Mechanism

The paper introduces TASR (Training-Free Adaptive Stopping Rule), a simple and universal one-line predicate to determine when to halt iterative retrieval. TASR triggers if and only if the model (1) repeats its normalized answer from the previous round and (2) the isotonically calibrated logit margin at the commit token exceeds a fixed threshold ($0.25$):

$$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$

where $\tilde{a}_r$ is the normalized answer in round $r$, and $m_r$ is the calibrated logit margin interpreted as the probability of being correct.

This rule needs only two cached scalars per round and requires no retraining between models, retrieval paradigms, or datasets, making it maximally auditable and extensible.

Figure 1: The TASR iterative loop, showing when the stopping condition is met as the agent alternates retrieval and answer generation.

Calibration and Signal Diagnostic

A key insight is that verbalized LLM confidence, especially for RLHF-tuned models, collapses (e.g., $>96\%$ of reported confidences being the maximum value), causing it to be uninformative for stopping. In contrast, the logit margin at the answer-commitment token provides a continuous signal with strong class-conditional separation: the mean logit margin for correct answers is over $3$ nats higher than for errors ($6.00$ vs $2.91$), resulting in $44\times$ better class separation than verbalized confidence.

Isotonic regression, applied per-round, translates this margin into a calibrated probability of correctness, explicitly accounting for the growing evidence set as iterations proceed. This per-round approach is superior in practice for threshold-based decisions, even when it slightly worsens Brier score compared to global calibration.

Figure 2: Distribution of logit margins by answer correctness, demonstrating substantially higher separation for correct vs wrong answers and monotonic informativeness of the margin.

Empirical Evaluation

Main Results

Across a $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$0-model $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$1 $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$2-dataset grid with HotpotQA and 2WikiMultiHopQA, TASR yields a macro F1 of $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$3 at $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$4 calls per query, outperforming fixed-$$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$5 by $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$6 F1 at similar cost and retaining $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$7 of fixed-$$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$8's F1 at only $$\mathrm{stop}_r \;=\; \mathbf{1}\!\left[\, \tilde{a}_r = \tilde{a}_{r-1} \;\wedge\; m_r > 0.25 \,\right]$$9 of its call budget. These gains are statistically significant in half of the grid cells and never result in significant regressions.

Figure 3: Macro F1 as a function of average number of calls, showing the TASR frontier versus fixed-$\tilde{a}_r$0 baselines on both distractor and open-domain settings.

A per-model analysis reveals the largest gains for weaker single-pass models (notably Gemma-4-31B-IT), indicating the rule's particular benefit when answer convergence is less certain and additional retrieval rounds are more likely to refine the prediction.

Figure 4: Per-model breakdown of F1/call tradeoffs, with Gemma showing the greatest improvements via TASR; other models realize smaller gains or ties.

Generalization and Robustness

TASR maintains its Pareto dominance on nine open-domain BM25 cells (including HotpotQA-fullwiki, NQ-Open, and TriviaQA-Open), as well as nine dense-retrieval cells spanning Contriever-MSMARCO and DKRR-TQA retrievers. Calibration parameters, locked from the distractor setting, are directly transferable—demonstrating that the stopping signal is driven by model internals rather than retrieval specifics.

Signal Search and Negative Results

The authors exhaustively enumerate 381 possible stopping signal rules, testing all OR-combinations of cached scalars involving answer stability, logit margin, token overlap, reranker scores, and other signals. No alternative outperforms the conjunction of answer stability and calibrated margin. Notably, adding snapshot signals through OR logic leads to pathological early stopping, while AND-conjunctions collapse to the most restrictive term (which is the TASR predicate itself).

Alternative signals (including reranker confidence, BM25 $\tilde{a}_r$1-scores, continuous answer similarity, and judge-based entailment scores) are either less robust, fail to generalize, or actively degrade performance due to misalignment with actual decision fidelity.

Model Pathology and Calibration Analysis

Experiments with model variants (Qwen, Devstral, Gemma) show that Gemma's overconfident raw logit margins (mean $\tilde{a}_r$2 nats vs. $\tilde{a}_r$3 for Qwen) are successfully re-mapped by isotonic calibration, enabling TASR to select more retrieval rounds only where they bring concrete F1 gains. This calibration approach, taking model-specific margin distributions as input but using a locked threshold, is central to TASR's transferability and robustness.

Prompt Sensitivity

Prompting regime influences which signal provides the best convergence indicator. When $\tilde{a}_r$4--$\tilde{a}_r$5 confidence is elicited before answer commitment, its informativeness improves (higher entropy), and calibrated confidence surpasses answer stability as a stopping signal. Nonetheless, single-signal performance dominates; combinations via OR logic again hurt F1, and the fundamental rule structure (AND over convergence and stability) is retained.

Figure 5: Distribution and entropy of verbalized confidence scores under canonical and "confidence-first" prompt orders, showing significant collapse with standard prompts.

Implications and Future Directions

The findings establish TASR as a strong, interpretable, and auditable baseline for adaptive stopping in RAG agents, with implications for cost control, energy efficiency, and architecture-agnostic evaluation. The rule's simplicity emerges not from regularization or bias but from an exhaustive search over plausible signal combinations, supporting the view that answer stability and model-calibrated convergence are the universal core of the iterative stopping decision.

On a theoretical level, these results suggest that model-side signals dominate retriever-side signals for trustworthy stopping and that answer stability serves as a general indicator of trajectory convergence. Future developments should evaluate if these properties generalize to tasks outside multi-hop QA (e.g., fact verification, long-form synthesis), to dialog systems, and under different retrieval schemas.

The study also highlights the need to move away from standard calibration metrics (e.g., Brier score) toward metrics directly tied to the operational decision boundaries of threshold-based controllers.

Conclusion

TASR demonstrates that a single, universally-applicable, calibration-driven predicate outperforms both static call-budget baselines and more complex, learned stopping policies across diverse tasks, models, and retrieval environments. The rule's auditability, strong empirical performance, robustness to transfer, and independence from retraining make it a practical tool and a new standard baseline for future research into RAG agent efficiency and control (2606.13814).