- The paper introduces a trajectory-aware evaluation of multi-round RAG stopping that separates candidate reachability, state ranking, first-threshold decisions, answer quality, and retrieval cost.
- The structured S2G-style judge reduced Search-R1 retrievals by 77 calls, or 3.70%, lowering mean searches from 2.60125 to 2.50500 while keeping the 0.625-point EM loss within the prespecified two-point non-inferiority margin.
- The policy’s early-stop decisions remained risky, with 27 of 69 selected stops unsafe—a 39.13% unsafe fraction—showing that improved ranking and average accuracy do not guarantee calibrated stopping safety.
Motivation and problem statement
Multi-round retrieval-augmented generation (RAG) systems such as Search-R1 interleave reasoning with search-engine calls, but they must also decide when to stop searching. Retrieving too little leaves answers unsupported; retrieving after a sufficient state wastes compute and can even flip a correct candidate to an incorrect one as additional context enters the prompt. The paper's central methodological observation is that stopping is a sequential selection problem, not an independent state-classification task: the deployed policy is determined by the first threshold crossing on each trajectory, so one early false positive suppresses all later states, while an overly conservative classifier changes nothing at all. State-level metrics such as average precision therefore do not by themselves explain why a deployed stopping policy succeeds or fails.
The study asks four questions: how often a strong search policy continues after a correct answer is already reachable; how much quality and cost headroom exists when only the stopping time may change; whether structured sufficiency-and-gap supervision improves state ranking and the earliest-stop policy; and what answer risk accompanies the retrieval reduction that survives confirmatory evaluation.
Positioning relative to prior work
The paper adapts the structured judging mechanism of S2G-RAG — which predicts both sufficiency and the missing information needed for the next retrieval — to a frozen Search-R1 pipeline (Jin et al., 12 Mar 2025). It differs from SIM-RAG's answer-level continue/stop controller (Yang et al., 5 May 2025), from Adaptive-RAG's pre-trajectory question routing (Jeong et al., 2024), and from training-free stopping approaches such as TASR (Kieback et al., 11 Jun 2026). Notably, a pilot using SIM-RAG's released checkpoint as an external controller produced eight harmful changes and zero rescues over 200 questions, which the authors retain only as exploratory evidence of cross-system transfer failure.
Crucially, the intervention surface is minimal: the 7B reasoner, E5 top-3 retrieval over Wiki-2018, prompting, and the four-search budget are frozen. The only change is whether a reachable state is accepted early. This makes all oracles conditional on the frozen interface rather than global upper bounds.
The evaluation chain separates five layers: candidate reachability bounds what stopping can achieve; state scores diagnose representation quality; the first threshold crossing determines the deployed decision; and answer quality, selected-stop risk, and retrieval cost are evaluated separately. A state zi,t​ carries the accumulated context and native history, and a frozen answer-only probe g maps (qi​,Ci,t​) to a candidate answer without re-entering the native trajectory. A safe early-stop opportunity exists when the native policy proposes another SEARCH while the probe already matches the official answer under benchmark normalization.
Two conditional oracles are defined diagnostically: a quality oracle that selects the earliest reachable correct probe answer, and a native-preserving cost oracle that applies earlier correct stops only where Native Search-R1 is already correct. Neither is deployable; their role is to bound achievable improvement.
Experimental design
The upstream system is the official Search-R1 Qwen2.5-7B PPO checkpoint, configuration-verified against reference EM values within Wilson 95% intervals (45.67% NQ, 46.00% HotpotQA in a 1,200-question audit). The main data use the first 1,000 HotpotQA distractor development questions, split into 700 training, 100 grouped-validation, and 200 reserve questions; the reserve was activated via a predeclared gate, yielding 900 training questions and 3,009 clean states. A Qwen3.5-2B judge was trained for three epochs to emit a JSON schema with a Boolean sufficient field followed by gap_items; supporting-fact labels were used only for one-way conflict filtering and never shown to the judge. The online policy uses the log-probability margin of the Boolean field at a frozen threshold (~7.875), chosen on grouped validation to maximize recall subject to STOP precision ≥ 0.90 and ≥ 10 predicted STOP states.
The design is explicitly confirmatory: indices 200–999 were reserved, a two-percentage-point EM non-inferiority margin and a paired search-reduction criterion were frozen before evaluation, and inference followed a blind sequence. The authors disclose one procedural nonconformance — a preblind script hashed the final-label file despite a no-read rule — rather than silently waiving it.
Stopping headroom in a strong search policy
On the exploratory H0 set (200 questions, 523 native SEARCH-preceding states), 27.3% of native SEARCH actions occur at states where the frozen probe already produces the correct official answer, spanning 70 of 200 questions. Sixty questions exhibit cost-only over-search and ten exhibit harmful over-search. The quality oracle reaches 51.5% EM versus native 46.5%, and the cost oracle removes 25.05% of searches while holding native EM fixed. Uniform budgets cannot realize this trade-off: answer-only EM rises monotonically from 18.5% (k=0) to 47.0% (k=4), so per-question adaptive stopping has genuine room that fixed budgets cannot capture.
Ranking does not determine the deployed policy
A mechanism ladder exposes a ranking–policy mismatch: a binary LoRA raises STOP AP from 0.732 to 0.828 yet collapses official EM from 36.5% to 31.5% by stopping too aggressively. Replacing the binary target with a minimal S2G target reduces wrong STOP decisions from 99 to 24 and premature STOP decisions from 18 to 4, recovering EM to 35.5%. Relative to Structured Base, however, the EM difference is only +0.015 with a paired 95% CI of [−0.010,0.040] — evidence for structured gaps as an anti-over-stopping mechanism, not system-level superiority.
Confirmatory results
Both frozen criteria pass on indices 200–999:
| Metric |
Native |
Expanded S2G |
Difference [95% CI] |
Criterion |
Verdict |
| Official EM |
0.44875 |
0.44250 |
−0.00625 [−0.01250, 0] |
CI lower > −0.02 |
PASS |
| Mean searches |
2.60125 |
2.50500 |
−0.09625 [−0.12000, −0.07375] |
CI upper < 0 |
PASS |
In absolute terms this is 77 fewer retrieval calls out of 2,081 (3.70%), with six questions flipping Native-correct → policy-wrong and one flipping the other way. The expanded judge also improves STOP AP over Structured Base by 0.03983 (95% CI [0.00571, 0.07228]), confirming better state ranking.
The reduction carries material stop risk
The positive result has a sharp boundary. The expanded policy makes 69 early stops (8.63% of questions), of which 42 are safe and 27 unsafe — a selected-stop unsafe fraction of 39.13%. The safe selections cover only 14.29% of the 294 questions containing a safe opportunity. State-level STOP precision falls from 0.9091 on grouped validation (based on just 11 predicted STOP states) to 0.6216 on the confirmatory set, so the validation precision constraint does not transfer. Calibration is weak: Brier score 0.2952 and ECE 0.2911 versus 0.1992 and 0.1179 for Structured Base, and a post-hoc operating point reaching ≥ 0.90 precision yields recall of only 0.0057. The 27 unsafe stops do not map one-to-one onto newly wrong final answers — many occur where Native was already wrong — underscoring that average-quality non-inferiority and conditional stop safety are distinct objectives.
Limitations and open questions
The paper is explicit about what its results do not establish. Accuracy is not unchanged or improved; the point estimate loses 0.625 percentage points. The policy is not risk-certified, and deployments requiring low conditional error among accepted stops would need separate risk-control procedures, more calibration data, or abstention. Fewer retrieval calls do not imply lower total inference cost, since judge compute, latency, and memory are not normalized against retrieval-equivalent units; matched-compute evaluation remains open. Scope is limited to HotpotQA distractor indices 200–999 under one frozen Search-R1 configuration, and the system result belongs to the complete combination of data, supervision, checkpoint, and threshold rather than any single component. Open questions include closing the gap between AP and calibrated decision utility via calibration-aware training, trajectory-level losses penalizing first unsafe crossings, and explicit cost-sensitive objectives — hypotheses the authors flag as untested.
Conclusion
Adapting an S2G-style structured judge to frozen Search-R1 yields a confirmatory, statistically supported retrieval reduction — 77 fewer calls (3.70%) with mean searches falling from 2.60125 to 2.50500 — while keeping the EM loss (0.625 percentage points) within a prespecified two-point non-inferiority margin. The contribution is as much methodological as empirical: the trajectory-aware framework separating reachability, ranking, first-crossing behavior, quality, and cost explains why state-ranking gains alone mislead, and why "broadly preserving" accuracy coexists with a 39.13% unsafe fraction among the policy's own early stops.