LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

Abstract: LLM agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.

• The paper introduces LoongReflect, which treats reflection as memory control over reversible trajectory trees and combines structured diagnosis, backtracking, and compressed working memory.
• The method pairs answer-masked teacher distillation with terminal-reward GRPO and look-ahead gradient coordination, achieving average F1 gains of 12.60 points on Qwen2.5-3B and 12.61 points on Qwen2.5-7B over AgenticRAG-R1.
• Ablations show that reinforcement learning, reflection, backtracking, and update coordination each contribute substantially, while limitations include retrieval dependence, exact-match reward mismatch, and single-seed evaluation.

LoongReflect addresses a structural weakness in how LLM search agents learn to reflect over long horizons. The authors observe that reflective decisions—whether to continue, revise, or abandon the current execution branch—are made from branch-local context, whereas their value is only revealed through the final trajectory outcome. This local–global mismatch leaves outcome-based reinforcement learning (RL) with sparse, delayed, and weakly attributable supervision for reflection, while assigning reflection an explicit intermediate reward invites reward hacking. LoongReflect responds by formulating reflection as an explicit memory-control policy and training it through two complementary channels coordinated by a look-ahead update (2608.11967).

Reflection as memory control

The framework models the agent state as a tuple consisting of the task, a trajectory tree $\mathcal{T}_t$, an active path $P_t$, and compressed working memory $\mathbf{m}_t$. Repeated recovery from unreliable states naturally induces a tree structure: backtracking archives the abandoned suffix as an inactive branch while resuming from a validated prefix. Only the active path and its compressed memory are exposed to subsequent generation, which prevents contaminated states from influencing future decisions while preserving archived branches for diagnosis.

Two control actions instantiate this mechanism. The <reflect> action produces a structured summary comprising verified evidence, branch-specific risks, a candidate return point, and a control intent (continue or backtrack). When the active state is deemed unreliable, <backtrack> rolls the trajectory back to a trustworthy checkpoint, removes the invalidated suffix from the active context, and writes a compact forward-looking corrective update into working memory. Reflection is thereby cast as state-control diagnosis rather than unconstrained verbal critique, distinguishing the approach from inference-time self-correction methods such as Self-Refine and Reflexion.

Two-channel optimization

Learning proceeds through a fast channel and a slow channel. The fast channel distills reflective behavior from a privileged teacher—an EMA copy of the policy that receives an answer-masked structured hint constructed from the full trajectory tree and the binary terminal outcome. Supervision is restricted to tokens inside <reflect> and <backtrack> spans via a masked reverse-KL objective (a k3-style estimator with clipping), providing dense token-level signal for state diagnosis and recovery decisions. Answer masking is central: because every answer span and alias is replaced with neutral tokens before teacher scoring, the fast channel supervises local diagnosis rather than answer imitation.

The slow channel applies GRPO over complete trajectories with group-relative advantages derived from terminal exact-match rewards, applied only to policy-generated tokens on the final active execution. This ensures reflection is credited only when it improves task success.

Because the two update directions may conflict, LoongReflect introduces look-ahead coordination in an extragradient style: $K$ provisional fast updates produce a candidate policy $\widetilde{\theta}$; the slow gradient is evaluated at $\widetilde{\theta}$; and if the fast direction has negative inner product with the slow direction, its opposing component is projected out before the fused outer update is committed. The coordination calibrates rather than dilutes the fast signal, preserving locally aligned updates while suppressing those that conflict with trajectory-level outcomes.

Main results

Training uses HotpotQA and 2WikiMultiHopQA; evaluation spans seven QA benchmarks plus MATH and GSM8K, with Qwen2.5-3B and 7B backbones under a shared retrieval configuration (November 2023 Wikipedia snapshot, identical retriever, top-$k$, budgets, and normalizer).

Backbone	LoongReflect avg F1	Best baseline (AgenticRAG-R1)	Margin
Qwen2.5-3B	46.15	33.55	+12.60
Qwen2.5-7B	49.21	36.60	+12.61

LoongReflect achieves the highest F1 on every benchmark at both scales, including out-of-domain transfers such as NQ (55.37 vs. 37.15 for 3B) and TriviaQA (73.72 vs. 56.62). On 3B, out-of-domain average rises from 31.59 to 43.77 relative to AgenticRAG-R1, indicating that explicit reflection generalizes beyond the training distribution rather than fitting in-domain tasks.

Transfer to non-retrieval reasoning is positive but modest: on Qwen2.5-3B, LoongReflect reaches 56.0 F1 on MATH and 82.4 on GSM8K, improving over AgenticRAG-R1 by 1.2 and 1.8 points and over RLSD by 2.4 and 1.7 points. These gains suggest the learned diagnosis-and-recovery behavior carries over to multi-step reasoning without external retrieval, though the margins are considerably smaller than on QA.

Stage-wise analysis attributes most of the improvement to RL: SFT alone adds 4.43 points (3B) over the raw backbone, while two-channel RL adds a further 11.39 points, for a total pipeline gain of 15.82 points. Component ablations confirm each element's contribution: removing <reflect> costs 15.31 average F1 points, removing <backtrack> costs 13.06, removing slow outcome optimization costs 7.04, removing fast distillation costs 5.64, and removing look-ahead coordination costs 4.94. Diagnosis alone is therefore insufficient without an explicit recovery mechanism, and dense local guidance must be paired with trajectory-level calibration. Training dynamics show task reward rising while the distillation reward moves from strongly negative toward zero, consistent with the intended division of labor between channels.

Hyperparameter sensitivity is nontrivial: with $w=1$, performance peaks at $K=3$ inner fast updates (46.15) and declines at $K=4$ (44.71); with $P_t$0, $P_t$1 outperforms both $P_t$2 and $P_t$3. Too few inner updates provide insufficient local adaptation, while excessive fast weight can dominate the outcome-aligned direction.

Limitations and open questions

The authors concede several dependencies. The method relies on retrieval quality and on controller judgments of checkpoint selection and recovery; errors in early diagnosis can propagate through later search and answer decisions. The normalized exact-match terminal reward can under-credit valid semantic aliases that receive partial credit under evaluation-time token F1, creating a train–evaluation mismatch. The EMA teacher's binary terminal outcome may shape the style of local diagnoses in ways not separately quantified. Empirically, results are reported from a single recorded seed configuration across two training datasets, and the coordination hyperparameters ($P_t$4, $P_t$5) were selected on the same benchmarks used for evaluation, leaving open whether adaptive schedules or held-out selection would generalize. Specific open questions include measuring robustness to retrieval noise, quantifying checkpoint-selection accuracy, analyzing error propagation through compressed updates, and extending evaluation beyond QA and mathematical reasoning.

Conclusion

LoongReflect reframes long-horizon reflection as explicit memory control over a reversible trajectory tree and trains this policy by combining answer-masked privileged distillation with outcome-based GRPO, coordinated through a look-ahead projection that removes conflicting components of the local update. The approach yields consistent double-digit improvements over strong RL-based agentic RAG baselines across seven QA benchmarks at two model scales, with ablations confirming that structured reflection, reversible backtracking, dense local supervision, global outcome optimization, and their coordination each contribute measurably. The principal caveats—dependence on retrieval quality and controller correctness, single-seed reporting, and modest transfer gains outside retrieval settings—define the boundaries within which these results should be interpreted.