- The paper introduces RGLT, a latent-reasoning retriever that transfers stage-wise retrieval-effect signals from explicit chain-of-thought to silent-token trajectories, improving BRIGHT nDCG@10 to 34.20 and Recall@10 to 39.97.
- RGLT combines instruction-conditioned latent anchors, intermediate multi-positive contrastive supervision, reconstruction of cumulative reasoning stages, and retrieval-effect credit matching to ensure each reasoning step improves document ranking.
- The method preserves single-vector dense retrieval and unchanged document indexing, achieving a 1.12× latency overhead over the base retriever versus roughly 186× for rewrite-then-retrieve, while performance peaks at 16 silent tokens.
Motivation and problem statement
Reasoning-intensive retrieval requires query representations that encode inferred evidence rather than literal semantics: in domains such as mathematics, programming, and science, relevant documents may share little surface similarity with the query. Explicit chain-of-thought (CoT) approaches address this by generating a rationale before encoding, but incur autoregressive decoding latency and couple retrieval quality to the verbosity of the generated text. Latent-reasoning retrievers such as LaSER internalize explicit CoT into hidden states, yet their supervision is dominated by final retrieval objectives or state alignment between explicit and latent trajectories. The authors identify a specific gap in this design: intermediate latent transitions are never supervised for whether they produce incremental retrieval gains, so latent trajectories can converge to shortcut reasoning patterns that preserve final retrieval quality without meaningful stage-wise improvement.
The RGLT framework
RGLT (Retrieval Grounding Latent Reasoning) performs non-autoregressive reasoning inside the encoder by appending K fixed silent tokens to the instruction-conditioned query; the entire latent trajectory is computed in a single forward pass, and the terminal silent token's hidden state serves directly as the single-vector query embedding. Two architectural components shape the trajectory. First, an instruction-conditioned anchor: the query and instruction are compressed into an anchor representation that modulates the keys and values of every silent token at each layer via a gated low-rank residual, ensuring the trajectory reflects the retrieval intent under different instructions while document encoding remains untouched (preserving dual-encoder index reusability). Second, stage-wise supervision: the K silent tokens are partitioned into S stages matched to cumulative prefixes of an external LLM-generated CoT.
Training combines three losses. A process-supervised reconstruction loss requires each latent stage to decode its cumulative CoT prefix under teacher forcing, with information-leakage prevention by exposing only tokens up to the current stage. A direct multi-positive contrastive loss applies retrieval supervision not only to the final representation but also to intermediate stage states, preventing early steps from drifting outside the retrieval space. The central contribution is the retrieval-effect credit loss: for each adjacent pair of reasoning stages, the change in log-probability over positives versus hard negatives within a normalized local candidate distribution is computed separately for the latent trajectory (Δk) and the explicit CoT trajectory (Δk∗). Both credits are zero-mean centered under the explicit preceding-state distribution, and the latent credit is regressed onto the stop-gradient explicit target. Stop-gradients on the preceding latent state prevent the model from inflating transition gains by degrading earlier states. Crucially, the objective constrains ranking effects, not representation geometry — replacing it with conventional state alignment measurably hurts performance.
Main results
On BRIGHT, RGLT with a Qwen3-8B backbone achieves 34.20 nDCG@10 and 39.97 Recall@10, improving over LaSER by 14.4% and 16.6% respectively, exceeding GIRCSE by 8.20 nDCG points and autoregressive Rewrite-then-Retrieve by 6.10 points, all on identical training data and backbone. Domain-level analysis shows gains on 9 of 12 BRIGHT domains and higher Recall@10 on 11 of 12, indicating balanced rather than outlier-driven improvement; LaSER retains advantages on biology, earth science, and StackOverflow. On FollowIR, RGLT attains the best average score (13.70) among compared models, though its Core17 MAP@5 (4.60) falls below several baselines — a result the paper does not discuss in depth. On BrowseComp-Plus it leads on Recall@5/100/1000.
Ablations confirm each component matters: removing stage or intent retrieval-effect matching drops nDCG@10 to 31.26 and 31.69 respectively, and substituting state alignment yields 30.64. A naive terminal-silent-token readout without effect grounding reaches only 30.87, showing that the trajectory structure alone is insufficient. Stage-terminal analysis demonstrates progressive accumulation of retrieval utility: nDCG@10 rises monotonically from 26.39 (base state) through ST4–ST16 (32.25 → 34.20), directly supporting the claim that intermediate transitions contribute incremental gains rather than serving as mere scaffolding.
Efficiency
RGLT preserves single-vector nearest-neighbor search with no document-side modification. Measured latency is 24.0 ms/query versus 21.5 ms for the base retriever — a 1.12× overhead — compared with 1.51× for LaSER's recurrent latent construction and roughly 186× for explicit rewrite-then-retrieve pipelines. The supplement attributes this to KV-cache reuse across the anchor and latent branches, strict positional-integrity handling, and concurrent processing of all silent tokens in one causal block, with cached inference validated to produce rankings identical to full recomputation.
Limitations and open questions
Several constraints are acknowledged or evident. Performance does not scale indefinitely with the latent budget: metrics peak at K=16 and degrade at K=24, suggesting optimization difficulty at larger budgets whose cause remains unexamined. Stage segmentation relies on regular-expression parsing of LLM-generated CoT with fallback heuristics near equal-length partitions, so supervision quality depends on the segmentability of generated rationales. The method requires an external LLM to produce training-time CoT and assumes those rationales contain genuinely retrieval-improving stages; the credit-transfer mechanism inherits any noise in them. The Core17 regression on FollowIR is unexplained. Finally, evaluation covers reasoning-intensive benchmarks; behavior on conventional semantic retrieval tasks is not reported, leaving open whether retrieval-effect grounding transfers beyond this regime.
Conclusion
RGLT reframes latent-reasoning supervision for dense retrieval around stage-wise ranking effects rather than representation alignment or final-outcome objectives. By transferring per-stage retrieval credits from explicit CoT to latent transitions, it produces trajectories whose utility accumulates progressively, achieving state-of-the-art BRIGHT results with a 1.12× latency overhead and unchanged document indexing. The results indicate that grounding intermediate latent computation in measurable document-ranking improvements is a more effective supervision principle than aligning latent states with explicit ones.