---
title: 'RecaLLM: Robust Retrieval for Long Contexts'
url: https://www.emergentmind.com/topics/recallm-1bb57b02-8319-47c5-b9dd-152dceda833e
type: topic
---

# RecaLLM: Robust Retrieval for Long Contexts

RecaLLM denotes a class of post-trained large language models (LLMs) designed to explicitly interleave chain-of-thought (CoT) reasoning with robust, verifiable in-context retrieval. The approach targets the “lost-in-thought” phenomenon, wherein the act of multi-step reasoning degrades the model’s ability to access and verbatim retrieve evidence from long contexts. By enforcing alternation between reasoning and copying of evidence via constrained decoding, RecaLLM achieves state-of-the-art long-context performance with negligible computational overhead and minimal dependence on extremely long training samples, scaling to context lengths up to 128K tokens despite training on windows of at most 10K tokens [2604.09494].

## 1. Motivation: The Lost-in-Thought Phenomenon

RecaLLM was introduced to address a key bottleneck observed in long-context LLMs: as reasoning traces lengthen, faithful in-context retrieval performance deteriorates substantially. This “lost-in-thought” effect is quantified as a stark drop in retrieval accuracy after any sequence of reasoning tokens, even when the retrieval task that follows would be trivial in isolation:
- Let \(A_{\mathrm{direct}}\) be the accuracy of a direct key-value retrieval, and \(A_{\mathrm{reason}}\) the accuracy when retrieval is requested after a reasoning sequence of length \(L_r\).
- Empirically, for several open-source 7–8B parameter models (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, ProLong-8B-512K, etc.), \(A_{\mathrm{reason}}\) fell from ~80% to ~40% after only a short CoT trace at 4K context, and from ~25% to ~5% at 128K [2604.09494, Table 1].

Injection studies confirmed that even forcibly re-exposing the model

Source: https://www.emergentmind.com/topics/recallm-1bb57b02-8319-47c5-b9dd-152dceda833e