Lost-in-Thought in LLM Reasoning

• Lost-in-Thought is a term describing failures in the temporal allocation of attention in LLMs, evidenced by U-shaped positional biases and retrieval bottlenecks.
• Empirical studies reveal that performing intermediate reasoning steps can drop retrieval accuracy by 60–90% when models exceed their optimal chain-of-thought length.
• Interventions such as rebalanced training objectives, explicit in-context retrieval, and rollback corrections have shown promise in mitigating these performance drops.

Searching arXiv for papers on “Lost-in-Thought” and closely related long-context/reasoning positional-bias work. “Lost-in-Thought” is used in recent research to denote several related but non-identical phenomena at the interface of long-context language modeling, chain-of-thought reasoning, and cognition. In LLMs, the phrase has been applied to a U-shaped positional bias in which performance is high at the beginning and end of a sequence but weak in the middle; to a retrieval bottleneck in which a few reasoning steps sharply degrade subsequent in-context retrieval; and to a decline in accuracy when a reasoning model is pushed beyond its own optimal chain length and falls into unproductive rumination loops. In cognitive neuroscience, being “lost in thought” refers instead to spontaneous thought that arises relatively freely under weak cognitive constraints (Salvatore et al., 11 Oct 2025, Whitecross et al., 10 Apr 2026, Marjanović et al., 2 Apr 2025, Andrews-Hanna et al., 2017).

1. Terminology and conceptual scope

A central difficulty in the literature is that “Lost-in-Thought” is not a single standardized technical term. In "Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs" it is treated as the lost-in-the-middle effect: recall or prediction accuracy over item position $i$ in a sequence of length $L$ follows a U-shaped curve, with primacy at early positions and recency at late positions. In "RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval" it names a failure mode in which chain-of-thought reasoning and in-context retrieval interfere, so that reasoning steps make later retrieval more difficult. In "DeepSeek-R1 Thoughtology: Let’s think about LLM Reasoning" it describes the counter-intuitive drop in performance that occurs when a reasoning-capable LLM is forced to think beyond its own optimal reasoning length, together with persistent rumination loops in its chain-of-thought (Salvatore et al., 11 Oct 2025, Whitecross et al., 10 Apr 2026, Marjanović et al., 2 Apr 2025).

These usages overlap in one respect: each concerns the control of information selection over time. The first centers on position within the prompt, the second on retrieval after intermediate reasoning, and the third on the length and structure of generated thought itself. A plausible implication is that the phrase now functions as a family resemblance term for failures in temporal allocation of attention, recall, or reasoning budget rather than as a single canonical benchmark construct.

2. Positional bias, serial-position curves, and the lost-in-the-middle interpretation

In the long-context formulation, recall or prediction accuracy as a function of item position $i$ in a sequence of length $L$ exhibits a characteristic U-shaped “lost-in-the-middle” curve. A phenomenological model is

$P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$

with

$$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$

When $\beta>0$ and $\gamma>0$, $P(i)$ is large at $i\approx1$ and $L$0, producing the U-shape. The paper attributes this pattern not simply to information loss but to different information retrieval demands during training: long-term memory demand requires uniform recall across the entire input, whereas short-term memory demand prioritizes the most recent information (Salvatore et al., 11 Oct 2025).

The long-term, free-recall-style objective requires recall of any of the $L$1 presented items $L$2 in any order and imposes equal retrieval probability across all positions $L$3. The short-term, running-span-style objective uses a cue at the end of the list to signal recall of the last $L$4 items, with effective sampling weight

$L$5

normalized so $L$6. The paper’s interpretation is that recency aligns directly with short-term demand, while primacy is induced by uniform long-term demand and is additionally influenced by autoregressive properties and attention sinks.

The experimental program uses GPT-2 Small & Large, Llama-3.2 1B, RNN-seq2seq, and T5, with sequence length $L$7, training on 100 k random lists for 25 epochs. Free Recall produces strong primacy, Running Span with $L$8 last items produces strong recency, and the Combined objective yields an emergent U-shaped Serial Position Curve. The same structure generalizes to a masked sequence completion task: uniform sampling of the reveal window induces primacy, recency-weighted sampling induces recency, and combined sampling produces a U-shape. In decoder-only models, causal masking encourages strong attention to early tokens and amplifies primacy, whereas bidirectional models such as T5 lack both causal engineering and primacy. The analysis further defines attention sinks as heads for which the average attention received by token $L$9 from later positions is extraordinarily high; disrupting these sink heads via dropout removes primacy but not recency, flattening the left flank of the U-curve (Salvatore et al., 11 Oct 2025).

The significance of this formulation is methodological as much as descriptive. It reframes positional bias as an emergent adaptation to mixed retrieval objectives rather than as a monolithic pathology. That shift affects both diagnosis and intervention, because it ties observed serial-position effects to training distributions, architectural constraints, and structural attention dynamics rather than to a single failure source.

3. Reasoning-induced retrieval failure and explicit in-context recall

A second, narrower usage defines lost-in-thought as a bottleneck for test-time scaling in long-context reasoning models. RecaLLM characterizes two intertwined capabilities in long-context LLMs: chain-of-thought reasoning, which generates intermediate semantically rich tokens, and in-context retrieval, the implicit ability to copy or attend to relevant spans anywhere in the prompt. The identified failure mode is that performing a few reasoning steps sharply degrades subsequent retrieval accuracy from the same context, even when the underlying reasoning remains correct (Whitecross et al., 10 Apr 2026).

To isolate this effect, the authors construct a synthetic key–value benchmark with two tasks: direct Retrieval, “What is the value for key $i$0?”, and Reasoning-Retrieval, “Solve a math problem whose answer is the key $i$1, then look up its value.” Context windows range from 4 K to 128 K tokens and are filled with distractor entries. The paper defines

$i$2

and reports that the relative accuracy drop often exceeds 60–90%. One concrete example is Llama-3.1-8B-Instruct, where Retrieval is approximately $i$3 at 4 K while Reasoning-Retrieval is approximately $i$4. A follow-up injection experiment shows that even when the correct key and its exact lexical prefix are injected mid-generation, models hallucinate the value approximately 60–80% of the time at long contexts. The authors interpret this as evidence that the bottleneck is not the loss of the correct key but the inability to faithfully copy context spans after generating semantically related reasoning tokens.

RecaLLM addresses this by interleaving reasoning with explicit in-context retrieval via recall spans delimited by two new tokens, $i$5 and $i$6. Outside recall spans, generation is standard autoregressive decoding. Inside a recall span, constrained decoding restricts continuations to a valid set derived from contiguous substrings of the input prompt concatenated with previous generation. This guarantees that each recall span is a verbatim contiguous substring of the accessible context, anchoring subsequent reasoning on faithful evidence. The post-training pipeline combines a supervised finetuning cold start on approximately 1.8 K teacher traces with verbatim recall spans and RL with GRPO on 20 K examples spanning 10 task categories. The reward function combines format, answer quality, and retrieval quality, with retrieval scored by F1 overlap between gold passage intervals and recalled spans, modulated by a density penalty and a correctness penalty.

Empirically, both RecaLLM variants substantially outperform their base models across 4 K–128 K contexts. On Reasoning-Retrieval, RecaLLM-Qwen rises from $i$7 at short contexts and from $i$8 at long contexts. On entity citation, RecaLLM-Llama rises from $i$9 at short contexts and $L$0 at long contexts. On RULER, Qwen2.5-7B improves from $L$1 to $L$2, with especially dramatic gains at 128 K, where baseline $L$3 becomes $L$4. On HELMET, the same baseline improves from $L$5 to $L$6, with strong gains on Recall, ICL, Re-rank, and citation. These results hold despite training only on contexts of at most 10 K tokens (Whitecross et al., 10 Apr 2026).

4. Optimal thought length, rumination loops, and overthinking

A third usage comes from the analysis of DeepSeek-R1 and similar large reasoning models. Here, lost-in-thought is defined by two interacting phenomena: test-time scaling beyond a “sweet spot” of thought length, and persistent “rumination” loops in chain-of-thought. If $L$7 denotes accuracy when generating chains of exactly $L$8 tokens, the model’s optimal reasoning length is

$L$9

Chains longer than $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$0 incur decreasing accuracy; this is presented as the empirical hallmark of the phenomenon. To quantify repeated reconsideration, the paper decomposes a chain into one initial bloom cycle and $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$1 reconstruction cycles, flags a reconstruction cycle as a loop when it re-examines a prior decomposition above a similarity threshold, and defines the rumination rate

$P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$2

High $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$3 indicates persistent rumination (Marjanović et al., 2 Apr 2025).

The experimental setup spans AIME-24, $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$4 multiplication with $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$5 up to 20, and supplementary benchmarks MATH500 and GSM8k. For each problem, the authors sample unrestricted reasoning chains with a token limit of 32 k and temperatures from 0.6 to 1.0; $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$6 chains per AIME-24 problem and $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$7 per multiplication pair. Accuracy is then examined as a function of chain length via bins. Across AIME-24 and medium-sized multiplication, performance curves rise from small $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$8, peak near $P(i)\;=\;\alpha \;+\;\beta\,f(i)\;+\;\gamma\,g(i),$9, and then fall sharply for very long $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$0. For AIME-24, after min-max normalizing $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$1 to $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$2 and dividing into five bins, the reported aggregate accuracies are approximately $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$3, $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$4, $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$5, $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$6, and $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$7, with the sweet spot in the $$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$8–$$f(i)\;=\;\exp\bigl(-\kappa\,(i-1)\bigr) \quad\text{(primacy kernel)},\qquad g(i)\;=\;\exp\bigl(-\kappa\,(L-i)\bigr) \quad\text{(recency kernel)}.$$9 bin. Table 4.3 further reports that correct chains are significantly shorter than incorrect chains: on AIME-24, approximately 4000 tokens versus approximately 8000 tokens; on GSM8k, 2500 versus 6000; on MATH500, 1500 versus 2500. Imposing a hard token budget such as $\beta>0$0 on GSM8k reduces inference cost by approximately $\beta>0$1 with only a $\beta>0$2 drop in accuracy.

The paper’s structural analysis annotates 400 chains into Problem Definition, Blooming Cycle, Reconstruction Cycles, and Final Decision. Problem Definition and Final Decision lengths are stable, but reconstruction dominates variation across tasks. Longer “re-bloom” reconstructions occur early, whereas many shorter “rumination” cycles repeatedly revisit the same decompositions verbatim. This behavior is interpreted as a meta-cognitive failure: continual re-checking without progress. The same study also associates longer reasoning with increased jailbreak susceptibility and harmful output, and therefore treats lost-in-thought not only as an efficiency failure but also as a safety-relevant one (Marjanović et al., 2 Apr 2025).

A common misconception is that more test-time reasoning is monotonically beneficial. The DeepSeek-R1 analysis explicitly contradicts that assumption: extra inference time can impair model performance, and the decline is not confined to final-answer cost but extends to interpretability and safety.

5. Interventions, training objectives, and rollback-based correction

The proposed remedies differ across formulations because the failure mechanisms differ. For the U-shaped positional-bias setting, the suggested interventions target the training objective and attention mechanism. These include objective rebalancing through position-agnostic losses or oversampling middle tokens to flatten $\beta>0$3 and $\beta>0$4; learnable positional embeddings with annealed decay; dropout or regularization of sink heads only when long-range recall is unnecessary; bidirectional or cross-segment attention layers for global context; mixed-demand pre-training that explicitly alternates uniform and recency-biased tasks; and monitoring SPC, PFR, and CRP on hold-out synthetic lists to detect emerging positional biases early in training (Salvatore et al., 11 Oct 2025).

For overthinking and rumination, the recommended remedies are budget control and process monitoring. One dynamic termination criterion is to stop generation when marginal expected improvement falls below a threshold $\beta>0$5,

$\beta>0$6

The corresponding budget-aware RL proposal augments the reward with a length penalty,

$\beta>0$7

where $\beta>0$8 is the target budget and $\beta>0$9. On the CountDown arithmetic task, models trained with a “MaxDiff” length reward adhere closely to budgets while retaining $\gamma>0$0 of baseline accuracy. For long-context tasks in which DeepSeek-R1 becomes overwhelmed and produces incoherence or language drift, the same source suggests sliding windows or retrieval-augmented pipelines and proposes an explicit process-monitoring head that predicts the utility of additional thought steps. Early research on “Meta CoT” is cited as pointing in this direction (Marjanović et al., 2 Apr 2025).

A separate corrective framework, Thought Rollback (TR), addresses forward-only reasoning under hallucinations by allowing rollback to previously mistaken thoughts. TR replaces rigid chains, trees, or DAGs with a directed graph that includes backward edges $\gamma>0$1 for $\gamma>0$2. After generating thoughts $\gamma>0$3, TR invokes an error-analysis prompt to identify erroneous steps, rolls back to the earliest flagged error, appends the error analysis as “experience,” and regenerates from the earlier prefix. When $\gamma>0$4 complete answers are produced, TR applies weighted voting that favors paths with fewer outgoing rollbacks and more incoming rollbacks. On the MATH dataset with GPT-4, TR + W-Voting reaches $\gamma>0$5 at approximately 62 interactions, and TR + CoT$\gamma>0$6 + W-Voting reaches $\gamma>0$7; the abstract states that the solving rate of GPT-4 with TR outperforms the current best by $\gamma>0$8 on MATH. The same framework reports $\gamma>0$9 on MMLU with GPT-4 at 28 calls, compared with BoT at $P(i)$0 but $P(i)$1 calls. In the paper’s own summary, TR is presented as closing the gap of “lost-in-thought” by enabling the model to “think back,” diagnose errors, and reopen new reasoning branches guided by accumulated experiences (Chen et al., 2024).

Taken together, these interventions show that “Lost-in-Thought” is not countered by a single universal fix. Some variants require rebalancing positional supervision, some require explicit retrieval channels, some require early exit and budget-aware control, and some require cyclic reasoning with revision rather than deeper forward expansion.

6. Human spontaneous thought and adjacent phenomena

The phrase “lost in thought” has an older and broader meaning in cognitive science that should not be conflated with the LLM failure modes above. Christoff et al. define spontaneous thought as “A mental state, or a sequence of mental states, that arise relatively freely due to an absence of strong constraints on (a) the contents of each state and (b) the transitions from one mental state to another.” This framework distinguishes spontaneous from deliberate initiation, automatic from deliberate constraints on content, and emphasizes dynamic flow, including temporal variability, dwell time, and topic transitions. In neuroscientific terms, work summarized in the field review links spontaneous thought to the default mode network, variable engagement of the frontoparietal control network, and dynamic transitions that can be represented in a generic neural systems equation $P(i)$2, where spontaneous transitions correspond to shifts driven primarily by noise $P(i)$3 and reduced constraint from inputs $P(i)$4 (Andrews-Hanna et al., 2017).

This human literature matters because several LLM papers invoke cognitive parallels, especially primacy, recency, overthinking, and perseveration. The parallels are suggestive but not identical. The DeepSeek-R1 analysis explicitly compares lost-in-thought to human “overthinking” or perseveration, whereas the spontaneous-thought literature stresses the relaxation of constraints and the distinction between deliberate and automatic control. A plausible implication is that the same phrase can denote almost opposite conditions: in neuroscience, freer transition dynamics; in LLM evaluation, a failure of controlled retrieval or controlled reasoning.

An adjacent LLM phenomenon is “lost-in-the-later,” introduced by Tao et al. through CoPE, a framework for quantifying contextual knowledge (CK) versus parametric knowledge (PK). CoPE classifies atomic response sentences as CK when an entailment score exceeds $P(i)$5, defines $P(i)$6 as the proportion of response sentences grounded in the input, sets $P(i)$7, and measures context recall $P(i)$8 over equal-sized input segments. Using the multilingual MultiWikiAtomic dataset in English, Spanish, and Danish, with 100 Wikipedia topics per language and context windows of $P(i)$9 sentences, the study reports a hallmark context-recall pattern $i\approx1$0. Non-reasoning models plateau around $i\approx1$1, while reasoning-oriented models such as GPT-o3 and Qwen 3 235B never exceed approximately $i\approx1$2 CK. The paper further states that reasoning models, as well as non-reasoning models prompted with chain-of-thought, use context even less than non-reasoning models without CoT and fail to mitigate the lost-in-the-later effect. CoT responses are on average 40–60 tokens long, compared with 100–150 tokens for standard responses, and CK Prompt, which combines “Strict” and “Balanced” instructions, is the best overall prompt variant at 50-sentence contexts for Llama 3.2 90B, reaching $i\approx1$3 versus $i\approx1$4 for the original prompt (Tao et al., 7 Jul 2025).

The contrast between lost-in-the-middle and lost-in-the-later is conceptually important. One yields a U-shaped curve with both primacy and recency; the other yields monotone degradation toward later context. This suggests that positional bias in LLMs is heterogeneous rather than unitary, and that the interaction between reasoning, contextual grounding, and retrieval depends strongly on task framing, prompting, and training demand distributions.