Lost-in-the-Later in LLM Research

• Lost-in-the-Later is a phenomenon in LLM research characterized by the underuse of relevant later-stage information from context segments, retrieved passages, or model layers.
• It spans diverse applications—open-ended QA, multimodal retrieval, and multi-turn dialogue—exhibiting issues like primacy bias, self-contamination, and delayed stabilization.
• Empirical studies reveal that while simple single-fact retrieval can be robust, complex tasks involving multi-hop reasoning and multimodal inputs remain vulnerable to later-stage degradation.

Lost-in-the-Later is a label used in recent LLM research for a family of later-stage degradation phenomena in which relevant signal is underused when it appears “later” relative to a model’s processing trajectory. In the current literature, that “later” can mean later context segments in open-ended QA and summarization, later retrieved passages in multimodal retrieval-augmented question answering, later conversational turns after earlier assistant replies have entered the history, or later transformer layers in the forward pass. The literature is not uniform: some studies report strong primacy and clear later-stage failure, while a long-context needle-in-a-haystack study on Gemini 2.5 Flash reports no measurable end-of-context degradation for simple single-fact retrieval even near a 1M-token limit (Tao et al., 7 Jul 2025, Liu et al., 15 Jun 2026, Zheng et al., 26 May 2026, Chen et al., 23 May 2026, Zhou, 8 May 2026, McKinnon, 8 Nov 2025).

1. Conceptual scope and relation to “Lost in the Middle”

The phrase emerged against the background of “Lost in the Middle,” the now-standard observation that long-context LLMs often show a U-shaped accuracy curve: information near the beginning and end of a sequence is used better than information in the middle. Later work broadens that positional perspective. In moderate-length text contexts, the relevant pattern is often not a U-shape but a monotonic decay from early to late segments. In multimodal KB-VQA, the curve can flip from U-shaped to strongly primacy-dominated. In multi-hop QA, degradation depends not only on distance from the edges of the context but also on the distance between required pieces of evidence (Tao et al., 7 Jul 2025, Liu et al., 15 Jun 2026, Baker et al., 2024).

Level	Operationalization	Representative result
Context segments	Recall across context quartiles or document positions	Early segments are overused; later segments are underused (Tao et al., 7 Jul 2025)
Multimodal prompt slots	Gold passage moved from first to middle to last	Gold-at-first beats gold-at-last by 16 to 26 points (Liu et al., 15 Jun 2026)
Conversation turns	FULL vs SHARDED information-equivalent tasks	Multi-turn performance drops, then can be recovered by self-distillation (Zheng et al., 26 May 2026, Chen et al., 23 May 2026)
Forward-pass depth	Distance between layerwise and final next-token distributions	Instruction-tuned checkpoints settle later in the stack (Zhou, 8 May 2026)

This suggests a shared abstraction: later-stage underuse of relevant signal. The signal may be contextual evidence, retrieved passages, user-provided shards, or late-layer computation, but the empirical symptom is similar—performance depends on when or where the relevant information becomes available.

2. Text-context grounding: later segments, contextual knowledge, and response drift

A direct formalization of Lost-in-the-Later appears in the CoPE framework, which distinguishes Contextual Knowledge (CK)—content in a response entailed by the input context—from Parametric Knowledge (PK)—content not entailed by the context. CoPE atomizes both context and response into atomic sentences and computes sentence-level grounding. Its CK score is

$$\text{CK} = \frac{\sum_{i=1}^{n} \mathbb{I}(S_i \in C)}{n} \times 100,$$

and its position-sensitive context recall over context segment $Q_q$ is

$$\text{CR}_q = \frac{\sum_{i=1}^{n} \mathbb{I}\big(S_i \text{ is entailed by } Q_q\big)}{|Q_q|}.$$

Using MultiWikiAtomic in English, Spanish, and Danish, CoPE reports a strong positional bias: $\text{CR}_1$ is highest, $\text{CR}_4$ is lowest, and the gap widens with longer contexts up to 50 atomic sentences. Randomizing sentence order changes recall only by about 5%, so the effect is not explained merely by article structure. The same study reports a CK ceiling around 70–75 for the best non-reasoning models and around 55 for reasoning models, and shows that PK rises toward the final quartile of the response, linking later-context underuse to later-response hallucination risk (Tao et al., 7 Jul 2025).

The same work is notable for showing that more reasoning does not automatically improve contextual grounding. Reasoning models, as well as non-reasoning models prompted with CoT, use context even less than non-reasoning models without CoT. For Llama 3.2 90B at 50-sentence context, the reported CK scores are 70.45 for the Original prompt, 75.86 for Strict, 61.54 for Balanced, 78.72 for CK Prompt, 73.10 for CoT, and 75.10 for CoT + CK Prompt. The paper attributes part of the CoT degradation to shorter answers and reduced effective bandwidth for context-grounded content (Tao et al., 7 Jul 2025).

A further complication is contradiction. In true-first versus false-first settings, CK is higher in True-First than False-First across languages, indicating that early content anchors the model more strongly than later corrections. In practical terms, later clarifications can be systematically under-integrated even when the total context is short and far below any advertised context limit (Tao et al., 7 Jul 2025).

3. Long-context retrieval: absence, persistence, and task dependence

Long-context retrieval studies show that Lost-in-the-Later is not universal. “Retrieval Quality at Context Limit” probes the question directly for Gemini 2.5 Flash using the full transcript of Friends with 20 manually authored non-canonical snippets injected evenly from beginning to end and 26 distinct QA pairs asked in one call. Across context sizes from 0.13 to 0.92 of the model’s context limit, the model answers all 26 questions correctly, yielding accuracy $= 1.0$ at every tested size. At 1.05 of the context limit, the API rejects the request with “The input token count (1105498) exceeds the maximum number of tokens allowed (1048576).” In this single-fact needle-in-a-haystack regime, the study reports no evidence of either Lost in the Middle or Lost-in-the-Later (McKinnon, 8 Nov 2025).

That negative result is tightly scoped. The same paper limits its claim to simple factoid QA with a single injected fact, no paraphrased or adversarial questions, no multi-needle settings, no multi-hop reasoning, no multimodal input, and no contradictory facts. A plausible implication is that modern long-context architectures can now solve single-needle retrieval reliably while still failing on structurally harder forms of retrieval.

Other benchmarks support that narrower interpretation. GM-Extract studies retrieval of automotive control variables with two distinct metrics: Document Metric for spatial retrieval capability and Variable Extraction Metric for semantic retrieval capability. At roughly 4k tokens, 7–8B models already show strong position dependence. At roughly 12k tokens, long-context models degrade again, and the degradation is sharper for Document Metric than for Variable Extraction Metric, indicating that spatial localization collapses earlier than semantic extraction. The paper also reports that simply changing representation from key–value format to paragraph-style QA shifts Document Metric by 15–20 percentage points on average and changes perplexity (Gupte et al., 17 Nov 2025).

Multi-hop QA extends the same point. “Lost in the Middle, and In-Between” shows that performance degrades not only with respect to absolute position but also with respect to spacing between required evidence documents. On MuSiQue 3-hop with GPT-3.5-turbo, full documents, adjacent evidence at $(1,2,3)$ yields 58.95%, whereas spread evidence at $(1,5,10)$ yields 49.47%. The paper characterizes this as an “in-between” effect: even if evidence is not buried in the geometric middle, large distractor gaps between hops reduce successful integration (Baker et al., 2024).

Taken together, these results separate at least three regimes. Simple single-needle retrieval can be position-invariant near the context limit (McKinnon, 8 Nov 2025). Dense multi-document retrieval can remain spatially fragile even at moderate lengths (Gupte et al., 17 Nov 2025). Multi-hop reasoning introduces an additional dependence on evidence spacing, so later-position failure can be compounded by inter-evidence distance rather than by position alone (Baker et al., 2024).

4. Multimodal retrieval-augmented QA and “Lost at the End”

The strongest explicit later-position result appears in multimodal KB-VQA. “Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering” introduces a gold-position protocol in which the set of retrieved passages is held fixed and only the slot position of the gold passage changes within the reader prompt. The setting uses three frozen open-source 7B/8B VLM readers, two KB-VQA benchmarks, and $k \in \{1,3,5,10,20\}$. Because image, question, and passage set are identical across variants, any accuracy change is reader-side position dependence (Liu et al., 15 Jun 2026).

The central empirical result is that the curve flips from U-shaped to primacy-dominated. At $k=10$, gold-at-first beats gold-at-last by 16.5 points for Qwen2.5-VL-7B on InfoSeek, 26.4 points for Qwen2.5-VL-7B on E-VQA, 17.9 points for InternVL3-8B on InfoSeek, 23.2 points for InternVL3-8B on E-VQA, 16.1 points for Qwen3-VL-8B on InfoSeek, and 18.2 points for Qwen3-VL-8B on E-VQA. On five of six reader-by-benchmark cells, there is no recency rebound at the end; the summary relation is “first $Q_q$0 middle $Q_q$1 last” rather than a classical U-shape (Liu et al., 15 Jun 2026).

The ablations localize the effect to prompt slot 0 of the instruction-tuned reader. A text-only control shows that the multimodal KB-VQA setting amplifies an already-present text-mode primacy by roughly 2.2–4.5 times. Moving the image from the start to the end of the prompt does not reduce the first-versus-last gap, rejecting image-token proximity as the main driver. A distractor-shuffle experiment decouples retrieval rank from slot index and finds that the reader follows slot 0 rather than retrieval similarity: on Qwen2.5-VL-7B × InfoSeek, averaged over two shuffle cells, slot-0 selection among distractor-lift errors is 0.425, or 3.83× the $Q_q$2 baseline, while rank-1 distractor selection is 0.112, essentially chance (Liu et al., 15 Jun 2026).

This result has direct metric implications. The paper argues that recall@k is the wrong metric for deployed KB-VQA because “gold somewhere in top-$Q_q$3” is not enough when later slots are systematically underused. Reader-aware evaluation must track where the gold passage sits in the prompt, not merely whether retrieval surfaced it (Liu et al., 15 Jun 2026).

5. Lost-in-conversation: later turns, self-contamination, and recovery

A separate meaning of Lost-in-the-Later concerns multi-turn dialogue. “MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation” defines the LiC gap as the performance drop between a FULL single-turn view and an information-equivalent SHARDED multi-turn view. The paper traces part of this gap to self-contamination: intermediate assistant replies enter later context and carry early deviations forward. In Qwen2.5-7B-Instruct, MAIGO improves SHARDED accuracy from 52.8 to 66.1 and the SHARDED/FULL ratio from 66.5% to 84.1%, while keeping FULL accuracy within 2.3 points. The method uses history-cleaned references for middle turns, paired full-view references for the answer turn, and reliability weighting for noisy middle-turn samples (Zheng et al., 26 May 2026).

FiC addresses the same phenomenon from a different angle. “Found in Conversation” models a conversation as turns $Q_q$4 and defines information-equivalent single-turn and multi-turn views satisfying $Q_q$5. It then trains with View-Asymmetric Self-Distillation, where the same backbone serves as a strong single-turn teacher on the concat view and as a weak multi-turn student on the sharded view. Across Llama, Qwen, Phi, and OLMo backbones from 3B to 14B, FiC recovers at least 92% of single-turn performance and reaches 100% on two Llama backbones. On the math benchmark, for example, Llama-3.1-8B-Instruct rises from 44.7 in SHARDED to 71.7 after FiC, matching its 71.5 concat score; Qwen2.5-7B-Instruct rises from 58.9 to 78.7 against a concat score of 85.0 (Chen et al., 23 May 2026).

These conversation papers are explicit that the failure is behavioral rather than purely capability-bounded. The same model can solve the task when all information is presented at once, yet fails when it must defer, preserve uncertainty, and revise beliefs across turns. A plausible implication is that later-turn degradation is not simply a context-length problem; it is a state-management and policy-learning problem in which the model conditions on its own earlier outputs as if they were trusted evidence (Zheng et al., 26 May 2026, Chen et al., 23 May 2026).

6. Later layers, mechanistic interpretation, and evaluation implications

A third meaning of Lost-in-the-Later is internal rather than contextual. “The Convergence Gap: Instruction-Tuned LLMs Stabilize Later in the Forward Pass” defines the layerwise convergence gap

$Q_q$6

where $Q_q$7 is the next-token distribution decoded from layer $Q_q$8 and $Q_q$9 is the model’s own final distribution. Across six paired pretrained and instruction-tuned checkpoints in native prompting regimes, instruction-tuned checkpoints remain farther from their final predictions later into the stack. Under endpoint-matched controls, the pooled IT − PT late KL is $$\text{CR}_q = \frac{\sum_{i=1}^{n} \mathbb{I}\big(S_i \text{ is entailed by } Q_q\big)}{|Q_q|}.$$0 nats for raw readouts and $$\text{CR}_q = \frac{\sum_{i=1}^{n} \mathbb{I}\big(S_i \text{ is entailed by } Q_q\big)}{|Q_q|}.$$1 nats for tuned-lens readouts. Endpoint-free checks show $$\text{CR}_q = \frac{\sum_{i=1}^{n} \mathbb{I}\big(S_i \text{ is entailed by } Q_q\big)}{|Q_q|}.$$2 adjacent JS and $$\text{CR}_q = \frac{\sum_{i=1}^{n} \mathbb{I}\big(S_i \text{ is entailed by } Q_q\big)}{|Q_q|}.$$3 future top-1 flips for IT relative to PT (Zhou, 8 May 2026).

The causal interventions in that paper isolate late MLP windows as the strongest tested handle on the delay. Across dense families, late IT grafts into PT hosts increase late KL by +0.34 nats, while PT-late swaps into IT hosts reduce it by -0.51 nats; matched random late perturbations give only +0.003 versus +0.327 for the true late graft. In a Gemma 3 4B case study, a late PT swap substantially degrades assistant-register behavior and preference while leaving content-like metrics relatively intact, supporting the interpretation that instruction-following computation is concentrated in later layers (Zhou, 8 May 2026).

Several broader evaluation lessons follow. First, Lost-in-the-Later is not a single scalar defect. It can denote contextual grounding failure, slot-position bias, conversational self-contamination, or delayed internal stabilization. Second, “more reasoning” is not a uniform fix: CoT can reduce contextual grounding in CoPE (Tao et al., 7 Jul 2025), and retrieval-side fixes such as MMR, oracle reranking, and rank-based reordering do not remove multimodal primacy gaps on frozen readers (Liu et al., 15 Jun 2026). Third, apparent success on one regime does not transfer automatically to another: perfect simple-needle retrieval near a context limit does not imply robust multi-needle, multimodal, or multi-turn behavior (McKinnon, 8 Nov 2025).

The most defensible encyclopedic characterization is therefore disjunctive. Lost-in-the-Later names a cluster of later-stage failure modes rather than one unitary mechanism. In text grounding, it denotes underuse of later context segments and drift toward PK in later response segments (Tao et al., 7 Jul 2025). In multimodal RAG, it denotes a primacy-dominated reader that systematically underuses later passages, often without recency recovery (Liu et al., 15 Jun 2026). In conversation, it denotes degradation at later turns because self-generated history corrupts later state (Zheng et al., 26 May 2026, Chen et al., 23 May 2026). In mechanistic analyses of instruction tuning, it denotes the relocation of predictive commitment to later layers (Zhou, 8 May 2026). The main exception documented so far is simple single-fact retrieval in Gemini 2.5 Flash, where no end-of-context degradation is observed up to about 92% of the tested context window (McKinnon, 8 Nov 2025).