ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Abstract: Understanding and reasoning over long contexts has become a key requirement for deploying LLMs in realistic applications. Although recent LLMs support increasingly long context windows, they often fail to use relevant evidence that is already present in the input, revealing a gap between context access and effective context utilization. In this work, we propose Recursive Evidence Replay as LLM Harness for Long-Context Reasoning (RECONTEXT), a training-free inference method for improving long-context reasoning. RECONTEXT uses model-internal relevance signals to construct a query-conditioned evidence pool and replays it before final generation while preserving the full original context. This recursive selection process separates evidence organization from answer generation without training, external memory, or context pruning. We also provide a theoretical analysis based on associative memory, which characterizes the context as a memory store, the question as a retrieval cue, attention as cue-trace association, and replay as trace reactivation. Experiments on eight long-context datasets with 128K context length show that RECONTEXT consistently improves evidence utilization across Qwen3-4B, Qwen3-8B, and Llama3-8B, achieving the best average rank on all three backbones. Code is available at https://github.com/Yanjun-Zhao/ReContext.

• The paper introduces a training-free, inference-time method that uses recursive evidence replay to harness internal attention signals for long-context reasoning.
• It extracts high-relevance tokens to form an explicit evidence pool and refines context alignment recursively, achieving up to a 24.6% relative improvement in QA accuracy.
• The approach enhances interpretability and transparency while incurring minimal overhead, making it a promising complement for various LLM deployments.

ReContext: Recursive Evidence Replay for Efficient Long-Context Reasoning

Motivation and Problem Formulation

Recent advances in LLMs have enabled the processing of extremely long contexts—up to 128K tokens or more. However, substantially extending context windows does not solve the persistent problem that LLMs often fail to utilize the appropriate evidence when generating answers within these extended contexts. The critical bottleneck is not merely in context access but in context harnessing: the dynamic and effective identification, organization, and use of relevant information conditioned on the current query, amidst large quantities of irrelevant or distracting data.

Empirical analysis establishes that a minuscule subset of context tokens (top 0.1%), typically only 128 out of 128K, accounts for 50%-80% of the question-conditioned attention relevance mass across leading LLMs such as Qwen3 and Llama3. This pronounced sparsity motivates novel mechanisms to turn internal model signals into actionable, explicit evidence for answer generation.

Figure 1: Top 0.1% of context tokens already accounts for about 50%/80% accumulated relevance across three LLMs in 128K-token contexts.

Method: Recursive Evidence Replay Harness

ReContext proposes a training-free, inference-time harness that explicitly organizes and replays evidence for long-context QA by leveraging the model's own internal relevance signals:

1. Evidence Identification: From an input $[C; q]$ (context, question), attention-derived relevance is computed by aggregating attention weights from a set of query-side cue tokens. This produces a relevance score per context token using selected attention heads over model layers.
2. Span Materialization: Sparse, high-relevance tokens are mapped to their containing sentences or context spans, ensuring atomicity and grounding of evidence.
3. Evidence Replay: The extracted evidence sentences form an explicit evidence pool that is reinserted (replayed) into the prompt, near the query, but without removing the original context.
4. Recursion: The process is performed over a small, fixed number of rounds ($R$), where the evidence pool is expanded by recursively conditioning on the previously selected evidence and recomputing relevance.
5. Final Generation: The LLM generates answers from the composite prompt containing the entire original context, the recursive evidence scaffold, and the query.

Figure 2: Overview of ReContext’s evidence extraction, replay, and integration into the original prompt.

Key design differences from existing approaches:

• Non-invasive: No modification of attention mechanisms at decode-time.
• Full-context retention: Evidence is highlighted for emphasis; no context pruning occurs, maintaining fallback access to all original content.
• Training-free: Operates solely at inference and leverages model-internal signals—no additional model parameters or retraining required.

Associative Memory Theoretical Perspective

The mechanism is formalized through the lens of associative memory. The context $C$ acts as a memory store; the query $q$ is a retrieval cue; inter-token attention reflects cue-trace association; replay constitutes the reactivation of traces just prior to prediction. Under mild assumptions, it is shown that recursive evidence replay monotonically increases the cosine similarity between the model’s hidden representation and the answer embedding at each round, directly improving evidence alignment for answer generation.

Experimental Validation

Datasets and Baselines

ReContext is evaluated on eight established long-context benchmarks—NQ, TriviaQA, HotpotQA, PopQA, NarrativeQA, InfBench QA, InfBench MC, and CLIPPER—across Qwen3-4B, Qwen3-8B, and Llama3-8B. Baselines include direct vanilla prompting, attention sharpening (AttnSharp), dynamic attention scaling (DySCO), agentic memory (A-MEM), and dynamic attention-aware compression (DAC).

Main Results

• Aggregate Performance: Across backbones and metrics, ReContext achieves the best average rank on all three LLMs. On Qwen3-4B, it leads on every score; on Qwen3-8B and Llama3-8B, it consistently tops average ranks and substantially outperforms the vanilla baseline in both accuracy and F1 (24.6% relative gain).
• Shorter Contexts & Robustness: Gains are preserved at 64K tokens and when enabling explicit intermediate reasoning (“thinking mode”).
• Interpretability: The explicit evidence pool enables more transparent diagnosis of model reasoning.

Figure 3: Ablation: Left—effect of recursive evidence-selection rounds $R$; Right—effect of evidence token budget $K$.

• Increasing recursion depth significantly improves performance up to a point; optimal values are dataset-specific, not monotonic.
• Larger evidence token budgets help in information-dense tasks but can introduce noise for certain queries.

Figure 4: Qualitative examples of evidence replay highlighting query-relevant spans and correcting vanilla errors.

Cases illustrate that ReContext resolves vanilla generation mistakes by foregrounding selected supporting sentences, and provides evidence attributions missing in baseline compression and retrieval approaches.

Efficiency

Despite additional evidence scoring and replay, ReContext only marginally increases inference time and minimally increases memory overhead relative to vanilla prompting. Its runtime is substantially lower than attention intervention mechanisms that require modifying the model’s forward pass.

Figure 5: Runtime comparison shows ReContext incurs only moderate overhead versus vanilla; DySCO is much slower.

Implications and Future Directions

Practical and Theoretical Contributions:

• Inference-time applicability: No model retraining or adaptation required.
• Transparency: Evidence replay explicitly surfaces the basis for predictions.
• Complementarity: Methodology is orthogonal to prompt compression, memory architectures, or context expansion—facilitating integration with a range of LLM deployments.
• Theoretically grounded: Monotonic improvement guarantees are rare for inference-time interventions in deep models, attesting to the method’s principled operation.

Limitations: Access to internal model attention signals is required, restricting use in closed models or API-only scenarios. Additional prompt construction stages increase end-to-end latency, but this is minor relative to certain dynamic attention methods.

Future Directions:

• API-level integration: Investigating surrogate alternatives to direct attention readout for commercial LLM APIs.
• Extending beyond QA: Applying recursive evidence organization to agentic planning, summarization, and program synthesis.
• Joint evidence selection and answer refinement: Integrating evidence replay with iterative refinement pipelines.

Conclusion

ReContext establishes a novel paradigm for long-context LLM harnessing: training-free, recursive evidence replay that bridges internal relevance signals and explicit answer grounding. The framework achieves strong empirical gains on challenging benchmarks, is theoretically justified via an associative memory view, and maintains operational transparency and efficiency. It demonstrates that optimal context utilization in LLMs is as much about evidence organization as hardware scaling or architectural extension.

---

Reference: "ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning" (2607.02509)