Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG

Abstract: Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being concatenated into a model's input context, making both document ordering and context size critical yet controversial design choices. Prior work reports position-based effects such as lost in the middle and related long-context phenomena. However, empirical findings remain inconsistent and hard to reproduce across models, datasets, and evaluation protocols. In this paper, we present a systematic reproducibility study that revisits these claims and examines how they evolve with contemporary LLMs under a controlled evaluation framework. We first show that topic sampling is a major source of variance: small topic sets can mask or exaggerate ordering effects. Based on repeated subset sampling across multiple topic budgets, we provide a practical calibration procedure that identifies topic counts yielding stable trends at feasible cost. Using these fixed topic sets, we then reproduce and extend results on position sensitivity, re-evaluating lost in the middle and positional biases in modern LLMs. Then, we also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents. In this setting, we re-examine a recent industry study and identify discrepancies to evaluation choices such as limited topic coverage and reliance on LLM-based judges. Finally, we conduct an analysis of how retrieval order and context size affect downstream LLM performance under imperfect retrieval. Our results demonstrate that both factors interact strongly with retrieval quality and model choice, and that conclusions drawn from idealised setups do not always transfer to real-world RAG pipelines. We release all code and configurations to support reproducibility and future work on robust RAG evaluation.

• The paper demonstrates that calibrated topic sampling significantly reduces variance, enabling reliable comparisons of context size and ordering effects in RAG pipelines.
• The paper replicates findings showing that previously reported U-shaped accuracy curves are not robust on modern LLMs, challenging earlier claims.
• The paper reveals that reverse evidence ordering improves multi-hop QA accuracy while practical retrieval quality remains a critical bottleneck.

Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG

Introduction and Motivation

The paper "Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG" (2605.27105) conducts a rigorous reproducibility study of context size and document ordering effects in Retrieval-Augmented Generation (RAG) pipelines. Addressing significant discrepancies and conflicting empirical findings in recent literature, it disentangles how critical design factors—namely, the number of retrieved passages (context size) and their locations within the model's prompt—impact answer accuracy in long-context LLM QA setups.

The authors highlight that arbitrary or small topic sampling introduces high variance, obfuscating actual performance differences and making previous conclusions unreliable. By introducing a dataset-specific calibration procedure for stable subset evaluation, they methodologically standardize comparisons across RAG ordering and size schemes, LLM architectures, and retrieval models, focusing on HotpotQA (multi-hop, complex reasoning) and Natural Questions (single-hop, factoid).

Methodological Framework and Calibration

A central contribution is the systematic calibration of topic budgets to ensure experiment stability. The authors empirically determine that small topic samples (e.g., 500) induce order-dependent performance oscillations and "zero-crossings," where the relative ranking between ordering or sizing flips due to sampling noise.

Figure 1: HotpotQA results for 500 topics show substantial variability in $\Delta$F1 between ordering strategies; variance impedes reliable comparison.

Figure 2: HotpotQA results with 1000 topics demonstrate decreased variance, stabilizing comparative rankings between ordering strategies.

By progressively increasing topic counts (500 → 1000+ for HotpotQA), they show that variance in pairwise accuracy differences ($\Delta$F1) declines, preventing zero-crossings and producing reproducible, representative trends. The calibration procedure ultimately supports all subsequent analyses by anchoring results on topic sizes that yield deterministic performance deltas.

Reproduction of Long-Context Position Effects

The study critically re-examines two prominent prior findings:

• Lost in the Middle: Previous work reported U-shaped accuracy curves in QA with answer-containing evidence centered in a long prompt [liu-etal-2024-lost-in-the-middle]. The authors replicate this on AmbigQA with LLaMA-3.1:8B and Mistral-NeMo:12B, observing that the purported position effects are not robust—accuracy remains flat across placements.

  Figure 3: Reproduction demonstrates that long-context position effects are not clearly present for contemporary LLMs on AmbigQA; the U-shaped curve is not reliably observed.

• Lost but Not Only in the Middle: This follow-up claimed non-monotonic, dataset-dependent position effects [Yu2024InDO]. On HotpotQA and NQ (using stable topic budgets), the authors again report nearly flat accuracy curves across document positions, contradicting strong position-dependency claims.

Figure 4: Reproduction shows near-flat performance across positions in HotpotQA and NQ; order sensitivity is negligible for modern LLMs under the authors' protocol.

These results undermine the generalizability of previously reported position-bias phenomena, emphasizing that findings are highly sensitive to dataset, evaluation protocol, and model specifics.

Context Size and Ordering under Imperfect Retrieval

Moving to end-to-end RAG setups, the study analyzes realistic settings where the retriever does not guarantee access to all gold evidence. The authors replicate industry studies claiming weak sensitivity to context size or passage order [cao-bloomberg-2025], but find that—contrary to prior claims—actual performance shows measurable, context-dependent ordering effects, especially as $k$ increases.

Figure 5: HotpotQA shows clear sensitivity to context order with 500 topics, contradicting prior claims of ordering robustness as context grows.

By leveraging their calibrated topic budgets, they further expose ordering and context-size trends unobstructed by sampling variance.

Controlled RAG Analysis: Retrieval Quality, Ordering, and Model Scale

Using large, stable topic sets ($n = 1000$ for HotpotQA, $n = 2000$ for NQ), the authors systematically dissect the interactions among ordering, context size, retrieval quality, and LLM family/scale.

Ordering and Context Size Effects

On HotpotQA (multi-hop), model accuracy is highly sensitive to ordering strategies; placing high-value passages at the end (reverse) yields superior performance as $k$ increases. NQ (single-hop) shows steady accuracy gains with increasing $k$, and order sensitivity is muted.

Figure 6: HotpotQA with 1000 topics illustrates reverse ordering outperforming standard schemes at large context sizes; NQ results show smaller order effects.

Oracle vs. Realistic Retrieval

Oracle contexts (all gold passages) set an upper bound that is significantly higher than practical BM25 retrieval results, quantifying the potential for improved retrieval and evidence positioning.

Figure 7: Closed-book, BM25 retrieval, and oracle context comparison on HotpotQA; oracle ranking marks the attainable upper bound, demonstrating retrieval bottleneck.

Reverse ordering in oracle+BM25 setups consistently confers advantages, supporting the hypothesis that LLMs preferentially attend to evidence at prompt boundaries in multi-hop QA tasks.

Retrieval Quality and Reranking

Dense reranking (BM25+E5) substantially improves answer accuracy in short contexts, mitigating the necessity for long context sizes and reducing order sensitivity. As $k$ grows, order effects remain relevant if retrieval quality is suboptimal.

Figure 8: HotpotQA with BM25+E5 reranking demonstrates higher performance for small $k$, with diminished ordering gaps relative to standard BM25.

Model Family and Scale

Larger LLMs (e.g., LLaMA-3.1:70B) exhibit greater robustness to ordering in multi-hop QA: $\Delta$F1 between reverse and standard ordering decreases with scale. However, even large models retain some order sensitivity in multi-hop datasets.

Figure 9: HotpotQA (1000 topics) shows decreased order sensitivity ($\Delta$0F1) for larger models; all architectures reach peak accuracy with optimal ordering.

Implications and Future Directions

The study makes several bold, empirically substantiated claims:

• Sampling Variance Dominates Small-Scale Evaluation: Findings on order and context-size effects are heavily confounded by topic sample size. Small-topic experiments cannot reliably inform practical RAG design.
• Position Effects Are Not Universally Reproducible: Previously reported U-shaped position sensitivity (lost-in-the-middle) is not robust for contemporary LLMs on popular QA datasets under the authors' protocol.
• Ordering Remains Crucial in Multi-Hop QA: In realistic QA pipelines, reverse evidence ordering consistently improves accuracy for multi-hop questions; single-hop tasks are less sensitive.
• Retrieval Quality Is the Bottleneck: Oracle contexts substantially outperform practical retrieval schemes; improved ranking and reranking narrow the gap and reduce order sensitivity.
• Larger Models Achieve Greater Robustness: While larger models display improved resilience to ordering and context size, multi-hop reasoning still benefits from strategic evidence placement.

Practically, the research recommends calibrated evaluation protocols and warns against drawing generalizable conclusions from small-topic or order-insensitive experiments. Theoretically, it nuances our understanding of how LLMs attend to long contexts and how prompt engineering can mitigate reasoning failure modes. Future work should advance retrieval models to close the gap to oracle upper bounds and optimize context construction for heterogeneous datasets and task profiles.

Conclusion

This study advances reproducible RAG evaluation with a robust, topic-calibrated analysis of context size and passage ordering effects. It establishes that ordering and context size interact strongly with dataset structure, retrieval quality, and model architecture, and that prior claims often fail to transfer to real-world scenarios. By exposing the empirical foundations of order and size effects, the authors enable practitioners and researchers to make better-informed decisions when designing retrieval pipelines and prompt layouts for LLM-based QA. The released code and evaluation configurations are poised to drive further methodological rigor and innovation in retrieval-augmented language modeling.