Position Bias Correction is Insufficient for One-Pass Attention Sorting

Abstract: Long-context LLMs suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple sort-and-generate cycles increase deployment cost. We hypothesize that position bias is the primary bottleneck and propose Debiased One-Pass Attention Sorting, which estimates a per-prompt position-bias curve from the low-attention majority of documents and uses it to correct raw attention scores (via subtraction or division) to enable single-pass sorting. Our experiments on two models refute this hypothesis in the tested setting: on LLaMA-2-7B-32K-Instruct, debiasing produces identical results to uncalibrated single-pass sorting (94.83\% containment accuracy), while on YaRN-Llama-2-7b-64k, debiasing improves accuracy by 8.67 percentage points but remains 14.84pp behind iterative sorting, closing only 37\% of the gap. These results suggest that position-bias correction is insufficient to match iterative sorting, and that repeated reordering provides additional benefits beyond bias correction.

• The paper shows that debiasing alone cannot recover the accuracy of multi-pass attention sorting in long-context scenarios.
• Debiased one-pass methods improved accuracy on models with severe recency bias, yet remained inferior to iterative sorting by closing only 37% of the performance gap.
• Iterative attention sorting demonstrates that nonlinear attention dynamics and context refinement are crucial for document selection beyond simple position bias correction.

Position Bias Correction Versus Iterative Attention Sorting in Long-Context LLMs

Introduction

This paper, "Position Bias Correction is Insufficient for One-Pass Attention Sorting" (2606.27793), critically examines the limits of position-bias correction as an alternative to iterative attention-based document reordering in long-context LLMs. The paper focuses on document selection in retrieval-augmented generation (RAG) and long-context question answering, where "lost-in-the-middle" position bias causes models to underutilize relevant content that is not near context boundaries. Attention Sorting addresses this by iteratively reordering documents based on their attention scores, which, while effective, introduces significant inference latency due to multiple sort-and-generate cycles. The central hypothesis tested is whether explicit, per-prompt position-bias correction of attention scores enables a one-pass reordering that matches the multi-pass iterative baseline.

Methodology: Debiased One-Pass Attention Sorting

The proposed approach, Debiased One-Pass Attention Sorting, estimates position-dependent biases in attention allocation from within each prompt. This is predicated on the observation that most documents in typical RAG or QA settings are non-relevant distractors whose attention scores encode only positional, not semantic, information. The method operates in several steps:

1. Raw Attention Extraction: For each document, aggregate attention weights directed to its span during the generation of the first output token.
2. Position-Bias Curve Estimation: Exclude the highest-attention documents (potential answers), then bin the remainder by position and compute a bias curve via statistical aggregation. This yields an estimate of how position alone predicts attention.
3. Debiasing: For each document, adjust its raw attention by subtracting or dividing by the estimated position bias.
4. Reordering and Decoding: Sort documents based on debiased scores, rearrange the input, and generate the output.

In contrast, standard iterative Attention Sorting applies attention-based sorting and generation $k$ times, leveraging shifting attention patterns after each reordering.

Figure 1: Overview of iterative Attention Sorting (multi-pass) versus Debiased One-Pass Attention Sorting, including position-bias curve estimation for correction.

Experimental Results

Experiments were performed on the SynthWiki@28K benchmark using two models: LLaMA-2-7B-32K-Instruct (moderate bias) and YaRN-Llama-2-7b-64k (severe recency bias). Each context comprised approximately 166 documents, with only one containing the answer. Four approaches were compared: no sorting, single-pass uncalibrated attention sorting ($k=1$), single-pass debiased sorting, and five-pass iterative attention sorting ($k=5$).

Key results:

• LLaMA-2-7B-32K-Instruct: Debiasing did not improve accuracy over the raw $k=1$ attention sort (both 94.83%), and both trailed $k=5$ by only 0.67pp, a difference within statistical variation.
• YaRN-Llama-2-7b-64k: Debiasing improved $k=1$ accuracy by 8.67 percentage points (55.83% vs. 47.17%), but the debiased method was still 14.84pp below $k=5$ (70.67%), closing only 37% of the gap.

These results rigorously refute the hypothesis that position-bias correction alone can recover iterative Attention Sorting accuracy. Notably, gains from debiasing are sharply model-dependent, and, for models already robust to contextual position, negligible.

Figure 2: Accuracy and mean gold document position over increasing attention sorting iterations for LLaMA-2-7B-32K-Instruct, showing rapid convergence and a negligible gap after $k=2$.

Analysis of the iterative strategy also reveals that most improvement is achieved in the initial iterations; for LLaMA, final iterations yield diminishing returns, suggesting that attention-based reordering rapidly makes the relevant document salient, with further passes offering minor refinement.

Implications and Theoretical Insights

This work clarifies limits of position-bias correction in attention-based document selection. Empirically, the findings support several important claims:

• Iterative Sorting's Benefits Are Not Explained Solely by Bias Correction: Even after precise position-bias calibration, iterative cycles contribute unique gains, especially for highly biased models. This suggests the existence of nonlinear effects or context-dependent patterns in model attention that are not capturable by prompt-level debiasing alone.
• Model-Specificity: The necessity and efficacy of debiasing are strongly dependent on the underlying LLM's degree of intrinsic position bias. Instruct-tuned models (such as LLaMA-2-7B-32K-Instruct) may internally mitigate much of the bias, making external correction redundant.
• Tradeoff between Latency and Quality: While single-pass debiasing is substantially faster, particularly for large $k$, it cannot guarantee the accuracy of multi-pass sorting for models with severe recency or boundary bias.

Theoretically, these results indicate that simple additive or divisive corrections for position bias, even when fit per-prompt, are insufficient to capture all determinants of the attention distribution relevant to downstream task performance. This points to the involvement of more complex dynamic effects—possibly interaction between attention and content relevance, successive refinement in context representation, or other iterative self-consistency mechanisms.

Practical Considerations and Future Directions

For practitioners, these findings suggest that when deploying attention-based document reordering for long-context RAG applications, model and task characteristics must guide method selection:

• If the target model is relatively position-invariant beyond training length, single-pass strategies suffice.
• For models trained with aggressive position-interpolation or large context windows (e.g., YaRN-extension), iterative strategies remain critical for optimal quality.
• Debiasing may have utility as an acceleration technique on highly position-biased models but should not be viewed as a general replacement for iterative re-ranking.

Future research may focus on characterizing the additional mechanisms introduced by iterative sorting, such as attention context refinement or implicit ensemble effects across passes. Understanding how multi-pass attention operations interact with content relevance—beyond static position modeling—may lead to new techniques for more efficient and robust long-context reasoning. Cross-model and cross-benchmark evaluations, as well as analysis of non-extractive QA or generative RAG scenarios, remain as important extensions of this work.

Conclusion

The empirical evidence presented demonstrates that explicit position-bias correction of document attention scores does not eliminate the advantage of multi-pass iterative Attention Sorting, particularly for models with strong inherent position bias. The practical takeaway is that single-pass debiasing should be reserved for cases where model bias warrants it and latency constraints preclude iterative sorting, but cannot be expected to match the performance of iterative approaches in all settings. These findings inform both the deployment and further methodological development of long-context LLMs in retrieval-intensive applications.