Attention Calibration for Position-Fair Dense Information Retrieval

Abstract: Dense retrieval models exhibit positional bias: retrieval effectiveness degrades when relevant information appears later in a passage (Zeng et al., 2025). We ask whether this bias can be reduced at inference time, without retraining and without sacrificing overall retrieval effectiveness. To this end, we adapt inference-time attention calibration (Schuhmacher et al., 2026) to downstream retrieval and extend it with a strength coefficient lambda that interpolates between the original and fully calibrated attention distributions. Across three embedding models on SQuAD-PosQ and FineWeb-PosQ, we examine how basket size, calibrated layer set, and strength affect the trade-off between positional fairness and retrieval effectiveness, finding that partial calibration frequently outperforms full calibration. A single configuration (B=128, lambda=0.5, 50% layer depth) improves the harmonic mean of nDCG@10 across positional groups on FineWeb-PosQ for all three models without per-model tuning, and applies to both <s>-pooled and last-token-pooled architectures. This default configuration transfers without modification to PosIR, which spans 10 languages and 31 domains, reducing the Position Sensitivity Index in all 16 length-quartile x model x retrieval-setting combinations, while preserving or improving aggregate nDCG@10. We release our extended codebase at https://github.com/impresso/fair-sentence-transformers

• The paper introduces an inference-time attention calibration method that adjusts attention distributions in dense retrievers to counteract positional bias.
• It employs a basket-based redistribution strategy with a tunable strength parameter to balance fairness and retrieval accuracy.
• Empirical results across multilingual benchmarks show that partial calibration consistently reduces PSI while maintaining overall nDCG@10 performance.

Attention Calibration for Position-Fair Dense Information Retrieval

Introduction

The study systematically analyzes and mitigates positional bias in dense retrieval models, a phenomenon where retrieval effectiveness diminishes as relevant content is located further from the passage's start. Previous work identified front-loaded attention allocations as a key contributor to this bias. This paper advances the field by introducing an inference-time attention calibration method with a tunable strength coefficient $\lambda$, which interpolates between the original and an equalized attention scheme, facilitating fine-grained control over the trade-off between positional fairness and retrieval fidelity. The empirical evaluation comprises English, multilingual, and cross-lingual retrieval settings across a diverse set of models and benchmarks.

Methodology

The proposed approach extends attention calibration as introduced by Schuhmacher et al. (Schuhmacher et al., 23 Jan 2026). The calibration mechanism partitions the key sequence into contiguous baskets of size $\mathfrak{B}$, enforcing equal total attention mass per basket for the pooling token's query row within a set of selected transformer layers. The attention for each content token $k$ in basket $b(k)$ is recomputed as:

$$a^{\mathrm{cal}}_k = \frac{a_k}{\sum_{j \in b(k)} a_j} \cdot \frac{1}{\text{n}_{\mathrm{baskets}}}$$

A strength parameter $\lambda \in [0,1]$ permits linear interpolation between the unmodified and fully calibrated attention distributions:

$$\hat{a}_k = \lambda a^{\mathrm{cal}}_k + (1 - \lambda) a_k$$

This constraint is applied at inference, requiring no model retraining and no modification on query encodings.

Figure 1: Attention distributions for the pooling token before and after basket-level calibration; $\lambda$ modulates partial redistribution.

The calibration is evaluated over several parameters:

• Basket size ($\mathfrak{B}$) dictates the granularity of redistribution.
• Layer set ($\mathfrak{L}^C$) controls at which transformer layers calibration is applied (last-layer or 50% deepest layers).
• Calibration strength ($\mathfrak{B}$0) governs the degree of attention equalization.

Experimental Framework

The experiments consider three embedding models: gte-multilingual-base, bge-m3-dense (both <s>-pooled), and Qwen3-Embedding-0.6B (last-token-pooled). Evaluation is conducted on SQuAD-PosQ and FineWeb-PosQ (English, position-aware benchmarks) and the Position Aware IR Benchmark (PosIR), covering 10 languages and 31 domains with length-controlled document bucketing.

Key evaluation metrics include:

• nDCG@10: Aggregate and positional-group-specific retrieval quality.
• Harmonic mean of nDCG@10 per group: Reflects retrieval fairness across positions.
• Position Sensitivity Index (PSI): Quantifies relative disparity between the strongest and weakest positional groups.

Results

Fine-Grained Calibration Effects

Calibration achieves minimal gains on short contexts (SQuAD-PosQ), as short passages usually do not allow for meaningful multi-basket partitioning. On longer contexts (FineWeb-PosQ), calibration consistently affords harmonic mean improvements for all three models. The gain is characterized by slightly reduced nDCG@10 for beginning segments and improved scores for later positions, evidencing a trade-off between positional flattening and potential loss of early-span precision.

Figure 2: Partitioning of token sequences into calibration baskets versus evaluation segments; impact varies by passage length and $\mathfrak{B}$1.

Smaller basket sizes favor higher harmonic means, maximizing the benefit of fine-grained redistribution over extended passages. The interplay of $\mathfrak{B}$2 and calibrated layer depth reveals that moderate redistribution (e.g., $\mathfrak{B}$3, 50% of layers) frequently outperforms aggressive, full calibration. Excessive flattening, especially over many layers, can harm begin-group performance more than it benefits later groups.

Figure 3: Incremental nDCG@10 changes across calibration strengths for FineWeb-PosQ; peak fairness-retrieval trade-off arises at intermediate $\mathfrak{B}$4.

Model-Agnostic Default and Large-Scale Validation

A single configuration—$\mathfrak{B}$5, $\mathfrak{B}$6, 50% layer calibration—was found to be robust, consistently improving the harmonic mean of nDCG@10 and reducing PSI across models and pooling mechanisms. Critically, unlike model-specific parameter tuning, this setting generalizes out-of-the-box to PosIR’s multilingual and cross-lingual retrieval regimes, where it:

• Reduces PSI in all document length and retrieval setting combinations, with accentuated benefits for longest contexts.
• Preserves or marginally improves aggregate nDCG@10, decisively outperforming a simple position-agnostic baseline (Segment Embed Average), which reduces PSI only via severe degradation of retrieval effectiveness.

Implications and Future Directions

The results establish that inference-time calibration can mitigate positional bias with negligible performance cost and without retraining or fine-tuning. The findings clarify that the main driver of dense retriever position bias lies in the pooling token’s native preference and is largely independent of the pooling architecture.

A salient, empirically supported claim of this work is that partial—rather than full—attention calibration yields optimal trade-offs between fairness and retrieval effectiveness, contradicting earlier representation-focused perspectives that full flattening is always preferable.

In terms of practical systems design, these findings enable the deployment of more position-fair dense retrievers in both English and multilingual IR settings, including for resource-heterogeneous environments where retraining large models is infeasible.

Open questions remain regarding the mechanism by which partial calibration achieves this optimum. Directions for future work include:

• Analytical and causal investigations of the interaction between basket-level calibration and learned attention distributions.
• Development and evaluation of additional calibration variants targeting more complex forms of positional and context-specific bias.
• Application to hybrid and retrieval-augmented generation scenarios.

Conclusion

This study rigorously demonstrates that inference-time attention calibration, with strength control, constitutes a robust mitigation for position bias in dense retrieval. The partial calibration setting achieves consistent PSI reductions and net retrieval gains across model architectures, languages, and domains, surpassing naive position-agnostic baselines. This establishes partial attention redistribution as a practical, plug-and-play remedy for position bias in embedding-based IR systems, supporting broader deployment of position-fair retrieval pipelines in real-world applications.