ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting

Abstract: The strong capabilities of recent LLMs have made them highly effective for zero-shot re-ranking task. Attention-based re-ranking methods, which derive relevance scores directly from attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First, attention signals are highly concentrated a small subset of tokens within a few documents, making others indistinguishable. Second, attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose \textbf{ReAttn}, a post-hoc re-weighting strategy for attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight attention on query-overlapping tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs entropy-based regularization to mitigate over-concentrated attention, encouraging a more balanced distribution across informative tokens. Both adjustments operate directly on existing attention weights without additional training or supervision. Extensive experiments demonstrate the effectiveness of our method.

• The paper introduces ReAttn, a training-free post-hoc method that combines cross-document IDF weighting with entropy-based regularization to correct attention concentration and lexical bias.
• Experiments across 11 BEIR datasets show consistent gains of about 0.7–1.1 nDCG@10 points, with QRhead+ReAttn reaching 50.9 using Llama-3.1-8B and outperforming several supervised baselines.
• The method adds negligible computational overhead and improves long-context retrieval, but its effectiveness varies by dataset and remains untested for multilingual and IR-specialized LLMs.

ReAttn is a training-free, post-hoc re-weighting strategy for attention-based re-ranking with LLMs. The paper identifies two failure modes of methods that use raw attention as a relevance signal — signal concentration and lexical bias — and addresses them by combining a cross-document IDF re-weighting of token-level scores with an entropy-based regularization at the document level (2602.19969).

Motivation and problem setting

The paper builds on attention-based list-wise re-ranking, notably in-context re-ranking (ICR) (2602.19969), which concatenates all candidate documents followed by the query into an LLM input and aggregates the attention that document tokens receive from query tokens across all layers and heads into token-level relevance scores; QRhead (2602.19969) refines this by restricting aggregation to query-focused retrieval heads. These non-generative approaches avoid the decoding cost, prompt sensitivity, and instruction-following unreliability of list-wise generation methods such as RankGPT.

Despite these advantages, two issues persist. Signal concentration: total attention mass is absorbed by a few tokens in a few documents, leaving most candidates nearly indistinguishable. Lexical bias: attention disproportionately highlights tokens lexically similar to the query, so irrelevant documents sharing surface expressions receive inflated scores. The authors additionally adopt the calibration scheme of ICR, subtracting scores obtained under a content-free "N/A" query to remove biases toward punctuation and other meaningless tokens before applying their adjustments.

The ReAttn method

Cross-document IDF weighting. For each query-overlapping token, ReAttn computes its document frequency across the retrieved candidate set and rescales its calibrated attention score by a normalized IDF weight $$w(t_{i,j}) = \log\frac{N+1}{\mathrm{df}(t_{i,j})+1}/\log(N+1)$$. Tokens appearing in many candidates carry little discriminative evidence, so down-weighting them suppresses spurious lexical matches while preserving distinctive terms. Non-query tokens are left unchanged, and document base scores $B_i$ are summed over the filtered token set.

Entropy-based regularization. Token scores within each document are normalized into a probability distribution whose Shannon entropy $E_i \in [0,1]$ measures internal attention dispersion. A dispersion weight $W_i = 1 + (E_i - \bar{E}_B)$, where $\bar{E}_B$ is the base-score-weighted mean entropy, mildly boosts high-entropy documents (broad coverage over informative tokens) and penalizes narrowly focused ones. Final scores are normalized to sum to one. Both components operate directly on existing attention weights, require no training or supervision, and add negligible computational overhead — an important practical property given that the method targets efficiency-oriented alternatives to generation-based re-ranking.

Main results

Experiments cover eleven BEIR datasets (BM25 top-200 candidates, nDCG@10, 512 sub-sampled queries per domain) with Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B-Instruct, plus long-context benchmarks LongMemEval and CLIPPER.

Method	Base LLM	Avg nDCG@10
ICR	Llama-3B	46.5
ICR + ReAttn	Llama-3B	47.3
QRhead	Llama-3B	48.6
QRhead + ReAttn	Llama-3B	49.3
ICR / ICR + ReAttn	Llama-8B	48.5 / 49.3
QRhead / QRhead + ReAttn	Llama-8B	50.0 / 50.9
QRhead / QRhead + ReAttn	Qwen-7B	44.2 / 45.3

Gains are consistent but modest: roughly +0.7 to +1.0 points for ICR and +0.7 to +1.1 for QRhead on average, with larger per-dataset improvements on entity-centric or lexically dense collections such as DBPedia (+2.1 for ICR+ReAttn on Llama-3B), FiQA (+1.1 to +1.3), and FEVER. Relative improvements are slightly larger for the 3B model, which the authors attribute to sharper attention distributions making smaller models more susceptible to attention collapse. A notable claim is that the parameter-free QRhead+ReAttn configuration (50.9 average nDCG@10 with Llama-8B) surpasses supervised cross-encoders such as MSMARCO-MiniLM (48.0) and GTR-T5-base (45.2), suggesting post-hoc attention refinement can substitute for task-specific training in zero-shot settings.

On LongMemEval and CLIPPER, ReAttn improves recall@k for both ICR and QRhead across both model scales and yields small gains in end-to-end generation performance (e.g., QRhead+ReAttn reaches 60.5 top-10 on LongMemEval with Llama-8B versus full-context 46.5). The authors argue long contexts magnify attention saturation and token redundancy, conditions under which the re-weighting is most beneficial.

Ablations and qualitative analysis

Ablations disentangle the two components and show dataset-dependent behavior. On lexically dominated benchmarks (NQ, NFCorpus, FiQA, DBPedia), IDF alone recovers most of ReAttn's gains; on corpora with longer, topically heterogeneous documents (Robust04, News, Climate), entropy regularization contributes more. One contradictory case stands out: on SciDocs, IDF in isolation slightly degrades performance, indicating distinctiveness-based down-weighting can penalize frequent yet domain-informative scientific terminology; the combined method restores and exceeds baseline performance, which the authors interpret as the two components addressing non-overlapping failure modes. Qualitative attention-map analysis on FEVER confirms that removing IDF inflates attention on verbatim query overlaps (e.g., "Kieślowski", "Jiang Wen") in distractor documents, promoting them to higher ranks, whereas full ReAttn shifts attention toward contextually diagnostic spans — though notably, in one DBPedia example, full ReAttn ranks a plausibly relevant passage lower than the no-IDF variant, illustrating that IDF suppression can occasionally demote genuinely relevant documents containing ubiquitous query entities.

Limitations

The paper concedes three limitations. Evaluation uses only general-purpose instruction-tuned models; IR-specialized LLMs may exhibit different attention dynamics that could interact unpredictably with the re-weighting. Head selection in QRhead remains heuristic, and the method does not analyze head-level contributions to relevance attribution. Finally, all experiments are English-only; cross-lingual robustness of attention-based regularization is untested.

Conclusion

ReAttn offers a simple, training-free correction to two documented weaknesses of attention-based re-ranking, delivering consistent if incremental improvements across model families, BEIR domains, and long-context reasoning benchmarks at essentially no additional cost. The main open questions it leaves concern generalization to IR-tuned and multilingual LLMs, and principled head-level attribution, since the component ablations show that neither adjustment is universally beneficial in isolation.