PosIR: Position-Aware IR Benchmark

• PosIR is a benchmark that isolates the effects of document length and evidence position, revealing inherent positional bias in dense retrieval models.
• It comprises 310 datasets across 10 languages and 31 domains, enabling detailed analysis of retrieval performance under varying document lengths and positional contexts.
• PosIR supports diagnostic and debiasing research by integrating metrics like nDCG, MRR, and a unique Position Sensitivity Index to quantify retrieval robustness.

PosIR, introduced by Zeng et al. as "PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark," is a position-aware information retrieval benchmark designed to diagnose retrieval models’ sensitivity to the location of relevant information within documents. It comprises 310 retrieval datasets spanning 10 languages and 31 domains, and it ties relevance to precise reference spans so that document length and information position can be analyzed separately rather than conflated under document-level labels (Zeng et al., 13 Jan 2026). In subsequent work, PosIR also serves as an evaluation substrate for mitigation methods targeting positional bias in dense retrieval, including inference-time attention calibration that reduces position sensitivity without retraining (Michail et al., 1 Jun 2026).

1. Conceptual scope and formalization

PosIR is motivated by a specific failure mode in dense retrieval: position bias. In this setting, a retrieval model favors evidence at particular locations in a document, such as the beginning under primacy bias or the end under recency bias, rather than treating all parts of the document equally. Existing position-agnostic benchmarks, including MS MARCO, BEIR, and MMTEB, assign relevance at the document level and therefore conflate two effects: a capacity effect, in which models have limited ability to encode long contexts at all, and positional sensitivity, in which models systematically ignore content in certain segments (Zeng et al., 13 Jan 2026).

The benchmark formalizes this distinction by attaching each relevant query-document pair to a true relevance span. Let a document be $D = (t_1, t_2, \ldots, t_n)$, and let the true relevance span be $S \subseteq \{1,\ldots,n\}$ with start index $s = \min S$ and end index $e = \max S$. The relative position of the span is defined as

$p = \frac{s + e}{2n} \in [0,1].$

Because each $(q,D)$ pair is annotated with $(s,e)$, retrieval performance can be analyzed as a function of $p$ while controlling for $n$ (Zeng et al., 13 Jan 2026).

A common misconception is that long-document failures in dense retrieval are adequately characterized by aggregate effectiveness degradation alone. PosIR is explicitly constructed to refute that simplification: poor performance on long documents may arise either because the model cannot process long contexts or because it is selectively weak on certain evidence locations. PosIR isolates these two factors by design (Zeng et al., 13 Jan 2026).

2. Benchmark composition and coverage

PosIR comprises 310 retrieval datasets organized as 10 languages $\times$ 31 domains. The languages are Arabic, Chinese, German, English, French, Italian, Korean, Portuguese, Russian, and Spanish. The domains are heterogeneous and include news, Wikipedia, patents, forums, and a FineWeb general domain (Zeng et al., 13 Jan 2026, Michail et al., 1 Jun 2026).

Coverage across document lengths is a central design feature. During construction, documents are partitioned into eight bins by length: $S \subseteq \{1,\ldots,n\}$0–$S \subseteq \{1,\ldots,n\}$1, $S \subseteq \{1,\ldots,n\}$2–$S \subseteq \{1,\ldots,n\}$3, $S \subseteq \{1,\ldots,n\}$4, $S \subseteq \{1,\ldots,n\}$5–$S \subseteq \{1,\ldots,n\}$6 tokens. The sampling ratio is $S \subseteq \{1,\ldots,n\}$7, so documents of length at most $S \subseteq \{1,\ldots,n\}$8 tokens are oversampled by $S \subseteq \{1,\ldots,n\}$9. Approximately $s = \min S$0 documents per domain per language are sampled to yield balanced coverage of document lengths (Zeng et al., 13 Jan 2026).

For evaluation, document lengths are further grouped into four quartiles: Q1 up to $s = \min S$1 tokens, Q2 $s = \min S$2–$s = \min S$3 tokens, Q3 $s = \min S$4–$s = \min S$5 tokens, and Q4 $s = \min S$6–$s = \min S$7 tokens. Retrieval is studied in both monolingual and cross-lingual settings. In the PosIR benchmark paper, cross-lingual retrieval uses non-English queries against an English corpus. In later PosIR-based evaluation, the same distinction appears as multilingual $s = \min S$8 and cross-lingual $s = \min S$9 retrieval (Zeng et al., 13 Jan 2026, Michail et al., 1 Jun 2026).

This breadth of language, domain, and length coverage is significant because it prevents positional analysis from being an artifact of a single corpus type or a single language family. A plausible implication is that positional effects measured on PosIR are more likely to reflect systematic properties of retrieval architectures and training regimes than narrow dataset idiosyncrasies.

3. Construction pipeline and annotation protocol

PosIR is built through a multi-stage pipeline intended to produce fine-grained, position-aware qrels. The source corpora are IndustryCorpus2 for English-Chinese and a FineWeb general domain, with tokenization performed by the Qwen3 tokenizer (Zeng et al., 13 Jan 2026).

In Stage I, length-aware corpus sampling creates balanced document pools across the eight length bins. In Stage II, position-aware question generation and span annotation proceed as follows. For each sampled document $e = \max S$0, a random positional constraint is selected from the first, middle, or last third of the document. An LLM, DeepSeek-V3.1, is then prompted to generate questions whose answers lie under that positional constraint, and it is prompted again to locate the reference span in $e = \max S$1. The resulting positive qrels are recorded as

$e = \max S$2

where $e = \max S$3 gives the character-level span (Zeng et al., 13 Jan 2026).

Stage III performs quality control. For reference-span verification, each $e = \max S$4 is scored with rerankers on the original document and on the document with the reference removed. A sample is retained only if the reranker score on the original exceeds the score on the ablated version under two rerankers. It is then further verified by a 5-level LLM assessor, which requires both $e = \max S$5 and $e = \max S$6. False-negative removal then recalls the top-1000 documents with an embedding model, reranks them, and removes any candidate document that exceeds the positive document’s score, together with its queries, to avoid ambiguous positives (Zeng et al., 13 Jan 2026).

The final multilingual translation stage translates each English corpus and query set into eight target languages via Qwen3-30B, reuses the same position-aware qrels, and applies length-based outlier filtering with IQR-based thresholds as well as prefix and refusal removal (Zeng et al., 13 Jan 2026).

The construction protocol is notable because relevance is not merely associated with a document but with a verified, position-constrained reference span. This is the basis for PosIR’s claim to strict disentanglement of document length from information position (Zeng et al., 13 Jan 2026).

4. Metrics and evaluation protocol

PosIR evaluates both retrieval effectiveness and position robustness. For effectiveness, it reports Mean Reciprocal Rank and Normalized Discounted Cumulative Gain. The benchmark defines

$e = \max S$7

and

$e = \max S$8

where $e = \max S$9 for top-$p = \frac{s + e}{2n} \in [0,1].$0 retrieval or graded relevance (Zeng et al., 13 Jan 2026).

Its distinctive diagnostic metric is the Position Sensitivity Index. PosIR discretizes $p = \frac{s + e}{2n} \in [0,1].$1 into 20 equal-width bins, lets $p = \frac{s + e}{2n} \in [0,1].$2 denote the average nDCG@10 in each bin, and defines

$p = \frac{s + e}{2n} \in [0,1].$3

Lower PSI indicates more position-robust retrieval behavior (Zeng et al., 13 Jan 2026).

The experimental setup in the benchmark paper evaluates 10 state-of-the-art embedding retrievers, including BGE-M3, Qwen3-Embedding, inf-retriever, NV-Embed-v2, llama-embed, and KaLM-Embedding. It uses MMTEB Retrieval Task nDCG@10 as a short-context reference point and studies both monolingual and cross-lingual retrieval under the Q1–Q4 document-length buckets. Correlation analysis uses Spearman’s $p = \frac{s + e}{2n} \in [0,1].$4 for model rankings and Pearson’s $p = \frac{s + e}{2n} \in [0,1].$5 for score correlations (Zeng et al., 13 Jan 2026).

A technical subtlety arises in downstream use of PosIR. In the attention-calibration study, PSI is reported within each length quartile using position quartiles rather than the benchmark’s 20-bin discretization, because the goal there is to compare calibrated and uncalibrated models under a fixed length bucket (Michail et al., 1 Jun 2026). This suggests that PosIR supports both fine-grained diagnostic analysis and coarser operational reporting, depending on the intervention under study.

5. Experimental findings and mechanistic analysis

PosIR’s core empirical result is that position bias is pervasive and intensifies with document length. Across the benchmark, all models degrade as document length grows from Q1 to Q4. One example given is Qwen3-Embedding-4B, whose nDCG@10 drops from approximately $p = \frac{s + e}{2n} \in [0,1].$6 in Q1 to approximately $p = \frac{s + e}{2n} \in [0,1].$7 in Q4 (Zeng et al., 13 Jan 2026).

The benchmark also shows that performance on PosIR in long-context settings correlates poorly with MMTEB. The overall Spearman correlation between MMTEB and PosIR nDCG@10 is $p = \frac{s + e}{2n} \in [0,1].$8 with $p = \frac{s + e}{2n} \in [0,1].$9, indicating only moderate alignment. By length bucket, the ranking correlation declines from $(q,D)$0 with $(q,D)$1 and $(q,D)$2 to $(q,D)$3 with $(q,D)$4 and $(q,D)$5 with $(q,D)$6 (Zeng et al., 13 Jan 2026). This directly challenges the assumption that strong short-context benchmark performance predicts long-context retrieval robustness.

PSI trends exhibit the same deterioration with length. In monolingual retrieval averaged across 10 languages, PSI rises from approximately $(q,D)$7 in Q1 to approximately $(q,D)$8 in Q4. In cross-lingual retrieval, PSI rises from approximately $(q,D)$9 in Q1 to approximately $(s,e)$0 in Q4. Most models exhibit primacy bias, with performance highest for early position bins and falling toward later bins. NV-Embed-v2 is an exception, showing recency bias with a J-shaped performance curve peaking in final bins (Zeng et al., 13 Jan 2026).

PosIR complements these benchmark-level observations with gradient-based saliency analysis. For a query $(s,e)$1 and a long document $(s,e)$2 with $(s,e)$3, it computes

$(s,e)$4

and token saliency

$(s,e)$5

After normalizing $(s,e)$6, positions are discretized into 100 relative bins and averaged. Qwen3-Embedding-8B shows an extreme peak at the start of the document and rapid decay, with a small spike at the final bin from last-token pooling. NV-Embed-v2 shows near-zero saliency in the first $(s,e)$7 of the document and steadily rising saliency toward the end (Zeng et al., 13 Jan 2026). The benchmark interprets these patterns as mechanistic explanations for primacy and recency bias, respectively.

6. PosIR in debiasing research and attention calibration

PosIR is not only a diagnostic benchmark; it is also used to test interventions aimed at mitigating positional bias. Michail et al., in "Attention Calibration for Position-Fair Dense Information Retrieval," adapt inference-time attention calibration to downstream retrieval and evaluate it on PosIR without retraining the retrieval models (Michail et al., 1 Jun 2026).

The intervention modifies only the pooling-token query row in a selected subset of transformer layers $(s,e)$8, applies only at inference time to passage or document encodings, and leaves query encodings unchanged. With basket size $(s,e)$9, original post-softmax attention weights $p$0, basket assignment $p$1, and number of baskets $p$2, fully calibrated attention is defined as

$p$3

and the calibrated distribution is interpolated with a strength coefficient $p$4:

$p$5

The default configuration selected from English-only PosQ experiments uses $p$6, $p$7, and calibration in the last $p$8 of transformer layers (Michail et al., 1 Jun 2026).

On PosIR, this default configuration transfers without modification across 10 languages and 31 domains. For the multilingual $p$9 setting, gte-multilingual-base moves from quartile PSI values $n$0, $n$1, $n$2, and $n$3 with aggregate nDCG@10 of $n$4 to $n$5, $n$6, $n$7, and $n$8 with nDCG@10 of $n$9. In the same setting, bge-m3-dense moves from PSI $\times$0, $\times$1, $\times$2, and $\times$3 with nDCG@10 of $\times$4 to $\times$5, $\times$6, $\times$7, and $\times$8 with nDCG@10 of $\times$9 (Michail et al., 1 Jun 2026).

For the cross-lingual $S \subseteq \{1,\ldots,n\}$00 setting, gte-multilingual-base moves from PSI $S \subseteq \{1,\ldots,n\}$01, $S \subseteq \{1,\ldots,n\}$02, $S \subseteq \{1,\ldots,n\}$03, and $S \subseteq \{1,\ldots,n\}$04 with nDCG@10 of $S \subseteq \{1,\ldots,n\}$05 to $S \subseteq \{1,\ldots,n\}$06, $S \subseteq \{1,\ldots,n\}$07, $S \subseteq \{1,\ldots,n\}$08, and $S \subseteq \{1,\ldots,n\}$09 with nDCG@10 of $S \subseteq \{1,\ldots,n\}$10. bge-m3-dense moves from PSI $S \subseteq \{1,\ldots,n\}$11, $S \subseteq \{1,\ldots,n\}$12, $S \subseteq \{1,\ldots,n\}$13, and $S \subseteq \{1,\ldots,n\}$14 with nDCG@10 of $S \subseteq \{1,\ldots,n\}$15 to $S \subseteq \{1,\ldots,n\}$16, $S \subseteq \{1,\ldots,n\}$17, $S \subseteq \{1,\ldots,n\}$18, and $S \subseteq \{1,\ldots,n\}$19 with nDCG@10 of $S \subseteq \{1,\ldots,n\}$20 (Michail et al., 1 Jun 2026). The study reports PSI reductions in all 16 length-quartile $S \subseteq \{1,\ldots,n\}$21 model $S \subseteq \{1,\ldots,n\}$22 retrieval-setting combinations, with the largest absolute reductions in longer quartiles, while overall nDCG@10 is preserved or slightly improved. By contrast, a Segment Embed Average baseline reduces PSI only by sacrificing $S \subseteq \{1,\ldots,n\}$23–$S \subseteq \{1,\ldots,n\}$24 nDCG@10 points (Michail et al., 1 Jun 2026).

The broader significance is methodological. PosIR makes it possible to verify whether a debiasing method improves position fairness without masking the outcome behind aggregate effectiveness alone. This suggests that PosIR functions as both a benchmark and an experimental control surface for position-robust retrieval research.

7. Research significance and open directions

PosIR is characterized in its benchmark paper as the first large-scale, multilingual, position-aware IR benchmark that strictly isolates the effects of document length and information position (Zeng et al., 13 Jan 2026). Its principal contribution is therefore diagnostic: it reveals that modern dense retrievers, including both bidirectional and causal architectures, can exhibit substantial and length-dependent position bias, and that these biases may not be visible on short-context leaderboards.

The benchmark’s findings motivate several research directions already articulated in the underlying work. These include position-aware training via span-balanced sampling across document positions, debiasing through adversarial perturbation of span positions or loss re-weighting, evaluation of cross-encoders, generative retrieval and retrieval-augmented generation systems, and extension of PosIR toward dynamic or user-study-based evaluation (Zeng et al., 13 Jan 2026). Later attention-calibration results add a complementary implication: some position bias can be mitigated at inference time, with no retraining and no per-model or per-language tuning, at least for the tested dense retrievers on PosIR (Michail et al., 1 Jun 2026).

Taken together, these results position PosIR as a benchmark for disentanglement rather than mere ranking. It measures not only whether a retriever finds relevant documents, but whether it does so uniformly across evidence locations, document lengths, domains, and languages. For long-context dense retrieval, that distinction is foundational rather than peripheral.