FineWeb-PosQ: Position-Aware Retrieval Benchmark

• FineWeb-PosQ is a position-aware benchmark derived from FineWeb-edu that stratifies long passages into beginning, middle, and end to assess retrieval performance.
• It employs GPT-generated queries and metrics such as nDCG@10 and PSI to quantify positional bias and the impact of evidence location.
• Results show dense retrievers exhibit strong head bias compared to rerankers, while calibration techniques can effectively reduce performance gaps.

Searching arXiv for the papers on arXiv to ground the article and verify identifiers. arxiv_search: query="FineWeb-PosQ positional bias dense retrieval Spokes (Lee et al., 13 Jun 2026, Yu et al., 26 May 2026, Michail et al., 1 Jun 2026, Zeng et al., 20 May 2025)", max_results=10 arxiv_search({"query":"FineWeb-PosQ positional bias dense retrieval Spokes (Lee et al., 13 Jun 2026, Yu et al., 26 May 2026, Michail et al., 1 Jun 2026, Zeng et al., 20 May 2025)","max_results":10}) FineWeb-PosQ is primarily a position-aware retrieval benchmark derived from FineWeb-edu and designed to measure how retrieval effectiveness changes when query-relevant evidence occurs in the beginning, middle, or end of a long passage. In the positional-bias literature, it is used to expose the “Myopic Trap” of dense retrievers and to evaluate mitigation strategies at both training time and inference time (Zeng et al., 20 May 2025, Yu et al., 26 May 2026, Michail et al., 1 Jun 2026). The same label also appears in a distinct pretraining-data-selection setting, where “PosQ” denotes a positive-quality condition on FineWeb defined by FineWeb-Edu classifier scores and integrated into Spokes; that usage is technically separate from the retrieval benchmark (Lee et al., 13 Jun 2026).

1. Benchmark definition and corpus construction

FineWeb-PosQ is built from long passages sampled from FineWeb-edu, with passage length in $[500,1024]$ words. Each passage is divided into three contiguous thirds—Beginning, Middle, and End—and queries are generated so that the answer lies within a designated segment. The retrieval task keeps the document text unchanged and instead stratifies queries by answer location, making the benchmark semantics-preserving rather than based on artificial document rotation or content shifting (Zeng et al., 20 May 2025, Michail et al., 1 Jun 2026).

In one detailed specification, the benchmark contains 13,902 passages and 25,775 queries. The positional-group counts are Begin: 8,748, Middle: 8,414, and End: 8,613. The average passage length is 762 words with $\sigma \approx 148$. Query generation uses GPT-4o-mini to produce one or more segment-focused questions for each segment, and each question is paired with its original passage as the relevant item while all other passages are treated as non-relevant under binary labels (Michail et al., 1 Jun 2026).

The positional grouping is defined over thirds of the passage. If a passage has length $L$, then Group 1 (Begin) corresponds to tokens $1 \ldots \lfloor L/3 \rfloor$, Group 2 (Middle) to tokens $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$, and Group 3 (End) to tokens $\lfloor 2L/3 \rfloor + 1 \ldots L$ (Michail et al., 1 Jun 2026). A central design property is that the same corpus is used throughout; only the query grouping changes with evidence position (Zeng et al., 20 May 2025).

2. Evaluation formalism and bias metrics

FineWeb-PosQ is typically evaluated with nDCG@10 computed separately for each positional group. For a query $q$, with binary relevance labels $r_i \in \{0,1\}$ at ranks $i=1,\dots,10$,

$$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$

Group-specific scores are then averaged over all queries in the corresponding positional group (Michail et al., 1 Jun 2026).

Several summaries of positional bias are used. A common measure is the Position Sensitivity Index,

$\sigma \approx 148$0

with $\sigma \approx 148$1 and $\sigma \approx 148$2. An equivalent form reported elsewhere is

$\sigma \approx 148$3

A PSI of 0 indicates perfectly flat performance across positions; larger values indicate larger spread between the best and worst positional group (Yu et al., 26 May 2026, Michail et al., 1 Jun 2026).

A second summary is the retrieval-performance gap

$\sigma \approx 148$4

which is proportional to PSI via $\sigma \approx 148$5 (Yu et al., 26 May 2026). The “Myopic Trap” literature also defines a head-bias statistic

$\sigma \approx 148$6

where $\sigma \approx 148$7 is average nDCG@10 for model $\sigma \approx 148$8 on bin $\sigma \approx 148$9. On FineWeb-PosQ this specializes to beginning-versus-end performance, so a large positive $L$0 indicates strong preference for early evidence (Zeng et al., 20 May 2025).

Inference-time calibration work further reports the harmonic mean across positional groups,

$L$1

as a summary of consistency across Begin, Middle, and End. This metric emphasizes configurations that avoid catastrophic underperformance on any one position (Michail et al., 1 Jun 2026).

3. FineWeb-PosQ as a benchmark for the “Myopic Trap”

FineWeb-PosQ was introduced as part of a broader semantics-preserving framework for benchmarking positional bias in retrieval. The principal empirical finding is that dense first-stage retrieval often degrades as relevant evidence moves later in the passage, whereas BM25 and cross-encoder rerankers remain comparatively flat (Zeng et al., 20 May 2025).

On FineWeb-PosQ, BM25 changes from 89.56 to 88.80 between beginning and end, corresponding to $L$2. Single-vector embedding models show markedly larger drops: bge-m3-dense goes from 88.64 to 80.35 ($L$3); stella from 88.19 to 78.96 ($L$4); voyage from 92.65 to 87.96 ($L$5); text-embed-3-large from 86.09 to 82.09 ($L$6); gte from 87.45 to 81.79 ($L$7); and NV-embed-v2 from 87.35 to 88.10 ($L$8) (Zeng et al., 20 May 2025).

ColBERT-style late interaction does not eliminate positional bias, but it can change its magnitude substantially. On FineWeb-PosQ, colbertv2.0 goes from 88.73 to 64.25, yielding $L$9, while bge-m3-colbert goes from 92.08 to 86.66, yielding $1 \ldots \lfloor L/3 \rfloor$0. Cross-encoder rerankers remain close to position-flat; for example, bge-reranker-v2-m3 changes from 95.18 to 94.66 with $1 \ldots \lfloor L/3 \rfloor$1 (Zeng et al., 20 May 2025).

These results support three benchmark-level conclusions. First, embedding-only first stages are prone to head bias on long web passages. Second, late interaction can mitigate bias under some training configurations, but not uniformly. Third, interaction-rich reranking appears substantially more robust to where the relevant span occurs in the document (Zeng et al., 20 May 2025).

4. Training-position distribution as a control variable

A later line of work uses FineWeb-PosQ as an evaluation benchmark for studying whether positional bias is built into retriever architectures or learned from training data. The central formal object is a training-position distribution $1 \ldots \lfloor L/3 \rfloor$2 over Beginning, Middle, and End, with $1 \ldots \lfloor L/3 \rfloor$3. Canonical configurations include $1 \ldots \lfloor L/3 \rfloor$4 for begin-only, $1 \ldots \lfloor L/3 \rfloor$5 for mid-only, $1 \ldots \lfloor L/3 \rfloor$6 for end-only, and $1 \ldots \lfloor L/3 \rfloor$7 for uniform training (Yu et al., 26 May 2026).

The controlled training data are synthetic rather than drawn from FineWeb itself. English Wikipedia articles are stratified into five length bins $1 \ldots \lfloor L/3 \rfloor$8, cut into Beginning, Middle, and End segments, and paired with GPT-4 mini queries generated under a persona-conditioned prompt. Candidate triples are filtered by requiring unanimous agreement of three cross-encoder rerankers with a margin threshold $1 \ldots \lfloor L/3 \rfloor$9. At $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$0, the pipeline retains 481,236 high-precision examples. Because retained counts remain skewed, the data are downsampled within each length bin to create four controlled splits; each split contains approximately 40k examples and has identical total size and length-bin composition (Yu et al., 26 May 2026).

The fine-tuning study covers eight pretrained models—BERT-base, Longformer-base, ModernBERT-base, ModernBERT-large, GPT-2-medium, BLOOM-560M, TinyLlama-NoPE, and Qwen3-0.6B—trained as dual-encoder retrievers with InfoNCE loss, chunk-aware negatives, batch size 256, AdamW, warmup 10%, 3 epochs, learning rate $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$1 for models under 400M parameters and $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$2 for larger models, similarity scale 20.0, seed 42, and no hard negatives or early stopping (Yu et al., 26 May 2026).

FineWeb-PosQ is then used to measure retrieval-level bias. Balanced training sharply reduces positional sensitivity. For ModernBERT-base, PSI falls from 0.476 under $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$3 to 0.108 under $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$4, a reduction of approximately 77.3%. For ModernBERT-large, PSI falls from 0.426 to 0.116, a reduction of approximately 72.8%. For Qwen3-0.6B, PSI falls from 0.359 to 0.116, a reduction of approximately 67.7% (Yu et al., 26 May 2026).

Balanced training is also competitive in mean retrieval quality on FineWeb-PosQ. ModernBERT-base has mean nDCG@10 values $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$5, $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$6, $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$7, and $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$8, with $$\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor$$9 best. ModernBERT-large has $\lfloor 2L/3 \rfloor + 1 \ldots L$0, $\lfloor 2L/3 \rfloor + 1 \ldots L$1, $\lfloor 2L/3 \rfloor + 1 \ldots L$2, and $\lfloor 2L/3 \rfloor + 1 \ldots L$3. Qwen3-0.6B has $\lfloor 2L/3 \rfloor + 1 \ldots L$4, which is best over 0.578, 0.533, and 0.535. This suggests that flattening positional sensitivity need not require sacrificing aggregate retrieval effectiveness (Yu et al., 26 May 2026).

Representation-level analyses in the same study further connect FineWeb-PosQ outcomes to embedding geometry. In an evidence-moving experiment, begin-trained models peak at early insertion positions, end-trained models peak at late positions, mid-trained models peak in the middle, and uniform training yields the smallest peak-to-trough gap $\lfloor 2L/3 \rfloor + 1 \ldots L$5. For ModernBERT-base, $\lfloor 2L/3 \rfloor + 1 \ldots L$6 is nearly flat with $\lfloor 2L/3 \rfloor + 1 \ldots L$7, compared with $\lfloor 2L/3 \rfloor + 1 \ldots L$8 for $\lfloor 2L/3 \rfloor + 1 \ldots L$9 and $q$0 for $q$1 (Yu et al., 26 May 2026).

5. Inference-time attention calibration on FineWeb-PosQ

FineWeb-PosQ also serves as the main downstream testbed for an inference-time mitigation strategy based on attention calibration. The method modifies the pooling token’s attention distribution without retraining. Let $q$2 be the softmax-normalized attention weights over $q$3 key tokens. Keys are partitioned into contiguous baskets of size $q$4, and a fully calibrated distribution $q$5 redistributes mass uniformly over baskets: $q$6 where $q$7 is the number of baskets. A strength coefficient $q$8 interpolates between original and calibrated attention,

$q$9

Calibration can be applied to the last layer only or to the last 50% of layers, with tested basket sizes $r_i \in \{0,1\}$0 and $r_i \in \{0,1\}$1 plus a sweep in steps of .05 for ablation (Michail et al., 1 Jun 2026).

The benchmark results show a consistent directional effect: calibration lowers Begin performance and raises Middle and End performance, thereby reducing PSI. Partial calibration frequently outperforms full calibration in harmonic mean, especially when applied to half the layers (Michail et al., 1 Jun 2026).

Model	No Cal. Harm / PSI	HL, $r_i \in \{0,1\}$2 Harm / PSI
GTE-multilingual-base	78.66 / 0.160	79.47 / 0.122
bge-m3-dense	79.10 / 0.190	80.34 / 0.130
Qwen3-Embedding-0.6B	76.49 / 0.259	79.94 / 0.094

For GTE-multilingual-base, full half-layer calibration further lowers PSI to 0.068 but reduces Harm to 77.63. For bge-m3-dense, the corresponding values are PSI 0.068 and Harm 77.21. For Qwen3-Embedding-0.6B, half-layer full calibration gives Begin 80.94, Middle 80.46, End 76.46, Harm 79.23, and PSI 0.055 (Michail et al., 1 Jun 2026).

A single default setting—$r_i \in \{0,1\}$3, $r_i \in \{0,1\}$4, and calibration on 50% of the layers—improves the harmonic mean of nDCG@10 across positional groups on FineWeb-PosQ for all three evaluated embedding models, without per-model tuning. The same work reports that this default transfers without modification to PosIR, reducing PSI in all 16 length-quartile $r_i \in \{0,1\}$5 model $r_i \in \{0,1\}$6 retrieval-setting combinations while preserving or improving aggregate nDCG@10 (Michail et al., 1 Jun 2026).

6. “PosQ” as positive-quality selection in FineWeb pretraining

A separate use of the label occurs in pretraining-data selection rather than retrieval benchmarking. In "Spokes: Optimizing for Diverse Pretraining Data Selection," “PosQ” on FineWeb means that positive quality scores from the FineWeb-Edu classifier are integrated into Spokes with a small $r_i \in \{0,1\}$7 (Lee et al., 13 Jun 2026).

The underlying optimization starts from a fixed-size subset objective,

$r_i \in \{0,1\}$8

where $r_i \in \{0,1\}$9 is a per-example quality score and $i=1,\dots,10$0 is the submatrix of the cosine-similarity kernel. The continuous relaxation introduces $i=1,\dots,10$1, expected quality $i=1,\dots,10$2, a weighted cosine kernel $i=1,\dots,10$3, and the relaxed objective

$i=1,\dots,10$4

Optimization uses exponentiated gradient descent with loss

$i=1,\dots,10$5

followed after $i=1,\dots,10$6 steps by selection of the top-$i=1,\dots,10$7 indices of $i=1,\dots,10$8. In practice $i=1,\dots,10$9 (Lee et al., 13 Jun 2026).

For the pure quality baseline on FineWeb, the quality score is given by the FineWeb-Edu classifier, and selection simply thresholds $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$0 to keep the top 50% of FineWeb by score. The chosen threshold is $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$1, which yields a 50% data fraction. For joint quality-plus-diversity optimization on FineWeb, the reported settings are $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$2, gradient embeddings computed on Qwen3-0.6B-Base truncated to the last two transformer layers, a Rademacher JL projection with $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$3 dimensions, the same overall data budget as quality-only via top-50% downstream weight ranking, and pretraining from scratch on LLaMA-1B for 100B tokens with batch size 2,048 and sequence length 4,096 (Lee et al., 13 Jun 2026).

On the OLMES world-knowledge suite of 10 English tasks, all models are pretrained on the same 100B-token budget of FineWeb with 50% data selected by each method. The reported averages are: FineWeb (Random) 45.1, FineWeb (SemDeDup) 44.0, FineWeb-Spokes (diversity-only, $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$4) 45.6, FineWeb (quality-only, $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$5) 44.6, and FineWeb-Spokes (joint, $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$6) 46.5 (Lee et al., 13 Jun 2026).

Compared to Random (Avg 45.1), diversity-only Spokes gains +0.5 pt, the quality-only baseline loses $$\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)}, \qquad \mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.$$7 pt, and joint Spokes gains +1.4 pt absolute to 46.5. On individual tasks, joint Spokes raises ARC-E from 52.4 to 54.7 (+2.3), HellaSwag from 46.4 to 47.9 (+1.5), and MMLU from 27.8 to 28.7 (+0.9) (Lee et al., 13 Jun 2026).

A common misconception is to treat this “PosQ” condition as interchangeable with the retrieval benchmark FineWeb-PosQ. The literature does not support that equivalence. In the retrieval papers, FineWeb-PosQ denotes a benchmark for answer-position sensitivity in long-passage retrieval (Zeng et al., 20 May 2025, Michail et al., 1 Jun 2026, Yu et al., 26 May 2026). In Spokes, “PosQ” denotes a positive-quality selection condition on FineWeb for pretraining-data optimization (Lee et al., 13 Jun 2026). This suggests that the shared label names two distinct technical objects: one evaluative and position-aware, the other selective and quality-aware.