---
title: 'FineWeb-PosQ: Position-Aware Retrieval Benchmark'
url: https://www.emergentmind.com/topics/fineweb-posq
type: topic
---

# FineWeb-PosQ: Position-Aware Retrieval Benchmark

Searching arXiv for the provided papers to ground the article and verify identifiers.
arxiv_search: query="FineWeb-PosQ positional bias dense retrieval Spokes 2606.15216 2605.26578 2606.02737 2505.13950", max_results=10
arxiv_search({"query":"FineWeb-PosQ positional bias dense retrieval Spokes 2606.15216 2605.26578 2606.02737 2505.13950","max_results":10})
FineWeb-PosQ is primarily a position-aware retrieval benchmark derived from FineWeb-edu and designed to measure how retrieval effectiveness changes when query-relevant evidence occurs in the beginning, middle, or end of a long passage. In the positional-bias literature, it is used to expose the “Myopic Trap” of dense retrievers and to evaluate mitigation strategies at both training time and inference time [2505.13950], [2605.26578], [2606.02737]. The same label also appears in a distinct pretraining-data-selection setting, where “PosQ” denotes a positive-quality condition on FineWeb defined by FineWeb-Edu classifier scores and integrated into Spokes; that usage is technically separate from the retrieval benchmark [2606.15216].

## 1. Benchmark definition and corpus construction

FineWeb-PosQ is built from long passages sampled from FineWeb-edu, with passage length in \([500,1024]\) words. Each passage is divided into three contiguous thirds—Beginning, Middle, and End—and queries are generated so that the answer lies within a designated segment. The retrieval task keeps the document text unchanged and instead stratifies queries by answer location, making the benchmark semantics-preserving rather than based on artificial document rotation or content shifting [2505.13950], [2606.02737].

In one detailed specification, the benchmark contains 13,902 passages and 25,775 queries. The positional-group counts are Begin: 8,748, Middle: 8,414, and End: 8,613. The average passage length is 762 words with \(\sigma \approx 148\). Query generation uses GPT-4o-mini to produce one or more segment-focused questions for each segment, and each question is paired with its original passage as the relevant item while all other passages are treated as non-relevant under binary labels [2606.02737].

The positional grouping is defined over thirds of the passage. If a passage has length \(L\), then Group 1 (Begin) corresponds to tokens \(1 \ldots \lfloor L/3 \rfloor\), Group 2 (Middle) to tokens \(\lfloor L/3 \rfloor + 1 \ldots \lfloor 2L/3 \rfloor\), and Group 3 (End) to tokens \(\lfloor 2L/3 \rfloor + 1 \ldots L\) [2606.02737]. A central design property is that the same corpus is used throughout; only the query grouping changes with evidence position [2505.13950].

## 2. Evaluation formalism and bias metrics

FineWeb-PosQ is typically evaluated with nDCG@10 computed separately for each positional group. For a query \(q\), with binary relevance labels \(r_i \in \{0,1\}\) at ranks \(i=1,\dots,10\),
\[
\mathrm{DCG@10}(q)=\sum_{i=1}^{10}\frac{2^{r_i}-1}{\log_2(i+1)},
\qquad
\mathrm{nDCG@10}(q)=\frac{\mathrm{DCG@10}(q)}{\mathrm{IDCG@10}(q)}.
\]
Group-specific scores are then averaged over all queries in the corresponding positional group [2606.02737].

Several summaries of positional bias are used. A common measure is the Position Sensitivity Index,
\[
\mathrm{PSI}(s)=1-\frac{\min_p s_p}{\max_p s_p},
\]
with \(s=\{s_{\mathrm{B}},s_{\mathrm{M}},s_{\mathrm{E}}\}\) and \(\max_p s_p>0\). An equivalent form reported elsewhere is
\[
\mathrm{PSI}=\frac{\max_i \mathrm{nDCG@10}_i-\min_i \mathrm{nDCG@10}_i}{\max_i \mathrm{nDCG@10}_i}.
\]
A PSI of 0 indicates perfectly flat performance across positions; larger values indicate larger spread between the best and worst positional group [2605.26578], [2606.02737].

A second summary is the retrieval-performance gap
\[
\Delta(s)=\max_p s_p-\min_p s_p,
\]
which is proportional to PSI via \(\Delta=(\max s)\left(1-\min s/\max s\right)\) [2605.26578]. The “Myopic Trap” literature also defines a head-bias statistic
\[
H_m=S_m(B_{\mathrm{early}})-S_m(B_{\mathrm{late}}),
\]
where \(S_m(B)\) is average nDCG@10 for model \(m\) on bin \(B\). On FineWeb-PosQ this specializes to beginning-versus-end performance, so a large positive \(H_m\) indicates strong preference for early evidence [2505.13950].

Inference-time calibration work further reports the harmonic mean across positional groups,
\[
\mathrm{Harm}=\frac{3}{\sum_{i=1}^3 1/\mathrm{nDCG@10}_i},
\]
as a summary of consistency across Begin, Middle, and End. This metric emphasizes configurations that avoid catastrophic underperformance on any one position [2606.02737].

## 3. FineWeb-PosQ as a benchmark for the “Myopic Trap”

FineWeb-PosQ was introduced as part of a broader semantics-preserving framework for benchmarking positional bias in retrieval. The principal empirical finding is that dense first-stage retrieval often degrades as relevant evidence moves later in the passage, whereas BM25 and cross-encoder rerankers remain comparatively flat [2505.13950].

On FineWeb-PosQ, BM25 changes from 89.56 to 88.80 between beginning and end, corresponding to \(H \approx +0.8\). Single-vector embedding models show markedly larger drops: bge-m3-dense goes from 88.64 to 80.35 (\(H \approx +8.3\)); stella from 88.19 to 78.96 (\(H \approx +9.2\)); voyage from 92.65 to 87.96 (\(H \approx +4.7\)); text-embed-3-large from 86.09 to 82.09 (\(H \approx +4.0\)); gte from 87.45 to 81.79 (\(H \approx +5.7\)); and NV-embed-v2 from 87.35 to 88.10 (\(H \approx -0.8\)) [2505.13950].

ColBERT-style late interaction does not eliminate positional bias, but it can change its magnitude substantially. On FineWeb-PosQ, colbertv2.0 goes from 88.73 to 64.25, yielding \(H \approx +24.5\), while bge-m3-colbert goes from 92.08 to 86.66, yielding \(H \approx +5.4\). Cross-encoder rerankers remain close to position-flat; for example, bge-reranker-v2-m3 changes from 95.18 to 94.66 with \(H \approx +0.5\) [2505.13950].

These results support three benchmark-level conclusions. First, embedding-only first stages are prone to head bias on long web passages. Second, late interaction can mitigate bias under some training configurations, but not uniformly. Third, interaction-rich reranking appears substantially more robust to where the relevant span occurs in the document [2505.13950].

## 4. Training-position distribution as a control variable

A later line of work uses FineWeb-PosQ as an evaluation benchmark for studying whether positional bias is built into retriever architectures or learned from training data. The central formal object is a training-position distribution \(d\) over Beginning, Middle, and End, with \(d_{\mathrm{B}}+d_{\mathrm{M}}+d_{\mathrm{E}}=1\). Canonical configurations include \(d=(1,0,0)\) for begin-only, \(d=(0,1,0)\) for mid-only, \(d=(0,0,1)\) for end-only, and \(d=(1/3,1/3,1/3)\) for uniform training [2605.26578].

The controlled training data are synthetic rather than drawn from FineWeb itself. English Wikipedia articles are stratified into five length bins \(L=\{256\!-\!512,\dots,4096\!-\!8192\}\), cut into Beginning, Middle, and End segments, and paired with GPT-4 mini queries generated under a persona-conditioned prompt. Candidate triples are filtered by requiring unanimous agreement of three cross-encoder rerankers with a margin threshold \(\delta\). At \(\delta=0.3\), the pipeline retains 481,236 high-precision examples. Because retained counts remain skewed, the data are downsampled within each length bin to create four controlled splits; each split contains approximately 40k examples and has identical total size and length-bin composition [2605.26578].

The fine-tuning study covers eight pretrained models—BERT-base, Longformer-base, ModernBERT-base, ModernBERT-large, GPT-2-medium, BLOOM-560M, TinyLlama-NoPE, and Qwen3-0.6B—trained as dual-encoder retrievers with InfoNCE loss, chunk-aware negatives, batch size 256, AdamW, warmup 10%, 3 epochs, learning rate \(4\times 10^{-5}\) for models under 400M parameters and \(2\times 10^{-5}\) for larger models, similarity scale 20.0, seed 42, and no hard negatives or early stopping [2605.26578].

FineWeb-PosQ is then used to measure retrieval-level bias. Balanced training sharply reduces positional sensitivity. For ModernBERT-base, PSI falls from 0.476 under \(M_B\) to 0.108 under \(M_U\), a reduction of approximately 77.3%. For ModernBERT-large, PSI falls from 0.426 to 0.116, a reduction of approximately 72.8%. For Qwen3-0.6B, PSI falls from 0.359 to 0.116, a reduction of approximately 67.7% [2605.26578].

Balanced training is also competitive in mean retrieval quality on FineWeb-PosQ. ModernBERT-base has mean nDCG@10 values \(M_B=0.554\), \(M_M=0.570\), \(M_E=0.571\), and \(M_U=0.640\), with \(M_U\) best. ModernBERT-large has \(M_B=0.592\), \(M_M=0.647\), \(M_E=0.595\), and \(M_U=0.675\). Qwen3-0.6B has \(M_U=0.604\), which is best over 0.578, 0.533, and 0.535. This suggests that flattening positional sensitivity need not require sacrificing aggregate retrieval effectiveness [2605.26578].

Representation-level analyses in the same study further connect FineWeb-PosQ outcomes to embedding geometry. In an evidence-moving experiment, begin-trained models peak at early insertion positions, end-trained models peak at late positions, mid-trained models peak in the middle, and uniform training yields the smallest peak-to-trough gap \(\Delta\). For ModernBERT-base, \(\mathcal{M}_U\) is nearly flat with \(\Delta=1.9\times10^{-3}\), compared with \(\Delta=21.2\times10^{-3}\) for \(\mathcal{M}_B\) and \(\Delta=20.6\times10^{-3}\) for \(\mathcal{M}_E\) [2605.26578].

## 5. Inference-time attention calibration on FineWeb-PosQ

FineWeb-PosQ also serves as the main downstream testbed for an inference-time mitigation strategy based on attention calibration. The method modifies the pooling token’s attention distribution without retraining. Let \(A=(a_1,\dots,a_K)\) be the softmax-normalized attention weights over \(K\) key tokens. Keys are partitioned into contiguous baskets of size \(B\), and a fully calibrated distribution \(C\) redistributes mass uniformly over baskets:
\[
C_k=\left(\frac{a_k}{\sum_{j\in b(k)} a_j}\right)\cdot\left(\frac{1}{M}\right),
\]
where \(M\) is the number of baskets. A strength coefficient \(\lambda \in [0,1]\) interpolates between original and calibrated attention,
\[
\hat A=(1-\lambda)\cdot A+\lambda\cdot C.
\]
Calibration can be applied to the last layer only or to the last 50% of layers, with tested basket sizes \(B\in\{128,256,512\}\) and \(\lambda\in\{0.5,1.0\}\) plus a sweep in steps of .05 for ablation [2606.02737].

The benchmark results show a consistent directional effect: calibration lowers Begin performance and raises Middle and End performance, thereby reducing PSI. Partial calibration frequently outperforms full calibration in harmonic mean, especially when applied to half the layers [2606.02737].

| Model | No Cal. Harm / PSI | HL, \(\lambda=0.5\) Harm / PSI |
|---|---:|---:|
| GTE-multilingual-base | 78.66 / 0.160 | 79.47 / 0.122 |
| bge-m3-dense | 79.10 / 0.190 | 80.34 / 0.130 |
| Qwen3-Embedding-0.6B | 76.49 / 0.259 | 79.94 / 0.094 |

For GTE-multilingual-base, full half-layer calibration further lowers PSI to 0.068 but reduces Harm to 77.63. For bge-m3-dense, the corresponding values are PSI 0.068 and Harm 77.21. For Qwen3-Embedding-0.6B, half-layer full calibration gives Begin 80.94, Middle 80.46, End 76.46, Harm 79.23, and PSI 0.055 [2606.02737].

A single default setting—\(B=128\), \(\lambda=0.5\), and calibration on 50% of the layers—improves the harmonic mean of nDCG@10 across positional groups on FineWeb-PosQ for all three evaluated embedding models, without per-model tuning. The same work reports that this default transfers without modification to PosIR, reducing PSI in all 16 length-quartile \(\times\) model \(\times\) retrieval-setting combinations while preserving or improving aggregate nDCG@10 [2606.02737].

## 6. “PosQ” as positive-quality selection in FineWeb pretraining

A separate use of the label occurs in pretraining-data selection rather than retrieval benchmarking. In "Spokes: Optimizing for Diverse Pretraining Data Selection," “PosQ” on FineWeb means that positive quality scores from the FineWeb-Edu classifier are integrated into Spokes with a small \(\alpha=0.0005\) [2606.15216].

The underlying optimization starts from a fixed-size subset objective,
\[
\max_{S:|S|=k}\;
\alpha \ln\!\left(\frac{1}{k}\sum_{i\in S} q_i\right)
+
(1-\alpha)\ln \mathrm{G\mbox{-}Vendi}(K_S),
\]
where \(q_i\) is a per-example quality score and \(K_S\) is the submatrix of the cosine-similarity kernel. The continuous relaxation introduces \(w\in\Delta^n\), expected quality \(Q(w)=\sum_i w_i q_i\), a weighted cosine kernel \(K(w)\), and the relaxed objective
\[
\max_{w\in\Delta^n}\;
\alpha \ln Q(w)+(1-\alpha)\ln \mathrm{G\mbox{-}Vendi}(K(w)).
\]
Optimization uses exponentiated gradient descent with loss
\[
\mathcal{L}(w)=-
\big[\alpha \ln Q(w)+(1-\alpha)\ln \mathrm{G\mbox{-}Vendi}(K(w))\big],
\]
followed after \(T\) steps by selection of the top-\(k\) indices of \(w^{(T)}\). In practice \(T\lesssim 20\) [2606.15216].

For the pure quality baseline on FineWeb, the quality score is given by the FineWeb-Edu classifier, and selection simply thresholds \(q_i\) to keep the top 50% of FineWeb by score. The chosen threshold is \(q \ge 1.0693\), which yields a 50% data fraction. For joint quality-plus-diversity optimization on FineWeb, the reported settings are \(\alpha=0.0005\), gradient embeddings computed on Qwen3-0.6B-Base truncated to the last two transformer layers, a Rademacher JL projection with \(k=1024\) dimensions, the same overall data budget as quality-only via top-50% downstream weight ranking, and pretraining from scratch on LLaMA-1B for 100B tokens with batch size 2,048 and sequence length 4,096 [2606.15216].

On the OLMES world-knowledge suite of 10 English tasks, all models are pretrained on the same 100B-token budget of FineWeb with 50% data selected by each method. The reported averages are: FineWeb (Random) 45.1, FineWeb (SemDeDup) 44.0, FineWeb-Spokes (diversity-only, \(\alpha=0\)) 45.6, FineWeb (quality-only, \(q\ge1.0693\)) 44.6, and FineWeb-Spokes (joint, \(\alpha=0.0005\)) 46.5 [2606.15216].

Compared to Random (Avg 45.1), diversity-only Spokes gains +0.5 pt, the quality-only baseline loses \(-0.5\) pt, and joint Spokes gains +1.4 pt absolute to 46.5. On individual tasks, joint Spokes raises ARC-E from 52.4 to 54.7 (+2.3), HellaSwag from 46.4 to 47.9 (+1.5), and MMLU from 27.8 to 28.7 (+0.9) [2606.15216].

A common misconception is to treat this “PosQ” condition as interchangeable with the retrieval benchmark FineWeb-PosQ. The literature does not support that equivalence. In the retrieval papers, FineWeb-PosQ denotes a benchmark for answer-position sensitivity in long-passage retrieval [2505.13950], [2606.02737], [2605.26578]. In Spokes, “PosQ” denotes a positive-quality selection condition on FineWeb for pretraining-data optimization [2606.15216]. This suggests that the shared label names two distinct technical objects: one evaluative and position-aware, the other selective and quality-aware.

Source: https://www.emergentmind.com/topics/fineweb-posq