SQuAD-PosQ in QA and Retrieval

• SQuAD-PosQ is a dual-purpose label from SQuAD literature used for both answerability-aware machine reading comprehension and position-aware retrieval benchmarks.
• In the QA formulation, models integrate span extraction with no-answer prediction to balance accurate answer selection and abstention from adversarial unanswerable questions.
• For retrieval, the label stratifies answer spans by their character-level positions to quantify positional bias and assess passage ranking effectiveness.

SQuAD-PosQ is a label used in two distinct ways in the SQuAD literature. In one usage, it denotes the answerability-centric formulation induced by SQuAD 2.0, in which a system must extract a span when a paragraph supports the question and abstain when it does not (Rajpurkar et al., 2018). In another, later usage, it denotes a semantics-preserving, position-aware retrieval benchmark derived from the answerable subset of SQuAD v2, where queries are stratified by the character-level start position of the gold answer span in order to measure positional bias (Zeng et al., 20 May 2025). Both usages descend from the original SQuAD benchmark, whose initial release was a positive-only span-extraction dataset of 107,785 question–answer pairs over 536 Wikipedia articles, with every question answerable by a contiguous span from the passage (Rajpurkar et al., 2016).

1. Origins in SQuAD and the emergence of the PosQ framing

The original Stanford Question Answering Dataset was explicitly positive-only: each question was answerable by extracting a contiguous text span from its associated Wikipedia paragraph (Rajpurkar et al., 2016). The dataset contained 107,785 question–answer pairs on 536 articles and 23,215 paragraphs, and evaluation was based on Exact Match and token-level F1. This formulation made SQuAD a span-extraction benchmark in which the central problem was locating the correct answer boundaries, not deciding whether an answer existed.

SQuAD 2.0 changed that formulation by combining the original answerable questions with over 50,000 adversarially written unanswerable questions designed to look answerable (Rajpurkar et al., 2018). The task therefore acquired a dual objective: answerability classification and span extraction. Models were no longer permitted to assume that every input admitted a valid span; instead, they had to determine when the paragraph failed to support the question and output a null prediction.

In the papers considered here, the label “SQuAD-PosQ” is used for both the answerability-sensitive reading-comprehension task and the later retrieval benchmark built from answerable SQuAD v2 instances. The distinction is structural rather than terminological.

Formulation	Construction	Primary task
SQuAD v1.1 PosQ	107,785 answerable question–answer pairs	span extraction
SQuAD 2.0 SQuAD-PosQ	answerable questions plus over 50,000 adversarial unanswerables	answerability classification plus span extraction
Retrieval SQuAD-PosQ	92,749 answerable SQuAD v2 examples binned by answer position	position-aware passage ranking

2. Answerability-aware machine reading comprehension

Under the SQuAD 2.0 formulation, a model produces both span scores and a no-answer score or probability, and abstains when the no-answer signal exceeds a threshold or when the best span is insufficiently better than the null alternative (Rajpurkar et al., 2018). The answerability label space is binary: positive for answerable questions and negative for unanswerable ones. The resulting task is not merely span prediction with an extra class; it is a calibrated decision problem in which extraction and abstention interact directly.

The official metrics remain Exact Match and token-level F1, but both are defined so as to incorporate abstention. For unanswerable questions, predicting no-answer yields EM = 1 and F1 = 1, whereas predicting any span yields zero on both metrics. For answerable questions, the usual span-based EM and token-overlap F1 are applied. As a result, leaderboard performance measures the combined quality of answerability decisions and span localization rather than either component in isolation (Rajpurkar et al., 2018).

This framing also altered model design practice. A common learning objective became

$L = L_{\text{span}} + \lambda L_{\text{noans}},$

with cross-entropy for the start and end indices and binary cross-entropy for the no-answer prediction. At inference time, systems typically either threshold directly on $p(\text{no-answer})$ or compare a scalar null score against the best-span score. The threshold is tuned on the development set, so calibration is operationally important rather than an auxiliary diagnostic (Rajpurkar et al., 2018).

The motivation for this answerability-centric view is empirical as well as formal. SQuAD 2.0 exposed the brittleness of models that always extract a span: a strong neural system that achieved around 86% F1 on SQuAD 1.1 achieved only roughly 66% F1 on SQuAD 2.0, largely because adversarial unanswerables induce false positives through high lexical overlap and plausible distractor spans (Rajpurkar et al., 2018).

3. Extended Stochastic Answer Networks and joint span–answerability modeling

A canonical answerability-aware architecture for this setting is the extended Stochastic Answer Network, which explicitly couples a span detector with a binary classifier and optimizes both components jointly (Liu et al., 2018). In the formulation described there, SQuAD-PosQ refers to the positive-question detection problem within SQuAD 2.0: the system must both extract a span when an answer exists and abstain when the question cannot be answered from the passage.

The model retains SAN’s shared lower layers and adds a lightweight task-specific classifier. Lexicon encoding uses 300-d GloVe, 16-d POS, 8-d NER, and 4-d hand-crafted matching features, plus question-enhanced passage word embeddings. Question and passage encodings are projected to a common dimension, yielding $E^q \in \mathbb{R}^{d \times m}$ and $E^p \in \mathbb{R}^{d \times n}$. A shared 2-layer BiLSTM then encodes both question and passage, with 600-d CoVe vectors concatenated to the inputs and between the two BiLSTM layers to reduce overfitting, producing $H^q \in \mathbb{R}^{4d \times m}$ and $H^p \in \mathbb{R}^{4d \times n}$.

Memory generation constructs a question-aware passage representation and then applies self-attention. After mapping $H^q$ and $H^p$ through a one-layer ReLU network, the model computes question–passage attention

$$C = \text{dropout}(f_{\text{attention}}(\hat{H}^q, \hat{H}^p)) \in \mathbb{R}^{m \times n},$$

forms

$U^p = \text{concat}(H^p, H^q C),$

applies self-attention on $p(\text{no-answer})$0 with the diagonal self-similarity dropped, and runs a BiLSTM over the concatenated result to obtain the working memory $p(\text{no-answer})$1.

On top of $p(\text{no-answer})$2, SAN performs multi-step reasoning with a recurrent controller. The controller is initialized by a question summary

$p(\text{no-answer})$3

and at each step attends to memory conditioned on the previous state, produces a memory summary $p(\text{no-answer})$4, and updates the state with a GRU:

$p(\text{no-answer})$5

At each reasoning step, begin and end distributions are computed bilinearly over memory tokens,

$p(\text{no-answer})$6

and then averaged across steps. Decoding uses a special NULL token appended to the passage; if the best span points to that last token, the prediction is “no answer.”

The answerability classifier pools the initial question summary $p(\text{no-answer})$7 and a memory summary $p(\text{no-answer})$8 and predicts

$p(\text{no-answer})$9

where $E^q \in \mathbb{R}^{d \times m}$0 is the predicted probability that the question is unanswerable. Training uses the joint objective

$E^q \in \mathbb{R}^{d \times m}$1

with $E^q \in \mathbb{R}^{d \times m}$2 by default. Two inference policies were studied. “Joint SAN” makes the decision through the span head alone, using the appended NULL token; “Joint SAN + Classifier” explicitly thresholds $E^q \in \mathbb{R}^{d \times m}$3 at 0.5 and returns an empty string when that threshold is exceeded (Liu et al., 2018).

4. Empirical behavior, ablations, and related posterior formulations

On the SQuAD 2.0 development set, the span-only SAN baseline with NULL obtained EM 67.89 and F1 70.68, while the jointly trained SAN obtained EM 69.27 and F1 72.20; adding explicit classifier thresholding yielded EM 69.54 and F1 72.66 (Liu et al., 2018). The joint loss therefore improved over the span-only baseline by +1.38 EM and +1.52 F1, and the explicit classifier added a further +0.46 F1. On the test set, Joint SAN attained EM 68.7 and F1 71.4. In the reported comparison without large pre-trained LLMs such as ELMo, Joint SAN reached EM 69.3 and F1 72.2 on dev and outperformed contemporaries such as R.M-Reader + Verifier at EM 68.5 and F1 71.5 (Liu et al., 2018).

The ablation behavior clarifies the interaction between extraction and abstention. Classifier accuracy on dev was 75.3% at threshold 0.5; increasing $E^q \in \mathbb{R}^{d \times m}$4 to 1.5 raised classifier accuracy to 76.8% but improved overall F1 by only about 0.2, indicating that better binary classification does not automatically translate into better joint QA behavior (Liu et al., 2018). The reported error modes include miscalibration between span confidence and $E^q \in \mathbb{R}^{d \times m}$5, false positives on adversarial unanswerables, over-abstention on difficult answerables, and the limited expressive power of a one-layer global-pooling classifier.

A related line of work made the joint structure even more explicit by modeling the posterior

$E^q \in \mathbb{R}^{d \times m}$6

where $E^q \in \mathbb{R}^{d \times m}$7 is answerability and $E^q \in \mathbb{R}^{d \times m}$8 are start and end indices (Yang et al., 2019). In that formulation, valid spans with $E^q \in \mathbb{R}^{d \times m}$9 represent answerable mass and invalid spans with $E^p \in \mathbb{R}^{d \times n}$0 represent no-answer mass, avoiding a sentinel token. The reported best model combined BERT word-level embeddings, character-level BiDAF attention, and this joint posterior predictor, and obtained F1 75.842% and EM 72.24% on the test PCE leaderboard (Yang et al., 2019).

Later work with a simplified ALBERT Retro-Reader likewise treated SQuAD 2.0 as a joint span-plus-answerability problem, using a span head and a [CLS]-based no-answer head. In that setting, re-initializing the last linear layer inside the ALBERT Transformer block before fine-tuning produced a consistent approximately 2-point absolute F1 advantage from epochs 2 to 4 on SQuAD 2.0 development experiments (Ho et al., 2021). This suggests that, within the answerability-centric interpretation of SQuAD-PosQ, architectural advances and optimization tricks continued to target calibration between span extraction and abstention rather than span extraction alone.

5. SQuAD-PosQ as a position-aware retrieval benchmark

In the retrieval literature, SQuAD-PosQ denotes a different object: a position-aware benchmark repurposed from SQuAD v2 to quantify positional bias in retrieval systems (Zeng et al., 20 May 2025). The construction merges the official SQuAD v2 train and dev sets, removes all unanswerable questions, and yields 92,749 answerable examples. Each example is a triple $E^p \in \mathbb{R}^{d \times n}$1, where the answer start is the character-level index provided by SQuAD v2. The benchmark is evaluated over 20,233 unique passages, and position sensitivity is measured by stratifying examples into six inclusive answer-start bins: [0–100], [100–200], [200–300], [300–400], [400–500], and [500–3120].

The benchmark is explicitly semantics-preserving: no content is edited, shifted, or padded. Position varies naturally in the source corpus, and retrieval effectiveness is compared across the positional buckets rather than under synthetic perturbation. A 10,000-example SQuAD-PosQ-Tiny subset is also defined for compute-heavy models, while preserving the same retrieval corpus (Zeng et al., 20 May 2025).

Evaluation uses NDCG@10, and the study reports model-family-specific positional degradation patterns. BM25 is largely stable across buckets, with NDCG@10 values 76.62, 79.37, 80.61, 81.06, 81.43, and 79.49 from the earliest to the latest bin. Single-vector embedding retrievers generally degrade as relevant content moves later in the passage: bge-m3-dense drops from 84.47 to 74.61, stella_en_400M_v5 from 85.78 to 75.69, text-embedding-3-large from 85.19 to 71.10, and gte-Qwen2-7B-instruct from 85.13 to 77.75, while NV-embed-v2 is more robust at 93.04 to 90.72 (Zeng et al., 20 May 2025).

Late-interaction retrievers reduce but do not remove the effect. colbertv2.0 drops from 91.85 to 84.57, and bge-m3-colbert from 89.88 to 86.36. Cross-encoder rerankers remain effectively stable, with bge-reranker-v2-m3 at approximately 93.5–94.7 across bins, gte-reranker-base at approximately 90.7–92.6, and bge-reranker-gemma at approximately 94.0–94.8 (Zeng et al., 20 May 2025).

Retrieval family	Representative positional trend	Example figures
BM25	largely unaffected	76.62 to 79.49 across bins
Single-vector embeddings	substantial late-position degradation	bge-m3-dense 84.47 to 74.61
ColBERT-style late interaction	smaller but persistent degradation	bge-m3-colbert 89.88 to 86.36
Cross-encoder rerankers	effectively stable	bge-reranker-v2-m3 ~93.5–94.7

The benchmark also includes a representation-level probe. On a 10k sample from SQuAD v2, the cosine similarity between the full-text embedding and segment embeddings is consistently highest for the beginning segment and lower for middle and end segments across most embedding models; for bge-m3-dense, the reported values are 0.8777 for the beginning, 0.7957 for the middle, and 0.7727 for the end (Zeng et al., 20 May 2025). The paper interprets this as an embedding-level signature of a “Myopic Trap,” namely a tendency to overweight early passage content.

6. Attention calibration, limitations, and current significance

Subsequent work used SQuAD-PosQ as a testbed for inference-time attention calibration in dense retrieval (Michail et al., 1 Jun 2026). In that setting, the method operates only at inference time and only on the pooling token’s attention row in selected transformer layers when encoding documents; queries are left unchanged. The key sequence is partitioned into contiguous baskets of size $E^p \in \mathbb{R}^{d \times n}$2, attention mass is equalized across baskets while preserving relative weights within each basket, and a strength coefficient $E^p \in \mathbb{R}^{d \times n}$3 interpolates between the original and calibrated attention distributions.

On SQuAD-PosQ, evaluation is reported per positional group using nDCG@10, together with the harmonic mean across groups,

$E^p \in \mathbb{R}^{d \times n}$4

and the Position Sensitivity Index, where lower PSI indicates less positional sensitivity (Michail et al., 1 Jun 2026). Because SQuAD passages are short on average, approximately 117 words per passage, calibration has limited room to redistribute attention across multiple baskets. The reported consequence is that improvements concentrate mainly in the 500+ bin and are often offset by small drops in earlier bins.

The model-agnostic default identified on longer-document benchmarks was $E^p \in \mathbb{R}^{d \times n}$5, $E^p \in \mathbb{R}^{d \times n}$6, and calibration of the last 50% of layers. On SQuAD-PosQ, however, this default reduced PSI without improving Harm: for gte-multilingual-base, baseline Harm 83.01 and PSI 0.072 became Harm 81.33 and PSI 0.027; for bge-m3-dense, 79.30 and 0.117 became 78.19 and 0.055; for Qwen3-Embedding-0.6B, 76.89 and 0.135 became 73.74 and 0.067 (Michail et al., 1 Jun 2026). Partial last-layer-only calibration was generally a milder trade-off on SQuAD’s short passages.

A practical implication is that references to SQuAD-PosQ benefit from explicit disambiguation. In answerability-aware reading comprehension, the label denotes a joint abstention-and-extraction problem over SQuAD 2.0, exemplified by models such as extended SAN and later posterior-factorized systems (Liu et al., 2018, Yang et al., 2019). In retrieval research, the same label denotes an answerable-only, position-stratified benchmark for measuring positional robustness in passage ranking (Zeng et al., 20 May 2025, Michail et al., 1 Jun 2026). The shared name reflects a common ancestry in SQuAD, but the underlying supervision, metrics, and scientific questions are different.