---
title: 'SPBERTQA: Span Pre-training & QA Retrieval'
url: https://www.emergentmind.com/topics/spbertqa
type: topic
---

# SPBERTQA: Span Pre-training & QA Retrieval

SPBERTQA refers in the literature both to advances in question answering (QA) pre-training objectives for language models and to practical two-stage QA retrieval systems. The term is most notably applied to a span selection pre-training objective for BERT and a BM25+SBERT-based answer retrieval pipeline, depending on context. This entry covers the principal variants as described in "Span Selection Pre-training for Question Answering" [1909.04120] and "SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts" [2206.09600].

## 1. Span Selection Pre-training for Question Answering

The SPBERTQA pre-training paradigm introduced in [1909.04120] seeks to adapt the inductive bias of BERT from token-level cloze tasks toward the extractive QA paradigm, where the model must select answer spans from natural passages given a question.

### Construction and Objective

Each training instance involves:
- A query $Q$ (natural sentence with answer masked as a special [BLANK] token).
- A passage $P$ (Wikipedia paragraph retrieved using Lucene BM25).
- An answer span contained in $P$ matching the originally masked portion of $Q$.

The SPBERTQA pre-training loss supplements BERT's original masked language model (MLM) and next sentence prediction (NSP) with a span selection loss. This encompasses:
- Pointer-based start/end prediction over the concatenated $Q$ and $P$.
- (Optionally) a binary answerability classification for "impossible" instances where no answer string is present.

Mathematically, for each instance:
\[
\mathcal{L}_{\mathrm{span}} = - [\log p(i=s^*) + \log p(i=e^*)] - [y\log p(\mathrm{possible}) + (1-y)\log(1-p(\mathrm{possible}))]
\]
with the total loss
\[
\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{MLM}} + \mathcal{L}_{\mathrm{NSP}} + \lambda \mathcal{L}_{\mathrm{span}}
\]
where $\lambda=1$ in practice.

### Data Creation

Instances are synthesized from Wikipedia by automatically selecting answer spans via heuristics (POS/NER), removing superficial lexical signals, and retrieving passages with BM25 ($k_1=1.2, b=0.75$). About 30% of instances are "impossible" with no exact answer present.

### Architectural Modifications

A single new [BLANK] token is added to the vocabulary, and linear heads are introduced for start/end prediction and, optionally, answerability. The transformer encoder remains unchanged.

### Pre-training Regimen

The model is initialized from BERT (base or large), and trained for 100M span-selection instances (batch size 256, LR $5\times10^{-5}$), requiring about 40% more compute than standard BERT pre-training.

## 2. Downstream QA Adaptation and Results

SPBERTQA is directly fine-tuned on standard extractive QA benchmarks:

- SQuAD 1.1/2.0: Performance increases over BERT-base and BERT-large, with +SSPT-large achieving F1 of 92.75/86.86 (SQuAD 1.1 F1/EM) and 85.03/82.07 (SQuAD 2.0 F1/EM), a substantial gain over non-span-pretrained models.
- Natural Questions: +SSPT-large yields short-answer F1 of 54.2 vs. 52.7 for BERT-large.
- HotpotQA: Joint reasoning with local/global encoding produces answer F1 of 86.17 and supporting fact F1 of 79.39 (base/large settings).

A pronounced advantage is observed in limited-resource settings: on SQuAD, the span-pretrained model gains 5–6 F1 over BERT at 10% supervised data.

| Model       | SQuAD 1.1 F1 | NaturalQ Short F1 | HotpotQA Ans F1 | Low-data Gain |
|-------------|-------------|-------------------|-----------------|--------------|
| BERT-large  | 90.97       | 52.7              | 85.27           | –            |
| +SSPT-large | 92.75       | 54.2              | 86.17           | +5–6 F1      |

## 3. BM25+SBERT-Based Answer Selection for Low-Resource Languages

A separate system named SPBERTQA is proposed in [2206.09600] for Vietnamese medical QA using a two-stage extractive pipeline:

1. **BM25-based sentence retrieval:** Passages (avg. 495 words, often exceeding transformer length limits) are split into sentences, then ranked via BM25. For each passage, the top $K=5$ sentences with highest BM25 overlap to the query are selected.
2. **SBERT semantic re-ranking:** Both the question and the K-sentence candidate passage are encoded by Sentence-BERT (Siamese PhoBERT) trained with Multiple Negatives Ranking (MNR) loss. Embeddings use mean pooling; similarity is cosine distance.

Key features of the architecture:
- Input truncation at 256 tokens, using VnCoreNLP for segmentation.
- Optimal performance with PhoBERT as the base encoder following contrastive (MNR) fine-tuning.
- Retrieval and re-ranking metrics are Precision@K and mean Average Precision.

On the ViHealthQA Vietnamese medical test set (2013 examples):

| Model          | P@1 (%) | P@10 (%) | mAP (%) |
|----------------|---------|----------|---------|
| BM25 baseline  | 44.96   | 70.09    | 56.93   |
| BM25-SXLMR     | 46.05   | 79.04    | 53.85   |
| SPBERTQA       | 50.92   | 83.76    | 62.25   |

SPBERTQA surpasses all bag-of-words and multilingual BERT/XLM-R baselines, especially at low lexical overlap (P@1 of 50–80% even with 0–3 shared words), verifying its robust lexical-gap generalization [2206.09600].

## 4. Analysis, Limitations, and Future Directions

For the span selection pre-training approach:
- The method tightly aligns model pre-training with downstream extractive QA, requiring models to find spans (not generate answers), resulting in more efficient transfer and learning, as demonstrated empirically by significant F1/EM gains and faster convergence in low-resource scenarios [1909.04120].
- There is no modification to the core transformer; all changes are at the input or output head level.

For the retrieval-based system:
- The two-stage mechanism addresses token-length limitations in input representations and the lexical-semantic gap in Vietnamese medical QA.
- Limitations include the absence of explicit answer span highlighting in retrieved passages; only sentence-level extraction is provided. A plausible implication is that integrating a machine reading comprehension span predictor atop the system could deliver concise answer extraction.

Both lines of work demonstrate the effectiveness of specialized pre-training (for span selection) and monolingual semantic encoding (contrastive PhoBERT in Vietnamese) for QA.

## 5. Related Directions and Distinction from Other SPBERT Variants

SPBERTQA in the span selection context is distinct from the SPBERT model trained on SPARQL for knowledge-graph QA [2106.09997], which addresses answer generation and SPARQL query construction using a different pre-training regime. However, SPBERTQA (span selection) aligns more closely with classical reading comprehension objectives, while the BM25+SBERT-based SPBERTQA exemplifies dense retrieval and semantic ranking optimizations in low-resource cQA.

## 6. Applications and Domain Transfer

The span-selection paradigm is particularly potent for extractive question answering, multi-hop reasoning (HotpotQA), and settings with incomplete lexical overlap between query and source. The BM25+SBERT SPBERTQA pipeline is deployed for end-user health information seeking (Vietnamese QA) and provides a template for building domain-adapted cQA back-ends in other languages with limited annotated resources.

Future work for SPBERTQA systems includes integrated span highlighting, real-time deployment for conversational agents, and extension to slot filling and speech-based SLU domains. 

---
**References:**  
- "Span Selection Pre-training for Question Answering" [1909.04120]  
- "SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts" [2206.09600]  
- For contrast with pre-training on SPARQL syntax: "SPBERT: An Efficient Pre-training BERT on SPARQL Queries for Question Answering over Knowledge Graphs" [2106.09997]

Source: https://www.emergentmind.com/topics/spbertqa