BEIR SciFact Benchmark

• BEIR SciFact Benchmark is a domain-specific evaluation suite that assesses retrieval models by matching scientific claims to supporting or refuting evidence in biomedical texts.
• It employs a zero-shot evaluation paradigm using diverse retrieval methods including sparse, dense, and re-ranking pipelines, supported by contrastive pretraining techniques.
• It provides rigorous performance metrics like nDCG@10 while exploring compression strategies such as int8 quantization to optimize scalability and resource efficiency.

BEIR SciFact Benchmark is a domain-specific evaluation suite within the BEIR (Benchmarking-IR) framework for information retrieval, designed to assess the capability of retrieval models to match scientific claims to supporting or refuting evidence in the biomedical and scientific literature. The benchmark serves as a rigorous testbed for model robustness, OOD (out-of-distribution) generalization, and compression strategies in dense and sparse retrieval settings, focusing on claim verification with an emphasis on resolving domain-specific terminology and semantic nuance.

1. Benchmark Structure and Dataset Definition

SciFact, originally introduced by Wadden et al. (EMNLP 2020), is an expert-annotated corpus for scientific claim verification. The BEIR SciFact benchmark adopts the 300-claim publicly released "dev" split as its test set, with a retrieval corpus of 5,183 PubMed abstracts (mean abstract length ≈213.6 words, mean claim length ≈12.4 words) (Wadden et al., 2020, Thakur et al., 2021, Kamalloo et al., 2023, Pati, 17 Nov 2025). Each claim may be supported or refuted by one or more relevant abstracts, with binary relevance labeling (SUPPORTS or REFUTES).

Task Formulation:

Given a short scientific claim as a query, the model must return a ranked list of abstracts from the corpus predicted to provide supporting or refuting evidence. In BEIR, retrieval is evaluated strictly; classification and rationale selection are not required.

Relevance Annotation:

• Claims–abstract pairs are labeled {Supports, Refutes, NoInfo}, but BEIR treats Supports and Refutes as equally "relevant" for binary IR scoring (Thakur et al., 2021, Kamalloo et al., 2023).

Corpus Characteristics:

• Domain: biomedical/scientific, high technicality, frequent synonymy and abbreviation.
• Retrieval challenge: high lexical overlap, but often requires resolving synonyms and paraphrases; fine-grained semantic bridging is critical (Wang et al., 2022, Thakur et al., 2021).

2. Experimental Protocol and Retrieval Pipelines

Zero-Shot Paradigm:

All BEIR benchmark models are evaluated under zero-shot conditions; no supervised fine-tuning is performed on SciFact-specific labels (Thakur et al., 2021, Kamalloo et al., 2023). Models are trained on general-domain retrieval data, the most common being MS MARCO, or, in the domain-adapted case (GenQ), synthetic queries targeting the SciFact domain.

Preprocessing and Indexing:

• Claims and abstracts are tokenized, lowercased, and indexed.
• For BM25 and sparse models: Lucene inverted index (Anserini/Pyserini), k1=0.9, b=0.4.
• For dense retrieval: abstracts embedded (e.g., via BERT-based encoder) into fixed-length vectors; stored in FAISS indices (Kamalloo et al., 2023, Wang et al., 2022).

Retrieval Pipelines:

• BM25: term-based retrieval over concatenated abstracts.
• Sparse Retrieval: DeepCT, SPARTA, docT5query (with/without expansion, BERT-derived).
• Dense Bi-Encoders: DPR, ANCE, TAS-B, GenQ, Contriever, E5 (Wang et al., 2022).
• Late-Interaction: ColBERT (token-level maxsim aggregation).
• Re-ranker: BM25 top-k + cross-encoder re-ranking.
• E5 Embeddings: "query:" and "passage:" prefixing, average-pool BERT-base hidden states, cosine similarity in 768-dim space (Wang et al., 2022).

3. Evaluation Metrics and Scoring

The principal metric is nDCG@10 (normalized Discounted Cumulative Gain at rank 10), which quantifies how well the top-10 retrieved abstracts match gold relevance. Supporting metrics include MAP (Mean Average Precision), MRR (Mean Reciprocal Rank), Recall@100, and Precision@k. All metrics are computed in accordance with TREC standards (Thakur et al., 2021, Kamalloo et al., 2023, Pati, 17 Nov 2025):

• $$\mathrm{nDCG@}k = \frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k}$$ with

$$\mathrm{DCG@}k(q)=\sum_{i=1}^{k}\frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)}$$

• $$\mathrm{MAP} = \frac{1}{|Q|} \sum_{q\in Q} \mathrm{AP}(q)$$
• $$\mathrm{MRR} = \frac{1}{|Q|} \sum_{q\in Q} \frac{1}{\mathrm{rank}_q}$$

4. Model Performance and Comparative Results

The BEIR SciFact benchmark highlights persistent strengths and weaknesses across retrieval paradigms. Lexical, sparse, dense, mixed, and re-ranking models exhibit different trade-offs between domain robustness, computational efficiency, and semantic recall (Thakur et al., 2021, Wang et al., 2022, Kamalloo et al., 2023).

Representative nDCG@10 results (zero-shot):

Model	nDCG@10
BM25	0.665
DeepCT	0.630
docT5query	0.675
ColBERT	0.671
TAS-B	0.643
Contriever	0.677
SPLADE	0.699
uniCOIL	0.686
BM25+CE re-ranker	0.688
E5-PT_base	0.737
E5-PT_large	0.723

Findings:

• BM25 is a strong baseline; document expansion (docT5query) and BM25+cross-encoder re-ranking are competitive or better.
• Unsupervised E5-PT_base, using contrastive pre-training on CCPairs, decisively outperforms all other models (+7.2 nDCG points above BM25).
• Modern sparse models (SPLADE, uniCOIL) surpass dense retrievers trained solely on MS MARCO.
• Dense retrievers (DPR, ANCE) underperform dramatically unless re-trained or adapted for the domain.
• ColBERT (late-interaction) and BM25+CE provide high nDCG but with higher latency and memory cost (Wang et al., 2022, Thakur et al., 2021, Kamalloo et al., 2023).

5. Architectural and Methodological Advances

Contrastive Pretraining in E5:

E5’s superiority on SciFact in zero-shot stems from large-scale weakly supervised contrastive pretraining on 270M text pairs ("CCPairs"), including scientific citation metadata, CommunityQA, and Common Crawl data. Its InfoNCE loss objective is:

$$L_{cont} = -\frac{1}{n} \sum_{i=1}^n \log\left( \frac{\exp[s(q_i, p_i)]}{\exp[s(q_i, p_i)] + \sum_j \exp[s(q_i, p_j^-)]} \right)$$

where $s(q, p) = \cos(E_q, E_p)/\tau$ with temperature $\tau=0.01$. In-batch negatives ($\leq$32K) sharply penalize non-relevant matches, making the embedding space highly discriminative.

Semantic Bridging:

E5’s use of scientific citation pairs confers strong paraphrase and terminology-bridging capabilities, crucial for SciFact’s abstraction and synonymy (Wang et al., 2022). The knowledge diversity and lack of synthetic cropping in CCPairs robustness on biomedical text.

Domain Adaptation and Expansion:

Methods like docT5query (document expansion) and GenQ (synthetic in-domain queries) show moderate improvement by narrowing the lexical/semantic divergence for domain-specific terminology, but do not match the effect of E5’s contrastive approach.

6. Compression Strategies for Dense Retrieval

With the rise of high-dimensional dense representations, storage and efficiency constraints are acute in IR deployment. On BEIR SciFact, two principal compression strategies have been evaluated (Pati, 17 Nov 2025):

Method	Bytes/Vector	Compression	nDCG@10 Loss (vs. float32)
float32 baseline	1536	1×	0.00000
float16	768	2×	0.00018
int8 quantization	384	4×	0.00178
AE-96 (autoencoder)	384	4×	0.06466
binary	48	32×	0.46621

Conclusions:

• int8 post-training quantization attains 4× reduction with negligible (≈0.18%) nDCG@10 loss.
• AE-96 (autoencoder to 96 dims at 4×) suffers >6% nDCG@10 drop.
• Binary quantization is catastrophic (>40% drop).
• Precision reduction (int8) outperforms dimensionality reduction for moderate compression; float16 is effectively lossless at 2×.

Implication: For SciFact-sized corpora and cosine-similarity search, scalar int8 quantization is practically optimal up to 4× compression.

7. Scientific and Practical Implications

The BEIR SciFact benchmark has clarified methodological requirements for high-stakes, domain-specific retrieval:

• Semantic retrieval, especially via contrastive pretraining on diverse and scientific data, is essential to surpass lexical methods like BM25.
• Sparse and document-expansion techniques close part of the gap but are sensitive to domain shift and require careful tuning.
• Dense models, with proper architectural and data-centric advances (E5), provide both high retrieval quality and superior OOD robustness—particularly salient in scientific literature with specialized terminology.
• Compression á la int8 quantization or hybrid pipelines significantly reduce resource costs while maintaining retrieval accuracy, providing a feasible deployment path for dense retrieval in memory-limited settings.

Continuous benchmarking via BEIR’s official leaderboard framework ensures replicability and standardized comparison across retrieval architectures and training protocols (Kamalloo et al., 2023). The SciFact subset thus acts as a keystone for evaluating retrieval models’ semantic generalization and deployability in scientific domains.