BM25 Retrieval: Methods and Applications

• BM25 retrieval is a probabilistic bag-of-words model that calculates relevance using term frequency saturation, document length normalization, and inverse document frequency.
• It underpins sparse retrieval pipelines and has evolved with extensions like query-side weighting, semantic variants, and fusion with dense models.
• Its efficiency and scalability enable robust performance across domains such as web, biomedical, legal, and hybrid retrieval systems.

BM25 retrieval is a probabilistic bag-of-words ranking framework that assigns relevance scores to documents for a given query by combining term frequency saturation, document length normalization, and inverse document frequency. BM25 is a cornerstone of sparse retrieval pipelines leveraging inverted indexes, exhibiting high efficiency and robust effectiveness across diverse information retrieval tasks and corpora, including web-scale, biomedical, legal, and multilingual domains. Recent work has expanded and adapted BM25 to hybrid retrieval, semantic variants, query-side weighting, and advanced fusion schemes, increasingly integrating BM25 into retrieval-augmented generation (RAG), neural reranking, and hybrid dense-sparse systems.

1. Mathematical Foundations and Scoring Function

The standard BM25 scoring function for a query $Q$ and document $D$ is:

$$\mathrm{score}(D,Q) = \sum_{t \in Q} \log \left( \frac{N - n_t + 0.5}{n_t + 0.5} \right) \times \frac{f(t,D) (k_1 + 1)}{f(t, D) + k_1 \left[ 1 - b + b \frac{|D|}{\mathrm{avgDL}} \right]}$$

where:

• $N$ is the corpus size.
• $n_t$ is the number of documents containing $t$.
• $f(t,D)$ is the term frequency of $t$ in $D$.
• $|D|$ is the document length; $\mathrm{avgDL}$ is mean document length.
• $k_1$ controls TF saturation (0.9–2.2 typical).
• $b\in[0,1]$ controls degree of length normalization.

Variants may introduce a query term frequency or other tunings ($k_3$ in certain implementations). The “IDF” factor down-weights ubiquitous terms, while TF saturation and length normalization prevent scoring of long and verbose documents from dominating retrieval results (Ge et al., 2 Sep 2025, Abdallah et al., 27 Feb 2025, Pokrywka, 6 Oct 2024, Sarkar et al., 2017, 0911.5046).

2. Implementations, Frameworks, and Deployment

BM25 is implemented in key IR toolkits:

• Lucene/Anserini/Pyserini: BM25 is the default, with direct Java/Python API controls and command-line configurability (Ge et al., 2 Sep 2025, 0911.5046).
• Elasticsearch/Solr: BM25Similarity with customizable analyzers, stopword lists, stemming, and language-specific processing as exemplified in Polish and Silt’e (Pokrywka, 6 Oct 2024, Johar, 2020).
• fastbm25 for efficient passage retrieval on large-scale collections (Pokrywka, 6 Oct 2024).
• BM25 is the backbone of high-efficiency practical systems—e.g., biomedical QA over 24 million PubMed abstracts achieves stable 82ms query latency and ~1.9s end-to-end RAG responses (Stuhlmann et al., 12 May 2025).

API usage is typically trivial:

searcher = SimpleSearcher('path/to/index') searcher.set_bm25(k1=0.9, b=0.4) searcher.set_query_generator('bm25') # or 'bag' hits = searcher.search(query_text, k=100)

3. BM25 Variants: Query-Side Weighting, Fusions, and Semantic Extensions

Query-side BM25 was introduced for long, reasoning-intensive queries (e.g., RAG prompts, full-text ODQA, and QBE). It applies BM25’s saturation and length normalization to both query and document vectors:

$$w_q(t) = \frac{f(t,Q)(k_1+1)}{f(t,Q)+k_1(1-b+b|Q|/\mathrm{avgQL})} \times \mathrm{IDF}(t)$$

$$\mathrm{score}(D,Q) = \sum_{t \in Q} \mathrm{IDF}(t) \times w_q(t) \times w_d(t)$$

This approach yields robust nDCG@10 gains ($\sim$3.5%, BRIGHT) for queries 16–256 tokens, with recall@100 also increased (Ge et al., 2 Sep 2025, Abolghasemi et al., 2022). For traditional short queries (<16 tokens), classic bag-of-words suffices.

Semantic BM25 is produced by transformer cross-encoders, which recompute BM25’s components:

• Attention heads aggregate soft-TF, term saturation, and length normalization.
• Low-rank embedding matrices encode IDF.
• Late attention heads combine signals into a relevance score resembling classical BM25 but with semantic matching, especially beneficial for synonym/paraphrase queries (Lu et al., 7 Feb 2025).

Hybrid retrieval frameworks fuse BM25 with dense retrievers (BioBERT, MedCPT, DPR), using fixed or dynamically tuned weighting per query (e.g., DAT: Dynamic Alpha Tuning). DAT leverages an LLM to score the relative effectiveness of top-1 BM25 and dense-retriever results, adaptively setting $\alpha(q)$ for score fusion, yielding up to 6–7 point gains vs. static weighting (Hsu et al., 29 Mar 2025, Abdallah et al., 27 Feb 2025, Stuhlmann et al., 12 May 2025).

LexBoost uses dense nearest-neighbor graphs for robust fusion of BM25 document and neighbor scores:

$$\mathrm{score}_{LexBoost}(q,d) = \lambda\,S[d] + \frac{1-\lambda}{k}\sum_{i=1}^k S[d_i]$$

With $k=16$ neighbors and $\lambda\sim0.7$, MAP and recall@1000 improvement over pure BM25 are consistent (+7–17%) across TREC DL, BEIR, COVID (Kulkarni et al., 25 Aug 2024).

BM𝔛/BMX extends BM25 with entropy-weighted similarity and semantic augmentation via LLM-generated paraphrases. The full scoring formula introduces entropy and query-document similarity terms, further boosting nDCG@10 by 1–2 points in zero-shot and long-context settings (Li et al., 13 Aug 2024).

4. Evaluation, Empirical Performance, and Best Practices

BM25 remains highly competitive and often outperforms neural retrievers under lexical matching conditions:

• Open-domain QA (Natural Questions): Top-1 BM25 passage recall is 22.1%; dense retrievers (DPR) reach 48.7%, but hybrid pipelines achieve up to 53.4% (Abdallah et al., 27 Feb 2025).
• Document reranking (BEIR, TREC-DL): BM25 nDCG@10 of 43.4 (BEIR average), improved to >52.6 via hybrid reranking (Abdallah et al., 27 Feb 2025, Askari et al., 2023, Kulkarni et al., 25 Aug 2024).
• Biomedical QA: BM25 achieves 0.72 accuracy/recall, optimal at 50 candidate retrieval followed by MedCPT reranking (total 0.90) (Stuhlmann et al., 12 May 2025).
• Legal retrieval (COLIEE): BM25 scored second place, micro-F1 of 0.0937, well above median (Rosa et al., 2021).
• Multilingual and low-resource settings: BM25 with language-specific preprocessing yields strong recall/precision (Pokrywka, 6 Oct 2024, Johar, 2020, Sarkar et al., 2017).

Score fusion (e.g., weighted sum) frequently underperforms direct injection of BM25 into neural rerankers. Injecting the BM25 score as a token into BERT-based cross-encoders yields consistent and statistically significant improvements across MRR@10, nDCG@10, and MAP, outperforming naive fusion methods (Askari et al., 2023).

Key best practices:

• Tune $k_1$ and $b$ for each corpus/task; typical $k_1$~=~1.2, $b$~=~0.75, but longer queries or passages may require larger values.
• Use multilingual or language-specific analyzers, stopword lists, and stemmers per language.
• Passage splitting and indexing granularity impact accuracy, especially in QA and legal retrieval (Rosa et al., 2021).
• For hybrid pipelines, retrieve with BM25, then rerank top-K with cross-encoders or dense embeddings for optimal accuracy/efficiency (Stuhlmann et al., 12 May 2025, Hsu et al., 29 Mar 2025).

5. Computational Efficiency, Scalability, and Optimization

BM25 is characterized by highly efficient term-based retrieval via inverted indexes, yielding sub-second latency at web scale:

• PubMed indexing (24M abstracts): BM25 indexing in Elasticsearch completes in ~156 min on 16-core CPU; query latency averages 82ms (Stuhlmann et al., 12 May 2025).
• BM25-guided index traversal (sparse-dense retrieval, MaxScore/DAAT): Two-level BM25/learned weight mixture controls allow 4–8× speedups over pure learned models with negligible loss in recall (Qiao et al., 2023).
• BM25 remains easily scalable for large collections, supporting sharding and caching; typical dense retrieval systems incur much higher compute and memory cost (Stuhlmann et al., 12 May 2025, Abdallah et al., 27 Feb 2025).

6. Limitations, Extensions, and Future Directions

BM25’s main limitations include insensitivity to synonymy/polysemy and poor performance in strictly semantic matching or generative contexts. Dense retrievers (BioBERT, MedCPT, DPR) outperform BM25 in semantic QA, though BM25 excels in contexts requiring exact lexical overlap. Simple dense–BM25 hybrids, semantic BM25 variants, entropy-weighted fusion (BMX), graph-based neighbor fusion (LexBoost), and LLM-tailored data augmentation offer empirically significant improvements for semantic coverage, long-context retrieval, multilingual adaptation, and hybrid systems (Li et al., 13 Aug 2024, Hsu et al., 29 Mar 2025, Lu et al., 7 Feb 2025, Kulkarni et al., 25 Aug 2024).

Recent integration efforts have focused on:

• Dynamic query-side saturation and normalization for ultra-long prompts (BRIGHT, RAG).
• Efficient hybrid score fusion, dynamically tuned by LLM effectiveness scoring.
• Plug-and-play semantic fusion and paraphrase augmentation for bridging sparse-dense performance gaps.

BM25’s modularity, interpretability, and robust effectiveness establish it both as a baseline for benchmarking and as a key building block for advanced, hybrid, and domain-specific retrieval pipelines.

7. Practical Guidelines and Recommendations

Application Scenario	BM25 variant / setting	Empirical Notes / Recommendations
Short keyword queries (<16 tokens)	Standard BM25 (BoW query)	Defaults (k₁ ≈ 1.2, b ≈ 0.75) suffices (Ge et al., 2 Sep 2025)
Long/Reasoning-intensive queries	Query-side BM25 (BM25Q)	Normalize query side; clear gains for 16–256 tokens (Ge et al., 2 Sep 2025, Abolghasemi et al., 2022)
Passage retrieval for QA	BM25 + cross-encoder reranking	Retrieve top-50 BM25, rerank with MedCPT (Stuhlmann et al., 12 May 2025)
Fusion with dense or sparse retrievers	Weighted sum (DAT), LexBoost	Fusion parameters (λ≈0.7–0.8), dynamic α tuning preferred (Hsu et al., 29 Mar 2025, Kulkarni et al., 25 Aug 2024)
Neural rerankers, cross-encoder	BM25 score injection as token	Statistically significant improvements; robust to query type (Askari et al., 2023)
Multilingual retrieval	Language-specific preprocessing	Lowercasing, custom stemmers, stopword lists recommended (Pokrywka, 6 Oct 2024, Johar, 2020, Sarkar et al., 2017)

In summary, BM25 remains foundational for lexical retrieval, with ongoing innovations in query-side weighting, semantic fusion, scalability, and hybrid integration ensuring continued relevance in knowledge-intensive information retrieval.