---
title: BM25 Ranking Function Overview
url: https://www.emergentmind.com/topics/bm25-ranking-function
type: topic
---

# BM25 Ranking Function Overview

BM25 is a probabilistically motivated, term-weighting-based document ranking function that occupies a central role in modern information retrieval (IR) systems. It operationalizes relevance as a function of exact query–document term overlap, subject to non-linear term-frequency scaling, document-length normalization, and global rarity of terms. BM25 continues to set a strong baseline for both classical and neural IR architectures, exhibiting robust retrieval performance across a wide range of domains and datasets [2105.05686].

## 1. Mathematical Formulation and Core Mechanics

BM25 assigns to each document–query pair a scalar relevance score via a weighted sum over all query terms present in the document. The canonical formula is:
\[
\mathrm{BM25}(D,Q) = \sum_{i=1}^{n} \mathrm{idf}(q_i) \frac{\,\mathrm{tf}(q_i,D)\,(k_{1}+1)}{\mathrm{tf}(q_i,D) + k_{1}\left(1 - b + b\,\frac{|D|}{\mathit{avgdl}}\right)}
\]
where:
- $Q = \{q_1, ..., q_n\}$: query terms,
- $D$: candidate document,
- $\mathrm{tf}(q_i,D)$: frequency of $q_i$ in $D$,
- $|D|$: document length (tokens),
- $\mathit{avgdl}$: average document length in the collection,
- $k_1$: term-frequency scaling parameter (controls TF saturation),
- $b$: document-length normalization (interpolates between no normalization, $b=0$, and strict normalization, $b=1$),
- $\mathrm{idf}(q_i)$: inverse document frequency:
  \[
  \mathrm{idf}(q_i) = \log\left(\frac{N - n(q_i) + 0.5}{n(q_i) + 0.5}\right)
  \]
  with $N$ the total number of documents, $n(q_i)$ the document frequency of $q_i$ [1608.01972][2105.05686][2301.09728].

Typically, $k_1 \in [1.2, 2.0]$ and $b \approx 0.75$ are used as robust defaults, but values such as $k_1=1.9$, $b=1.0$ are validated for specific domains (e.g., biomedical abstracts) [1608.01972].

## 2. Parameterization, Model Behavior, and Example

The parameters $k_1$ and $b$ fundamentally control how BM25 interpolates between raw count-based scoring and more nuanced "pivoted normalization" accounting for within-corpus variations in verbosity and length [2105.05686][2301.09728]:

- **$k_1$ (TF scaling):** Larger $k_1$ increases the linearity of term frequency scaling, giving more weight to repeated terms; smaller $k_1$ saturates faster, reducing the marginal gain of repeated term matches.
- **$b$ (length normalization):** $b=0$ ignores length; $b=1$ fully normalizes by the relative length with respect to $\mathit{avgdl}$. Longer-than-average documents are penalized when $b>0$.

For example, with $N=10^6$, $\mathit{avgdl}=1000$, $|D|=1500$, $k_1=1.5$, $b=0.75$, $\mathrm{tf}(q_i,D)=3$, $\mathrm{idf}(q_i)=2.0$, the score contribution is:
\[
\text{length factor} = 1 - 0.75 + 0.75 \cdot (1500/1000) = 1.375
\]
\[
\text{denominator} = 3 + 1.5 \times 1.375 = 5.0625
\]
\[
\text{numerator} = 3 \times 2.5 = 7.5
\]
\[
\text{score} = 2.0 \times (7.5 / 5.0625) \approx 2.96
\]
[2301.09728].

## 3. Variants and Extensions: Multi-Field, Proximity, Query-Dependent Normalization

BM25 forms the core of numerous adaptive scoring schemes:

- **BM25F (Multi-field):**
  Aggregates term frequencies across structured fields (title, abstract, body) using per-field weights and normalization:
  \[
  \mathrm{BM25F}(D,Q) = \sum_{t\in Q} \mathrm{IDF}(t)\;\frac{W(t,D)}{k_1 + W(t,D)}
  \]
  where $W(t,D)$ consists of field-specific boosted and normalized counts [1709.03260].

- **Proximity-based BM25:**
  Incorporates proximity heuristics via "Expanded Span" methods, extracting spans of near-occurrence query terms and replacing term frequency by a relevance-weighted sum over such spans. This approach rewards documents where query terms appear in close textual proximity, even across different fields [1709.03260].

- **Query-Dependent Length Normalization:**
  Standard BM25 normalizes only by ${|D|}/{\mathit{avgdl}}$, ignoring query length. Proximity-matching generalizations replace this with a two-variable factor $h(|d|,|q|)$, designed to down-weight verbosity mismatches and up-weight document–query pairs with similar lengths:
  \[
  h(x, y) = 
    \begin{cases}
      1 + \frac{b_1-1}{1 + \exp[B_1 (x - c\,y)]} & x < y \\
      1 & x = y \\
      1 + \frac{b_2-1}{1 + \exp[-B_2 (x - (1+c)\,y)]} & x > y
    \end{cases}
  \]
  This modification can yield substantial gains in applications where matching verbosity is correlated with relevance (e.g., penpal recommendation: 52% MRR improvement over BM25) [1701.01417].

## 4. Empirical Benchmarks and Applications

BM25 serves as a robust baseline and strong IR performer across diverse tasks and domains:

- **Legal Case Retrieval:** Achieved F1=0.0937 (second place) on COLIEE 2021 legal case retrieval with default Pyserini settings ($k_1=1.2, b=0.75$) and minimal preprocessing (default tokenization, stopword removal, stemming, and sliding-window document segmentation) [2105.05686].
- **Biomedical Data:** On PubMed, $k_1=1.9, b=1.0$ provided state-of-the-art performance for short queries over large collections (162,259 PubMed abstracts; mean average precision 0.2463–0.3136 on TREC Genomics). In user log data (28K PubMed queries; 27M docs), BM25 outperformed baseline IR models and, when combined with semantic matching in LambdaMART, increased NDCG@20 by 23% [1608.01972].
- **Zero-shot, Passage, and Semantic IR Benchmarks:** Remains highly competitive in MSMARCO and TREC-DL, where it is often used as the initial candidate generator for neural rerankers [2301.09728][2502.04645].

## 5. Integration with Neural and Learning-to-Rank Models

BM25 forms the foundation of multi-stage pipelines, where its exact-match bias and interpretability are leveraged alongside modern neural models:

- **Hybrid Feature Fusion:** Combining BM25 scores with semantic similarity (e.g., embedding-based methods) in learning-to-rank frameworks (e.g., LambdaMART) yields additive gains. On PubMed, such combinations increased NDCG@10 by up to 25% [1608.01972].
- **Cross-Encoder Signals:** Injecting normalized BM25 scores as integer tokens into BERT-based cross-encoder rerankers outperforms linear or non-linear score interpolation, with injected tokens shown (via feature attribution) to be among the model's most influential ranking features [2301.09728].
- **Semantic BM25 in Transformers:** Cross-encoders (e.g., MiniLM) intrinsically learn a "semantic BM25" circuit, with attention heads approximating soft term frequency (with built-in saturation and length normalization) and embedding singular vectors encoding a corpus IDF analogue. This learned variant not only recovers the effects of classical BM25 but is extended semantically—registering paraphrase and synonymy as soft TF [2502.04645].

## 6. Practical Implementation and Tuning

BM25 is widely implemented in IR toolkits (Lucene, Pyserini/Anserini) with defaults $(k_1=1.2, b=0.75)$ and is robust under minimal preprocessing (tokenization, stemming, stopword removal). Application-specific adjustments may include:

- **Passage/Window Segmentation:** Improves effectiveness in long-document domains by aligning retrieval units with local contexts [2105.05686].
- **Field Importance and Proximity Tuning:** Boosted field weights, proximity span sizes, and length-normalization parameters are best fit via grid search or, increasingly, as part of a differentiable learning-to-rank optimization [1709.03260].
- **Parameter Sensitivities:** BM25 is generally robust to moderate variation in $k_1$ and $b$, but domain adaptation (e.g., biomedical vs. legal) can benefit from targeted tuning [1608.01972][2105.05686].

## 7. Limitations, Enhancements, and Future Research

BM25's bag-of-words architecture makes it reliant on exact (or near-exact) term overlap, with limitations in capturing semantic similarity where lexicon does not match directly [1608.01972]. Adaptive extensions address some weaknesses:

- **Semantic Enhancements:** Neural models extend BM25 via learned soft-counts and embedding-based IDF proxies, providing semantic generalization beyond token identity [2502.04645].
- **Proximity and Field-Weighting:** Further development of proximity features (spans), query-dependent normalization, and learnable parameters offers domain- and application-specific gains [1709.03260][1701.01417].
- **Hybrid and Interpretable Models:** Recent research highlights the natural emergence of BM25 analogues in neural rankers, raising prospects for hybrid designs with explicit, transparent control over $(k_1, b)$ and interpretable IDF weighting [2502.04645].

In empirical IR practice and methodological research, BM25 persists as the paradigmatic lexical scoring approach—benchmarking systems from legal to biomedical retrieval, and providing a mechanistically transparent scaffold upon which both efficient sparse and state-of-the-art neural systems are built.

Source: https://www.emergentmind.com/topics/bm25-ranking-function