BM25-Based Filtering Overview

• BM25-based filtering is a probabilistic retrieval model that uses term frequency, document length normalization, and IDF weighting to estimate document relevance.
• It efficiently narrows down massive candidate sets, serving as a vital first-stage filter before applying more computationally expensive re-ranking methods.
• Its adaptability across domains, including text, code, and vision, is enhanced by domain-specific preprocessing, tuning, and hybrid integration with neural models.

BM25-based filtering is a foundational technique in information retrieval, serving as an efficient first-stage method for filtering massive candidate sets in both text and non-text domains. At its core, BM25 is a probabilistic retrieval model that scores document relevance for a (multi-term) query by combining lexical term matching, document length normalization, and global inverse document frequency (IDF) weighting. BM25-based filtering is widely adopted as the initial stage in modern multi-stage retrieval pipelines, where it quickly reduces search space before more expensive re-ranking, e.g., using cross-encoders or semantic ranking models. The following sections outline the mathematical underpinnings of BM25, its variants and domain adaptations, implementation details, empirical performance as a filter, integration into hybrid and learning-based systems, and recent extensions across text, code, and vision.

1. Mathematical Formulation and Theoretical Basis

BM25 computes a query-document score by summing over query terms, weighting by term importance and normalizing term frequency for document length. For a query $Q = \{q_1,\ldots,q_n\}$ and document $D$ (or passage/image), the BM25 score is given by:

$$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$

• $f(q_i, D)$: term frequency of $q_i$ in $D$
• $|D|$: document length (token or word count)
• $\mathrm{avgdl}$: average document length in the corpus
• $k_1$: term-frequency saturation parameter (typ. $[1.2, 2.0]$)
• $D$0: length normalization
• $D$1, with $D$2 the corpus size, $D$3 the number of documents containing $D$4

Interpretively, the model captures the diminishing returns of repeated term occurrences as controlled by $D$5, and penalizes or boosts long and short documents via $D$6. The IDF suppresses common terms and emphasizes rare, discriminative ones. The same core formulation extends to image retrieval (visual terms) (Han et al., 6 Mar 2026), code indexing (Radha et al., 18 May 2026), biomedical concept normalization (Torre, 1 May 2025), and legal case retrieval (Rosa et al., 2021).

Several variants have been advanced to address specific issues. In retrieval with long queries (e.g., LLM-generated prompts), query-side BM25 applies a symmetric saturation and normalization to the query terms, mitigating the overweighting of repeated generic tokens (Ge et al., 2 Sep 2025). In code search, a $D$7-logarithm modification to IDF amplifies rare identifier discrimination where tokenizer design is suboptimal (Radha et al., 18 May 2026).

2. Implementation, Preprocessing, and Indexing Considerations

BM25-based filtering can be implemented in various frameworks, with Apache Lucene/Anserini family providing reference implementations (0911.5046, Ge et al., 2 Sep 2025). Implementations maintain inverted indexes over the corpus vocabulary, precompute or dynamically compute per-term document frequencies, and aggregate per-document statistics for efficient scoring.

Preprocessing is task and language-specific:

• Text: Tokenization, lower-casing, stemming/lemmatization, and stop-word removal are standard (Pokrywka, 2024, Faessler et al., 2020, Torre, 1 May 2025).
• Image: Learned sparse auto-encoder encodes patch features into a visual vocabulary, supporting a Zipfian IDF distribution (Han et al., 6 Mar 2026).
• Code: Choice of identifier- and sub-tokenization is critical. When frozen infrastructure precludes such choices, $D$8-IDF transforms can compensate (Radha et al., 18 May 2026).

Parameter selection and tuning: Grid search, Bayesian optimization, or algorithmic configuration (e.g. SMAC) are applied to tune $D$9, $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$0, and any field/boost parameters, often with evaluation on held-out queries and metrics like nDCG or MRR (Torre, 1 May 2025, Faessler et al., 2020).

Efficiency and indexing: Sparse storage and eager score computation are keys for scale. For large text or vision corpora, precomputing term-document scores and storing only nonzero entries (CSC/COO matrices) enables $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$1–$$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$2 query-per-second speedup over standard implementations, without memory blowup (Lù, 2024, Han et al., 6 Mar 2026).

3. BM25 as First-Stage Filtering in Retrieval Pipelines

BM25 is established as the canonical first-stage candidate generator in modern IR pipelines. Its role is to rapidly reduce candidate sets from millions to thousands, maintaining high recall while enabling computationally expensive re-ranking only on the filtered set (Pokrywka, 2024, Kim et al., 2016, Askari et al., 2023, Torre, 1 May 2025).

Examples:

• In passage retrieval for Polish texts, BM25 using fastBM25 reduces search space to 3,000–1,500 candidates, enabling tractable cross-encoder re-ranking (Pokrywka, 2024).
• Legal case retrieval leverages BM25 over segmented case texts, then applies threshold and max-aggregation rules to produce supportable candidate pools (Rosa et al., 2021).
• Biomedical search, code retrieval, and precision medicine pipelines follow similar templates: BM25 filters, hybridizes with semantics (embeddings), or passes to LTR/neural re-rankers (Torre, 1 May 2025, Radha et al., 18 May 2026, Kim et al., 2016, Faessler et al., 2020).

Performance metrics (nDCG@10, MRR, F1) show that BM25 alone provides robust baselines, often rivaling or exceeding neural approaches in domain-specific, high-lexical-overlap settings (e.g., legal text). However, neural or semantic re-ranking consistently improves precision, especially in open-domain, trivia, or reasoning tasks (Pokrywka, 2024, Askari et al., 2023, Lu et al., 7 Feb 2025).

4. Hybrid, Neural, and Learnable Extensions

Contemporary retrieval research systematically hybridizes BM25 with semantic or neural models:

• Score injection: BM25 scores are injected as special tokens in cross-encoder rerankers, improving both semantic and exact-match accuracy without incurring pipeline complexity or training burden (Askari et al., 2023).
• Two-stage ranking: BM25 filters, then models such as LambdaMART combine lexical and semantic similarity measures to yield final rankings; BM25 plus a semantic “one-way Word Mover’s Distance” feature achieves up to 25% better NDCG on biomedical data (Kim et al., 2016).
• Neural augmentation and reweighting: Differentiable expansions and per-query re-weighted term importance vectors (learned end-to-end) enhance recall and transfer across datasets while retaining classic BM25 runtime and memory (Chen et al., 2023).
• Neural model interpretability: Cross-encoders trained for semantic ranking are shown to “rediscover” BM25’s soft term-frequency and IDF logic, realized in distributed attention and embedding layers, supporting tractable model editing and transparency (Lu et al., 7 Feb 2025).
• Hybrid retrieval in practice: Combining normalized BM25 and embedding similarities (with optimized linear weights) leverages their complementary precision and recall, as empirically validated in clinical unit harmonization and PubMed search (Torre, 1 May 2025, Kim et al., 2016).

5. Variants, Domain Adaptations, and Cross-modal Retrieval

BM25’s extensibility supports domain- and modality-specific adaptations:

• Query-side BM25: For long, LLM-generated prompts where repeated or generic tokens dominate, applying TF-saturation and normalization to the query vector itself (mirroring document-side BM25) reduces query noise, yielding $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$3 absolute nDCG@10 in the BRIGHT long-query benchmark (Ge et al., 2 Sep 2025).
• Code retrieval improvements: On code corpora with high hapax density from identifier tokens, adapting the IDF via a $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$4-log transform nearly doubles NDCG@10 ($$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$5, $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$6 relative) under frozen, generic tokenization (Radha et al., 18 May 2026).
• BM25 for vision: In sparse vision models (BM25-V), auto-encoded visual words attain empirical Zipfian frequency distributions, justifying BM25’s IDF for candidate filtering before dense reranking. This gives first-stage recall $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$7 and total retrieval within $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$8 accuracy loss vs. full dense matching, with the added benefit of interpretable retrieval provenance via high-IDF visual words (Han et al., 6 Mar 2026).
• Multi-field/structured retrieval: BM25F generalizes to structured documents using field-specific boosts, per-field length normalization, and composite score aggregation, yielding robust baselines on TREC and comparable platforms (0911.5046, Torre, 1 May 2025).
• Stop-word curation: In biomedical and legal corpora, domain-specific stop-word lists yield double-digit relative gains in baseline performance, and are systematically beneficial when used at both indexing and querying stages (Faessler et al., 2020, Pokrywka, 2024).

6. Empirical Results, Performance, and Best Practices

Across diverse public and industry datasets, BM25-based filtering remains a highly optimized and interpretable baseline:

• Passage retrieval: In Polish passage retrieval (Poleval 2023), BM25 alone gives NDCG@10=42.55; adding cross-encoder reranking raises this to 69.36, but BM25 alone wins on legal domains with high lexical overlap (Pokrywka, 2024).
• Legal retrieval: On COLIEE 2021, BM25 yields F1=0.0937 (second place) with only simple segmentation and parameter tuning (Rosa et al., 2021).
• Hybrid IR: Unit harmonization in clinical data yields MRR=0.8833 for BM25+embeddings, with the transformer reranker elevating MRR to 0.9833, reflecting the cumulative gain of staged filtering (Torre, 1 May 2025).
• Precision medicine search: Ablation shows that stop-word filtering can yield $$\mathrm{score}(D, Q) = \sum_{i=1}^n \mathrm{IDF}(q_i) \cdot \frac{f(q_i, D)\,(k_1 + 1)}{f(q_i, D) + k_1 \bigl(1 - b + b \frac{|D|}{\mathrm{avgdl}}\bigr)}$$9 infNDCG if omitted and that $f(q_i, D)$0, $f(q_i, D)$1 should be tuned per corpus (e.g., $f(q_i, D)$2 for ClinicalTrials.gov vs. $f(q_i, D)$3 for PubMed) (Faessler et al., 2020).
• Indexing and runtime: BM25S achieves $f(q_i, D)$4–$f(q_i, D)$5 speedups over Python and up to $f(q_i, D)$6 over Java backends, exploiting eager sparse matrix scoring (Lù, 2024).
• Vision retrieval: BM25-V achieves $f(q_i, D)$7 loss vs. dense retrieval at fivefold lower query latency and with full interpretability (Han et al., 6 Mar 2026).

Best practice recommendations:

• Always match preprocessing to language and domain; for code and biomedical settings, customize tokenization and stop-lists.
• Use BM25 with $f(q_i, D)$8 and $f(q_i, D)$9 tuned for the specific corpus/field; field-level boosting and query-side normalization may be necessary for structured or long-prompt queries (Ge et al., 2 Sep 2025, 0911.5046, Faessler et al., 2020).
• For large-scale IR, integrate BM25-based filtering as a pre-filter for neural, semantic, or learning-to-rank rerankers.
• Hybridize lexical (BM25) and semantic (embedding, cross-encoder) signals via proper normalization and late fusion, or by token-level injection (Kim et al., 2016, Askari et al., 2023).
• Report BM25-only baselines for transparency and to ensure downstream model performance can be properly attributed.

7. Limitations, Challenges, and Future Directions

While BM25-based filtering remains robust and efficient, several limitations and ongoing challenges persist:

• Vocabulary mismatch: Absence of explicit query terms in relevant documents (semantic gap) cannot be bridged by lexical models alone, motivating hybrid or neural augmentations (Kim et al., 2016, Chen et al., 2023).
• Tokenization rigidity: In code and non-English domains, frozen infrastructure or unguided analyzers necessitate adaptive weighting transformations (e.g., $q_i$0-log IDF adjusters) to recover missing discriminative capacity (Radha et al., 18 May 2026).
• Long query normalization: In large-prompt (RAG, LLM) workflows, the classic bag-of-words query vector overweights generic or repeated tokens; integrated query-side normalization resolves much of the effectiveness degradation (Ge et al., 2 Sep 2025).
• Structural metadata: In medical and legal informatics, handling missing, noisy, or field-biased documents requires attention to field boosts, query expansion, and fallback strategies (Torre, 1 May 2025, Faessler et al., 2020).
• Index update cost: For extremely large and dynamic corpora, index-time parameter locking and memory constraints can hinder agility; methods exploiting sparse matrix structure and streaming-friendly score computation are expanding BM25’s domain (Lù, 2024).

Ongoing research is focused on further improving the interpretability, efficiency, and adaptability of BM25-style filtering in hybrid symbolic–neural systems, cross-modal retrieval (vision, code), and evolving applications such as retrieval-augmented generation and complex interactive search. The enduring competitiveness of BM25 as a domain-agnostic, interpretable, and low-overhead filter remains foundational across IR and related fields (Pokrywka, 2024, Lu et al., 7 Feb 2025, Chen et al., 2023, Han et al., 6 Mar 2026, Torre, 1 May 2025, Ge et al., 2 Sep 2025, Lù, 2024, Radha et al., 18 May 2026, 0911.5046, Rosa et al., 2021, Kim et al., 2016, Faessler et al., 2020).