---
title: SPLADE-doc Variant in Sparse Retrieval
url: https://www.emergentmind.com/topics/splade-doc-variant
type: topic
---

# SPLADE-doc Variant in Sparse Retrieval

SPLADE-doc is a learned sparse retrieval (LSR) model variant that eliminates the neural query encoder and focuses entirely on pre-computing document-side sparse representations. Designed to leverage inverted index infrastructure, SPLADE-doc produces term-weighted, log-saturated, and aggressively sparse vectors for documents, enabling high-throughput and low-latency retrieval comparable to classical bag-of-words models. Several key advances—including precise sparsity control, novel regularization techniques (notably DF-FLOPS), and representation thresholding—have extended its efficiency and practicality for large-scale, real-world search deployments.

## 1. SPLADE-doc Architectural Principles

SPLADE-doc inherits its core architecture from SPLADE, which couples a Transformer-based encoder (such as BERT or DistilBERT) with a masked language modeling (MLM)–style "expansion head." For each document $d$, the encoder computes hidden vectors $h_i$ for each token position, then projects them into vocabulary space via:
\[
w_{ij} = \text{Transform}(h_i)^\top E_j + b_j
\]
where $E_j$ is the embedding for term $j$ and $b_j$ is a learned bias. To promote sparsity, document term weights are aggregated as:
\[
w_j^d = \max_{i \in d} \log(1 + \text{ReLU}(w_{ij}))
\]
Unlike the full SPLADE, SPLADE-doc omits any query-side encoding: at retrieval, the query is a simple bag-of-tokens, each assigned a unit weight, and document scores are summed over the corresponding precomputed $w_j^d$ [2107.05720, 2109.10086, 2207.03834].

## 2. FLOPS and DF-FLOPS Regularization

Term-level sparsity is induced through the FLOPS regularizer, designed to minimize the average activation per term over the batch:
\[
\ell_{\mathrm{FLOPS}} = \sum_{t \in V} \left( \frac{1}{N} \sum_{i=1}^N r_{i,t} \right)^2
\]
where $r_{i,t}$ is the weight for term $t$ in document $i$. However, standard FLOPS only controls "per-document" sparsity and does not prevent certain terms (such as stopwords) from accruing high document frequency (DF), resulting in long posting lists and increased retrieval latency.

DF-FLOPS addresses this by introducing a DF-dependent penalty:
\[
w_t = \text{activ}\left( \frac{DF_t}{|C|} ; \alpha, \beta \right)
\]
\[
\ell_{\mathrm{DF-FLOPS}} = \sum_{t \in V} \left( w_t \cdot \frac{1}{N} \sum_{i=1}^N r_{i,t} \right)^2
\]
The smooth cutoff function $\text{activ}(x;\alpha,\beta)$, typically a generalized logistic, ensures that terms appearing in a large fraction of the corpus (e.g., DF $>$ 10%) are heavily penalized for high average activation. When $w_t \equiv 1$, DF-FLOPS reduces to standard FLOPS [2505.15070].

## 3. Integrating DF-FLOPS in the SPLADE-doc Pipeline

DF-FLOPS regularization is seamlessly slotted into the standard SPLADE-doc training procedure. Documents are encoded through a DistilBERT-based encoder. The regularization strength $\lambda$ is quadratically ramped during a 50k-step pretraining phase. Every 100 validation steps, empirical DF estimates are refreshed using held-out mini-batch statistics. The per-term penalty $w_t$ is computed, and DF-FLOPS multiplies the gradient for each term by $w_t^2$, entailing no additional backward passes over standard automatic differentiation. The overall loss is:
\[
L = L_{\mathrm{rank}} + \lambda \cdot \ell_{\mathrm{DF-FLOPS}}
\]
This makes the approach a drop-in replacement for FLOPS in existing codebases and pipelines [2505.15070].

## 4. Sparsity, Latency, and Index Efficiency

Adopting DF-FLOPS regularization yields significant improvements in posting-list lengths and latency. Empirical results on 8.8M MS MARCO passages indexed in Apache Solr v9 show:

| Model                         | Top-token DF% | Matches/query | Passage terms | Latency Avg (ms) | 99th %ile (ms) |
|-------------------------------|---------------|---------------|---------------|------------------|---------------|
| SPLADE-Doc + FLOPS            | 95.8%         | 8.63M         | 583.8         | 922              | 1945          |
| SPLADE-Doc + DF-FLOPS         | 8.0%          | 1.91M         | 301.6         | 161              | 342           |
| BM25                          | —             | —             | —             | 68.9             | 241.3         |

DF-FLOPS pushes the majority of high-frequency tokens below 10% document frequency, reducing the mean number of candidate documents per query ($\approx 4.5\times$ fewer matches) and shrinking average passage vector length by nearly half. This slashes end-to-end retrieval latency by approximately $10\times$ (922ms $\rightarrow$ 161ms) and achieves p99 latency close to BM25, without wholesale stopword removal—critical terms with high DF can still be retained if salient [2505.15070].

## 5. Empirical Performance and Benchmarking

Table 1 from [2505.15070] summarizes in-domain effectiveness and speed on MS MARCO and TREC:

| Model                                     | MRR@10 | Recall@1K | Latency Avg (ms) |
|--------------------------------------------|--------|-----------|------------------|
| BM25                                       | 18.4   | 85.3      | 68.9             |
| SPLADE-Doc + FLOPS ($\lambda \approx 1e-3$)| 32.2   | 92.4      | 922.0            |
| + pruning@150                             | 32.0   | 92.1      | 792.1            |
| FLOPS ($\lambda=0.1$)                      | 29.2   | 88.8      | 331.6            |
| FLOPS ($\lambda=1.0$)                      | 28.3   | 88.4      | 160.9            |
| DF-FLOPS ($\alpha=0.1,\beta=10, \lambda\sim10$) | 30.0   | 92.9      | 161.0            |
| + pruning@150                             | 29.7   | 93.0      | 87.8             |

DF-FLOPS achieves a recall equal to or slightly exceeding FLOPS, with a mean reciprocal rank (MRR@10) only marginally reduced (32.2 → 30.0). Applying modest pruning yields latencies competitive with BM25. On 13 BEIR zero-shot cross-domain tasks, DF-FLOPS outperforms standard FLOPS on 12/13 datasets and generally narrows the performance gap to the BM25 baseline [2505.15070].

## 6. Trade-offs, Tuning, and Production Guidelines

- **Regularization strength ($\lambda$):** Since $\ell_{\mathrm{DF-FLOPS}}$ is numerically smaller for low-DF terms, viable $\lambda$ can be $10\times$–$1000\times$ larger than standard FLOPS without oversparsifying. Empirically, $\lambda \in [10^{-1},10^{3}]$ is robust.
- **DF cutoff ($\alpha$) and steepness ($\beta$):** Setting $\alpha \approx 0.1$ (cutoff for terms $>$10% corpus DF), $\beta=10$ (sharp transition) effectively suppresses unwanted stopwords. Adjustments allow for aggressive or conservative DF control.
- **Effectiveness vs. latency:** In-domain, a 2–4 point MRR@10 drop produces a 10$\times$ latency reduction. Cross-domain, DF-FLOPS often improves robustness and generalization relative to FLOPS, likely by limiting overfitting to dataset-specific high-DF tokens.
- **Deployment recommendations:** DF-FLOPS is recommended for all large-scale inverted-index deployments (e.g., Solr, Lucene) where p99 latency under 200ms is a requirement. For scenarios prioritizing maximum effectiveness (e.g., research benchmarks or two-stage reranking), standard FLOPS or relaxed regularization may be preferred [2505.15070].

## 7. Advances and Related Variants

Beyond DF-FLOPS, other mechanisms for efficiency include hybrid hard/soft thresholding on term weights [2306.11293] and leveraging decoder-only LLMs with echo embeddings (as in Echo-Mistral-SPLADE [2408.11119]). However, DF-FLOPS is distinctive in directly regularizing per-term posting list length via corpus-level statistics rather than relying on static stopword lists or global mask pruning. This raises system-wide gains in inverted index workload, without unduly sacrificing retrieval quality or losing the lexical specificity achievable with learned sparse retrieval.

SPLADE-doc and its DF-targeted regularization constitute a significant step in reconciling the demands of neural IR effectiveness with the strict latency regimes and index pressure of web-scale search. These advances underpin current state-of-the-art production LSR deployments, making the trade space between speed, index size, and search quality highly tunable for practical settings [2505.15070, 2109.10086, 2207.03834].

Source: https://www.emergentmind.com/topics/splade-doc-variant