---
title: 'Learned Sparse Models: SPLADE & uniCOIL'
url: https://www.emergentmind.com/topics/learned-sparse-models-splade-unicoil
type: topic
---

# Learned Sparse Models: SPLADE & uniCOIL

Learned sparse models in information retrieval, exemplified by SPLADE and uniCOIL, are neural architectures that encode queries and documents as high-dimensional sparse vectors aligned to a fixed vocabulary. Unlike classical bag-of-words models, the term weights are learned end-to-end from large-scale supervision, often harnessing masked language modeling heads or reweighting mechanisms on top of transformer embeddings. These models are designed to leverage the efficiency of inverted indexes while introducing semantic matching and expansion capabilities historically associated with dense approaches.

## 1. Model Architectures: SPLADE and uniCOIL

**SPLADE** operates by projecting transformer token embeddings through an MLM head to produce contextual scores for every vocabulary term, followed by max-pooling and log-saturation to enforce sparsity. The resulting representation for an input $x$ (either a query or document) is a vector $w[x]\in\mathbb{R}^{|V|}$ where:

$$
w[k] = \max_{i \in T} \log(1 + \text{ReLU}(W_{i,k})),
$$

$T$ is the sequence of input tokens, and $W_{i,k}$ is the MLM logit for token $i$ mapping to term $k$ [2107.05720, 2109.10086, 2305.18494]. The representations are highly sparse due to explicit regularization.

**uniCOIL** assigns a single learned weight to each token present in the input, omitting full-vocabulary expansion. The vector $d \in \mathbb{R}^{|V|}$ has non-zero entries only at the indices of observed tokens, and scoring is simply the sum over shared nonzero token indices [2112.09628, 2303.13416]. No expansion or reweighting of out-of-vocabulary tokens occurs.

Both models enable direct use of traditional inverted indexes due to sparsity.

## 2. Training Objectives and Regularization

The canonical loss for these models combines retrieval performance with explicit sparsity penalties:

- **Supervised loss** is typically a contrastive or distillation loss (e.g., in-batch negative log-softmax, Margin-MSE to teacher cross-encoder scores) [2107.05720, 2205.04733, 2109.10086].
- **Sparsity induction** employs $\ell_1$ or “FLOPS” regularization:
  $$
  R = \sum_{j \in V} |w_j|,
  $$
  or
  $$
  \ell_{\text{FLOPS}} = \sum_{j \in V} \left(\frac{1}{N}\sum_{i=1}^N w_{i,j}\right)^2,
  $$
  to encourage a small number of nonzeros per representation and balanced posting-list lengths [2107.05720, 2109.10086].

Advanced strategies include **DF-FLOPS**, whereby additional regularization penalizes activations of terms with high document frequency, directly targeting search latency and posting-list length by discouraging frequent terms unless salient [2505.15070].

## 3. Expansion, Weighting, and Vocabulary Design

Learned sparse models integrate lexical expansion, weighting, and controlled vocabulary in various ways:

- **Expansion**: SPLADE performs soft expansion of both queries and documents, assigning weights to vocabulary tokens absent from the input if the MLM head predicts them as contextually relevant [2107.05720, 2109.10086, 2306.16680]; uniCOIL applies only observed-token weighting.
- **Weighting**: Both models learn fine-grained term importance, outperforming binary (presence/absence) weighting as in naive bag-of-words approaches [2303.13416].
- **Vocabulary**: The choice and size of vocabulary directly affect sparsity and effectiveness. Expanded vocabularies (e.g., ESPLADE with 100K natural language tokens) enable greater representational capacity and finer granularity of matching, as established by controlled-vocabulary ablations [2509.16621, 2306.16680, 2401.06703]. Corpus-specific vocabularies and increased vocabulary size can reduce latency and improve recall by shortening average posting lists and preventing over-splitting of tokens.

## 4. Efficiency–Effectiveness Trade-offs and Indexing

The central challenge in learned sparse retrieval is balancing effectiveness with query/document lengths and latency:

- **Pruning Techniques**: Top-$k$ masking, static thresholding, and hybrid thresholding selectively constrain the number of nonzeros in representations, offering precise trade-offs between index size, latency, and retrieval quality [2112.09628, 2306.11293].
- **Two-Step SPLADE**: A cascade inference scheme using an aggressively pruned index for initial retrieval followed by full-SPLADE rescoring yields $12\times$–$40\times$ speed-ups without statistically significant effectiveness loss on most benchmarks [2404.13357].
- **Dynamic Pruning with Traversal Guidance**: Techniques such as BM25-guided traversal, two-level pruning, and block-max index traversal prune candidate lists more efficiently, though aggressive BM25 guidance can harm recall if expansion terms diverge from BM25 distributions [2305.01203].
- **DF-FLOPS Regularization**: By penalizing use of high-frequency terms during training, practical average and tail latency can be reduced by an order of magnitude with minimal decrease in effectiveness [2505.15070].
- **Inference-free Retrieval**: In settings where query encoding is the latency bottleneck, the Li-LSR framework replaces the transformer query encoder with a learned token score table, yielding sub-millisecond query times with negligible loss in accuracy [2505.01452].

## 5. Empirical Performance and Benchmarking

SPLADE and its variants consistently achieve state-of-the-art first-stage retrieval results:

- **SPLADE-v3** pushes first-stage MRR@10 over 40 on MS MARCO dev and boosts BEIR mean nDCG@10 to 51.7, outperforming BM25 and earlier variants by statistically significant margins [2403.06789].
- **Echo-Mistral-SPLADE**, a decoder-only LLM-based variant, outperforms all prior learned sparse and dense retrievers on zero-shot BEIR, achieving an average sparse nDCG@10 of 55.1 [2408.11119].
- **Comparative Table**:

| Model           | MS MARCO MRR@10 | BEIR nDCG@10 | Typical Latency | Notes                      |
|-----------------|-----------------|-------------|-----------------|----------------------------|
| BM25            | ~18.4           | ~41–45      | ~69 ms          | Non-learned sparse         |
| uniCOIL         | 36–37           | 45–48       | 10–12 ms        | No expansion               |
| SPLADE-v3       | 40.2            | 51.7        | 70–100 ms       | Full expansion             |
| DF-FLOPS-SPLADE | ~30             | >BM25       | 88–161 ms       | BM25-like latency          |
| Two-Step SPLADE | 40.0            | 47.6–70.0   | 2–2.4 ms        | 12–40x speedup             |
| Echo-Mistral-SP | —               | 55.1        | —               | LLM backbone, SOTA sparse  |

Empirical findings indicate that document weighting is essential, query weighting yields small but positive gains, and concurrent query and document expansion exhibits a cancellation effect, making single-side expansion generally optimal [2303.13416].

## 6. Adapting to Document Length and Proximal Scoring

For long documents, naive aggregation of segment-level representations introduces noise and degrades performance. Max-score aggregation, which selects the top-scoring segment for each query, enforces local proximity constraints and performs robustly across document length. Incorporating explicit proximity models, such as ExactSDM and SoftSDM, further improves effectiveness; ExactSDM, which uses only exact term dependence within local windows, generally matches or outperforms SoftSDM and does not rely on expansion, facilitating compatibility with both SPLADE and uniCOIL-like models [2305.18494].

## 7. Broader Implications and Design Recommendations

Recent studies clarify that sparse lexical representation in SPLADE-like models is less about traditional lexical semantics and more about introducing sparse high-dimensional "feature buckets." The effectiveness is governed by vocabulary size and regularization, not surface token meaning [2306.16680]. Consequently, future models may combine interpretable lexical features and latent dimensions or exploit expanded vocabularies to achieve enhanced recall and precise efficiency control. The core design guidelines are:

- Always learn weightings for both query and document sides, with document weighting critical to effectiveness [2303.13416].
- Prefer expansion on one side (typically documents).
- Tune sparsity regularizers (FLOPS, $\ell_1$, DF-FLOPS) to balance effectiveness against latency, using corpus-specific vocabularies for additional gains [2401.06703, 2509.16621, 2505.15070].
- Employ dynamic or static pruning, hybrid thresholding, or cascade retrieval to optimize the efficiency–effectiveness trade-off [2404.13357, 2306.11293].
- For production deployment, integrate latency- and cost-aware regularization such as DF-FLOPS, consider model distillation from cross-encoders, and benchmark index sizes and retrieval times comprehensively.

Learned sparse models, especially SPLADE and its modern descendants, provide a scalable, interpretable, and highly competitive alternative to both traditional sparse and modern dense retrievers, with ongoing advances in efficiency, representational power, and practical deployability [2107.05720, 2109.10086, 2403.06789, 2408.11119].

Source: https://www.emergentmind.com/topics/learned-sparse-models-splade-unicoil