---
title: Salient Phrase Aware Retriever (SPAR)
url: https://www.emergentmind.com/topics/salient-phrase-aware-retriever-spar
type: topic
---

# Salient Phrase Aware Retriever (SPAR)

The Salient Phrase Aware Retriever (SPAR) is a dense retrieval architecture designed to combine the semantic matching strengths of modern neural bi-encoders with the robust lexical and salient-phrase matching abilities characteristic of classical sparse retrievers such as BM25. By explicitly adding a learned lexical model to a standard dense retriever, SPAR produces a single, efficient retrieval system capable of both deep semantic understanding and exact lexical matching, eliminating the engineering complexity of hybrid systems while matching or surpassing their performance [2110.06918].

## 1. Motivation and Context

Dense retrieval systems, primarily based on bi-encoder Transformers like DPR and ANCE, have demonstrated strong open-domain question answering performance via learned continuous vector representations. These models excel at semantic matching but systematically fail when queries hinge on rare entities or out-of-vocabulary phrases. Sparse methods such as BM25, in contrast, reliably recover passages via exact token overlap, outperforming dense retrievers in out-of-domain settings and factual/entity-centric tasks, as substantiated by BEIR benchmark results. Prior work has attempted to address this by hybridizing sparse and dense methods (e.g., via retrieval+merging), but such methods increase system complexity: dual indices, disparate infrastructure (e.g., Lucene and FAISS), and latency overhead.

SPAR directly addresses whether a single dense retriever can emulate the matching properties of both approaches, thus bridging the gap traditionally thought to separate dense and sparse models without hybridization drawbacks [2110.06918].

## 2. System Architecture

SPAR is constructed by augmenting a standard dense retriever with an additional learned "lexical model." The architecture comprises:

- **Base Dense Retriever:** A bi-encoder with query encoder $Q$ and passage encoder $P$, both implemented as BERT-base models producing $d$-dimensional vectors. Semantic similarity is computed via the dot product $Q(q) \cdot P(p)$.

- **Learned Lexical Model $\Lambda$:** Also a bi-encoder $(\hat{Q}, \hat{P})$ with the same architecture, but trained via distillation to imitate a sparse retriever (e.g., BM25 or UniCOIL). Training queries are real corpus sentences, embedding salient lexical tokens (entities/phrases), with positives/negatives selected via the sparse retriever.

The base and lexical streams are joined post hoc by vector concatenation and a balanced scoring mechanism:

\[
\mathrm{score}_{\mathrm{SPAR}}(q,p) = Q(q) \cdot P(p) + \alpha\, \hat{Q}(q) \cdot \hat{P}(p)
\]

where $\alpha$ is a tunable scalar balancing the lexical component. At index time, concatenated passage vectors $v_p = [P(p), \hat{P}(p)] \in \mathbb{R}^{2d}$ are stored; at query time, $v_q = [Q(q), \alpha\, \hat{Q}(q)]$ is constructed.

This method allows a single FAISS ANN index to support both dense and lexical signals with no need for hybrid merging [2110.06918].

## 3. Training Paradigm

### 3.1 Model Distillation via Contrastive Loss

The lexical model $\Lambda$ is trained to replicate a sparse teacher’s ranking behavior. The process includes:

- Unlabeled queries $U$ gathered either as random Wikipedia sentences ($\sim$37M) or synthetic QA questions (e.g., 65M PAQ).
- For each $q \in U$, the teacher retrieves top $K$ passages, the best $n_p$ as positives, the next $n_n$ as negatives.
- Training uses in-batch InfoNCE-style contrastive loss:

\[
L(q) = -\frac{1}{n_p} \sum_{i=1}^{n_p} \log \frac{\exp(\hat{Q}(q) \cdot \hat{P}(p^+_i) / \tau)} 
{\sum_{j=1}^{n_p+n_n} \exp(\hat{Q}(q) \cdot \hat{P}(p_j) / \tau)}
\]

with temperature $\tau=1$. No auxiliary MSE or KL-divergence regularization is needed.

### 3.2 Training Setup

- Encoder: BERT-base, 12 layers, 768 dimensions.
- Batch: 32 per GPU (64 GPUs), $\sim$2K effective.
- $n_p=10$, $n_n=5$, $K=15$.
- Train time: $\sim$72 hours (20 epochs, Wiki) on V100 GPUs.
- The lexical model achieves MRR $>$92% vs. BM25 on held-out queries, demonstrating near-perfect behavioral imitation.

## 4. Empirical Performance and Evaluation

SPAR’s effectiveness is documented across several axes:

### 4.1 Open-Domain QA (ODQA)

In micro-averaged Acc@100 over five QA benchmarks (NQ, SQuAD, TriviaQA, WebQuestions, TREC):

| System             | Acc@100   |
|--------------------|-----------|
| DPR                | 83.0%     |
| BM25               | 81.7%     |
| DPR+BM25 (hybrid)  | 87.8%     |
| SPAR-Wiki          | 88.7%     |
| SPAR-PAQ           | 88.9%     |

SPAR consistently equals or exceeds the hybrid baseline, improving by +1.1 points on average.

### 4.2 MS MARCO Passage Retrieval

SPAR nearly matches the best hybrid approaches on MRR@10, with simplified infrastructure:

| System                      | MRR@10 |
|-----------------------------|--------|
| ANCE                        | 33.0   |
| ANCE+BM25                   | 34.7   |
| SPAR(ANCE+BM25)             | 34.4   |
| SPAR(ANCE+UniCOIL)          | 36.9   |
| RocketQA                    | 37.0   |
| RocketQA+UniCOIL            | 38.8   |
| SPAR(RocketQA+UniCOIL)      | 38.6   |

### 4.3 Out-of-Domain Robustness

SPAR, trained only on MS MARCO, sets a new SOTA on BEIR (average nDCG@10 on 11/14 tasks) and outperforms hybrids in entity-centric QA (EntityQuestions: Acc@20—SPAR(Wiki) 73.6%, BM25 70.8%, DPR 56.6%).

## 5. Robustness, Ablations, and Analysis

### 5.1 Lexical Overlap

Rank-Biased Overlap (RBO) analysis between top-100 retrievals:

- DPR vs. BM25: $RBO\approx0.10$
- $\Lambda$ (lexical model) vs. BM25: $RBO\approx0.50$–$0.60$

$\Lambda$ learns rankings closely aligned to its teacher’s.

### 5.2 Token-Shuffle Test

When query tokens are randomized, dense models’ (DPR) performance drops sharply, whereas BM25 and $\Lambda$ remain robust—a hallmark of bag-of-words matching.

### 5.3 Addition and Fusion Ablations

- Adding BM25 atop SPAR yields negligible (+0.1 pt) further improvement, indicating SPAR has incorporated nearly all sparse signal.
- Weighted vector concatenation of dense and lexical streams surpasses weighted sum or joint bi-encoder training by $\sim$1 point, favoring independent training with late fusion.

## 6. Implementation and Practical Considerations

- Encoder: BERT-base, 12 layers, 768-dim per stream (total 1536-dim per passage).
- FAISS HNSW index, $\sim$52 GB for MS MARCO (vs. 26 GB for DPR), supports single-index retrieval with concatenated vectors.
- Query latency: 20 ms (SPAR-concat), 10 ms (DPR), and 55 ms (BM25).
- Only a single FAISS index is required; no secondary index or external merging.

SPAR’s increased embedding dimension and index size represent a trade-off against the operational simplicity and retrieval accuracy gains.

## 7. Limitations and Prospective Directions

SPAR’s dense-concat method doubles vector size, impacting storage and latency; the weighted-sum variant offers a partial mitigation but induces a minor accuracy loss. Behavior of the lexical model is bounded by its teacher—any systematic BM25/UniCOIL error propagates. Potential future improvements include self-supervised lexical signal learning, which could decouple $\Lambda$ from sparse teacher limitations.

The empirical question of why post-hoc fusion outperforms joint optimization, and why SPAR’s edge over hybrids increases with higher $k$, is posed as an open area for future research [2110.06918].

In sum, SPAR demonstrates that a dense retriever—when equipped with a learned lexical stream—can recapitulate the exact-match strengths and robust generalization of sparse indices, challenging the presumed modality divide and advancing single-index neural retrieval.

Source: https://www.emergentmind.com/topics/salient-phrase-aware-retriever-spar