DPR: Dense Passage Retriever Overview

• DPR is a dual-encoder retrieval model that uses learned dense embeddings to replace sparse lexical matching for open-domain question answering.
• It employs separate BERT-based encoders for questions and passages with dot-product similarity and contrastive learning using in-batch and hard negatives.
• DPR achieves high retrieval efficiency and significant improvements in top-k accuracy on benchmarks like Natural Questions and TriviaQA, enhancing overall QA performance.

DPR, short for Dense Passage Retriever, is a dual-encoder retrieval model for the first stage of open-domain question answering: given a question, it retrieves a small set of passages likely to contain the answer. Introduced in "Dense Passage Retrieval for Open-Domain Question Answering" (Karpukhin et al., 2020), it replaces sparse term matching such as TF-IDF or BM25 with learned low-dimensional representations of questions and passages, scored by a simple inner product. The model was presented as a practically deployable dense retrieval system that achieves large gains in top-$k$ retrieval accuracy and improves end-to-end question answering without expensive pretraining or cross-attention at retrieval time (Karpukhin et al., 2020).

1. Problem setting and retrieval formulation

Open-domain question answering typically decomposes into retrieval followed by reading. DPR targets the retrieval stage: selecting candidate passages from a large corpus before an answer extractor or reader is applied. In the original formulation, this stage is evaluated over Wikipedia split into 21 million 100-word passages, with the retriever expected to return a small top-$k$ set that contains an answer-bearing passage (Karpukhin et al., 2020).

The central departure from classical retrieval is the abandonment of sparse lexical scoring in favor of learned dense embeddings. Traditional baselines in the original study are Lucene-BM25 and related sparse vector space models. DPR instead learns separate vector spaces for questions and passages and ranks passages by the similarity of the corresponding embeddings (Karpukhin et al., 2020). This suggests a retrieval regime in which relevance is modeled as geometric proximity rather than token overlap, while still preserving the decomposability required for efficient large-scale indexing.

The primary evaluation measure is top-$k$ accuracy: the fraction of questions for which at least one of the top $k$ retrieved passages contains the answer span. Gold passage labels are obtained either by matching the original context or by distant supervision using the top BM25 passage containing the answer (Karpukhin et al., 2020). In the original experiments, DPR was evaluated on Natural Questions (NQ), TriviaQA, WebQuestions (WQ), CuratedTREC (TREC), and SQuAD v1.1 using their standard train/dev/test splits (Karpukhin et al., 2020).

2. Dual-encoder architecture

At its core, DPR uses two separate Transformer encoders, one for questions and one for passages. Each encoder is initialized from BERT-base (uncased), and the final hidden state of the [CLS] token is used as the embedding. The embedding dimensionality is $d=768$ (Karpukhin et al., 2020).

The two encoders have the same architecture but do not share parameters, allowing specialization for queries and document contexts. The representations are

$$v_q = E_Q(q) \in \mathbb{R}^d,\qquad v_p = E_P(p) \in \mathbb{R}^d.$$

Similarity is the dot product

$\operatorname{sim}(q,p)=E_Q(q)^T E_P(p).$

This design is deliberately simple. Because the score decomposes into independent query and passage encodings, all passage vectors can be precomputed offline and indexed for maximum inner-product search. In contrast to interaction-heavy architectures, no cross-attention is applied at retrieval time (Karpukhin et al., 2020).

The use of non-shared encoders is a defining property of the original DPR system. A plausible implication is that the model can allocate representational capacity asymmetrically: one encoder can optimize for query semantics and the other for passage content organization, even though both start from the same BERT-base initialization.

3. Contrastive learning objective and negative construction

DPR is trained on tuples of a question, one positive passage, and multiple negative passages: $(q, p^+, \{p^-_j\}_{j=1}^n).$ The loss is the negative log-likelihood of the positive passage under a softmax over one positive and $n$ negatives:

$$L(q,p^+,p^-_1,\ldots,p^-_n) = -\log \frac{\exp(\operatorname{sim}(q,p^+))} {\exp(\operatorname{sim}(q,p^+))+\sum_{j=1}^n \exp(\operatorname{sim}(q,p^-_j))}.$$

In practice, the model relies heavily on in-batch negatives. If a batch contains $k$0 question-positive pairs, DPR computes a score matrix

$k$1

of size $k$2, and each off-diagonal score $k$3 serves as a negative for question $k$4 whenever $k$5 (Karpukhin et al., 2020). This multiplies the number of negatives without extra encoder calls.

The training procedure also incorporates one or two hard negatives per question mined by BM25: top-ranked passages that do not contain the answer. Ablations on NQ development data show that in-batch negatives dramatically outperform ordinary per-example negative sampling, that retrieval accuracy rises with batch size, and that adding a single BM25-mined hard negative further improves performance (Karpukhin et al., 2020). Alternative similarity functions such as $k$6 and cosine distance, and alternative loss forms such as triplet loss, were found to be comparable but slightly weaker than dot-product similarity with the softmax objective (Karpukhin et al., 2020).

An important empirical point is data efficiency: even 1,000 training examples suffice for DPR to surpass BM25 in top-20 accuracy on NQ development data (Karpukhin et al., 2020). This is one of the paper’s strongest arguments that dense retrieval can be trained effectively from relatively small supervised QA datasets.

4. Indexing, search, and systems characteristics

Once trained, DPR encodes all passages in the corpus offline with the passage encoder $k$7 and stores the resulting vectors in a FAISS index using an HNSW graph. Query-time inference consists of a single forward pass through the question encoder to obtain $k$8, followed by maximum inner-product search over the indexed passage embeddings (Karpukhin et al., 2020).

Because similarity is decomposable, DPR avoids the $k$9 cross-attention cost of interaction-based retrieval over a corpus of size $k$0. The original system reports retrieval of 100 passages in under 1 ms on a single CPU server, approximately 995 queries per second, whereas BM25 implemented with Lucene achieves approximately 24 queries per second per thread (Karpukhin et al., 2020). The paper also notes a nontrivial indexing cost: building the FAISS index for 21 million vectors takes approximately 8.5 hours (Karpukhin et al., 2020).

These systems properties are central to why DPR became a standard dense retriever in retrieval-augmented pipelines. Its efficiency stems not from approximate semantics alone, but from the specific architectural constraint that question and passage scoring reduces to an inner product between independently encoded vectors.

5. Retrieval and end-to-end QA performance

The original paper reports large absolute gains over BM25 in top-20 passage retrieval accuracy when models are trained per dataset. On NQ, top-20 accuracy rises from 59.1% for BM25 to 78.4% for DPR, a gain of 19.3 percentage points. On TriviaQA, it rises from 66.9% to 79.4% (+12.5 percentage points). On WQ, it rises from 55.0% to 73.2% (+18.2 percentage points). On TREC, it rises from 70.9% to 79.8% (+8.9 percentage points) (Karpukhin et al., 2020).

The gains are also strong at smaller retrieval cutoffs. At $k$1, DPR exceeds BM25 on NQ by 22 percentage points, 65.2% versus 42.9% (Karpukhin et al., 2020). Training a single joint DPR over NQ, TriviaQA, WQ, and TREC yields similar improvements, with especially notable benefits for smaller datasets such as WQ and TREC that benefit from additional training examples (Karpukhin et al., 2020).

The retriever improvements translate into better end-to-end question answering. The paper states that Exact Match with DPR plus reader exceeds BM25 plus reader and prior state-of-the-art systems by up to 12 percentage points on several benchmarks (Karpukhin et al., 2020). In this sense, DPR was not presented merely as a retrieval component, but as a replacement for sparse first-stage retrieval in complete open-domain QA systems.

6. Failure modes, caveats, and domain dependence

Despite its strong average gains, DPR has clear limitations. The original paper notes that it can miss very rare, highly specific phrases, where BM25 can succeed by spotting a unique string that a 768-dimensional vector fails to preserve (Karpukhin et al., 2020). It also performs less favorably on SQuAD, where questions were authored after seeing their contexts and token overlap is unusually high; in that setting BM25 remains competitive or better (Karpukhin et al., 2020).

A later stress test on specialized target domains reinforces this caveat. "Synthetic Target Domain Supervision for Open Retrieval QA" (Reddy et al., 2022) reports that standard DPR lags behind BM25 under domain shift in closed and specialized domains such as COVID-19, and proposes fine-tuning DPR with synthetic target-domain training examples generated from unlabeled target text. In the reported COVID retrieval results, adapted DPR outperforms both BM25 and the off-the-shelf DPR-Multi model, and an ensemble of BM25 with adapted DPR achieves the best Match@$k$2 scores and the best end-to-end open-retrieval QA F1 (Reddy et al., 2022).

These results indicate that DPR’s learned representations are sensitive to the domain of supervision. A plausible implication is that dense retrieval’s advantages are largest when the embedding space has been exposed to the relevant vocabulary, style, and answer-bearing discourse patterns, whereas sparse retrieval retains robustness when lexical specificity dominates.

7. Subsequent analyses and extensions

Several later works in the provided literature examine DPR beyond its original open-domain QA setting. "Dense Passage Retrieval: Is it Retrieving?" (Reichman et al., 2024) studies DPR mechanistically through probing, layer activation analysis, and model editing. Its main conclusion is that DPR fine-tuning decentralizes how knowledge is stored in the network, creating multiple access pathways to the same information, but that retrieval remains bounded by the internal knowledge of the pre-trained model (Reichman et al., 2024). This suggests that DPR fine-tuning primarily reconfigures access to existing knowledge rather than injecting fundamentally new knowledge into the encoder.

In multimodal retrieval-augmented systems, DPR has also been made differentiable and jointly optimized with downstream generation. "Retrieval Augmented Visual Question Answering with Outside Knowledge" (Lin et al., 2022) integrates DPR with answer generation for OK-VQA by normalizing top-$k$3 retrieval scores with a softmax and back-propagating downstream losses into the query encoder while keeping the document encoder fixed. The reported model improves both retrieval diagnostics and final VQA score over strong DPR-based baselines, while reducing the number of retrieved documents needed during training (Lin et al., 2022).

More recent retrieval work has questioned the adequacy of Euclidean and cosine geometry for dense embeddings in out-of-distribution settings. "MA-DPR: Manifold-aware Distance Metrics for Dense Passage Retrieval" (Liu et al., 16 Sep 2025) replaces standard vector-space distance with a shortest-path distance on a nearest-neighbor passage graph, arguing that DPR embeddings often lie on lower-dimensional, non-linear manifolds. The paper reports up to 26% relative Recall@20 improvements on out-of-distribution passage retrieval while maintaining comparable in-distribution performance and adding only a small query-time overhead (Liu et al., 16 Sep 2025).

Taken together, these later studies preserve the basic DPR template—dual encoders, decomposable scoring, and efficient nearest-neighbor search—while refining the understanding of what the model encodes, when it fails, and how it can be adapted or extended. The original contribution remains the same: a simple dual-encoder framework with dot-product similarity and contrastive training, using in-batch and hard negatives, is sufficient to surpass sparse retrieval in large-scale open-domain question answering (Karpukhin et al., 2020).