---
title: 'REALM: Retrieval-Augmented Language Model'
url: https://www.emergentmind.com/topics/realm-288645db-56b6-4c40-a464-d6eeaba6d260
type: topic
---

# REALM: Retrieval-Augmented Language Model

REALM, short for Retrieval-Augmented Language Model Pre-Training, is a retrieve-then-predict framework that augments masked language model pre-training with a learned, large-scale neural knowledge retriever operating over a corpus such as Wikipedia [2002.08909]. Its central premise is that world knowledge need not be stored solely in neural parameters. Instead, REALM factorizes prediction into retrieval of latent supporting documents and conditional prediction given those documents, thereby making knowledge storage more modular and more inspectable while preserving end-to-end training. In the original formulation, the same retrieval mechanism is used during pre-training, fine-tuning, and inference, and the retriever itself is trained unsupervised from masked language modeling signals rather than from labeled retrieval supervision [2002.08909].

## 1. Motivation and problem setting

REALM was introduced against the background of large pretrained language models whose factual competence is encoded implicitly in parameters. That paradigm yields strong performance, but it ties knowledge coverage to parametric scale: storing more facts requires ever-larger networks, and updating knowledge often entails retraining. REALM decouples these roles by treating the corpus as the primary knowledge store and the neural model as the mechanism that learns what to retrieve and how to use it [2002.08909].

Three motivations are explicit. First, **capacity and scaling**: knowledge growth becomes a corpus-and-index problem rather than only a parameter-count problem. Second, **interpretability**: predictions are conditioned on retrieved passages whose content can be inspected as provenance for the answer. Third, **modularity**: updating the knowledge source can be done by refreshing the corpus and index, even though some facts may still remain memorized in the encoder [2002.08909].

The target use case is knowledge-intensive language understanding, especially open-domain question answering. REALM was evaluated on NaturalQuestions-Open, WebQuestions, and CuratedTrec, where answers depend on broad factual coverage rather than narrow task-specific pattern matching [2002.08909]. In this setting, the method differs from heuristic retrieval pipelines that rely on BM25, TF-IDF, or entity linking to preselect a small candidate set, because REALM learns retrieval directly from masked language modeling and searches over millions of candidate passages via maximum inner product search [2002.08909].

## 2. Probabilistic formulation and model architecture

REALM formalizes both masked language modeling and open-domain QA with a latent retrieved document $z$ from a corpus $\mathcal{Z}$. For input $x$ and output $y$, the model marginalizes over retrieved documents:
$$
p(y \mid x) = \sum_{z \in \mathcal{Z}} p(y \mid z, x)\, p(z \mid x).
$$

This decomposition yields two components. The first is a **knowledge retriever** $p(z \mid x)$ implemented as a dense inner-product retriever. Query $x$ and document $z$ are encoded by Transformer-based encoders $\phi_\theta$ and $\psi_\theta$, with relevance score
$$
s_\theta(x, z) = \phi_\theta(x)^\top \psi_\theta(z).
$$
The induced retrieval distribution is a softmax over the corpus:
$$
p_\theta(z \mid x) =
\frac{\exp\left(s_\theta(x, z)\right)}
{\sum_{z' \in \mathcal{Z}} \exp\left(s_\theta(x, z')\right)}.
$$
In practice, the denominator is approximated over top-$k$ candidates returned by maximum inner product search, after which the model recomputes exact probabilities on that retrieved set [2002.08909].

The second component is a **knowledge-augmented encoder** $p(y \mid z, x)$, a separate Transformer that encodes the concatenation of the input and the retrieved passage so that cross-attention can bind them. During pre-training, REALM uses standard MLM decoding for masked tokens. If $\mathcal{M}$ indexes masked positions, then
$$
p(y \mid z, x) = \prod_{i \in \mathcal{M}} p(y_i \mid z, x),
$$
with each masked-token probability parameterized from the hidden state at the masked position. During QA fine-tuning, the answer is treated as an extractive span in a retrieved passage, scored by an MLP over start and end hidden states in the concatenated sequence $[x; z]$ [2002.08909].

Architecturally, the retriever and reader are distinct. The retriever uses BERT-style Transformers, [CLS] pooled representations, and learned linear projections for both queries and documents. The reader is another Transformer specialized for conditioning on retrieved passages. This separation is important: retrieval quality and answer extraction are optimized jointly through marginal likelihood, but retrieval remains a latent decision rather than an externally fixed preprocessing stage [2002.08909].

## 3. Unsupervised retriever learning from masked language modeling

The defining technical contribution of REALM is that the retriever is pretrained end-to-end without retrieval labels. For MLM, the loss over masked positions is
$$
\mathcal{L}(x)
=
- \sum_{i \in \mathcal{M}}
\log \left(
\sum_{z \in \mathcal{Z}}
p_\theta(z \mid x)\, p_\phi(y_i \mid x, z)
\right).
$$
This objective treats retrieval as a latent variable and optimizes the marginal likelihood of the correct masked tokens [2002.08909].

The retriever receives gradient signal according to whether a document makes the prediction better than expected. An intuitive form given in the paper is
$$
\nabla_\theta \log p(y \mid x)
=
\sum_{z \in \mathcal{Z}} r(z)\, \nabla_\theta s_\theta(x, z),
$$
where
$$
r(z)
=
\left(\frac{p_\phi(y \mid z, x)}{p(y \mid x)} - 1\right) p_\theta(z \mid x).
$$
Documents that improve prediction relative to the marginal baseline receive positive updates; unhelpful documents are demoted [2002.08909]. This is the core mechanism by which the masked language model itself trains the retriever.

Several inductive biases make this objective workable at corpus scale. REALM uses **salient span masking** rather than random masking, focusing on named entities and dates identified by a BERT-based NER tagger and regex rules. The purpose is to place masked tokens in knowledge-heavy contexts where external retrieval is likely to matter. The model also includes a learned **null document** $z_\emptyset$ so that it can learn not to retrieve when local context suffices. When the pre-training corpus and knowledge corpus coincide, the exact source document of the masked input is excluded from retrieval candidates to prevent degenerate lookup behavior. To avoid a cold start, the query and document encoders are warm-started with the **Inverse Cloze Task**, and the knowledge-augmented encoder is warm-started from BERT-base uncased with 12 layers, hidden size 768, and 12 heads [2002.08909].

REALM also defines a retrieval-centric diagnostic:
$$
\mathrm{RU}(z \mid x)
=
\log p(y \mid z, x) - \log p(y \mid z_\emptyset, x).
$$
This **retrieval utility** score measures how much a given document helps relative to the null document. The paper reports that retrieval utility increases steadily during pre-training and is more predictive of downstream QA performance than overall log-likelihood, which suggests that retrieval-specific learning quality is not fully captured by the standard MLM objective alone [2002.08909].

## 4. Knowledge store, indexing, and systems design

REALM’s knowledge store in the reported experiments is the English Wikipedia snapshot of December 20, 2018. Articles are split greedily into passages of up to 288 wordpieces, producing just over 13 million document blocks. Each block stores title and body, and the document encoder embeds the sequence `[CLS title SEP body SEP]` [2002.08909].

Because document embeddings depend on the evolving retriever parameters, the index becomes stale during training. REALM addresses this with **asynchronous index refresh**. Two jobs run in parallel: a trainer updates retriever and reader parameters, while an index builder periodically receives a parameter snapshot, re-embeds all documents, and rebuilds the MIPS index. In the experiments, refreshes occur approximately every 500 training steps; more frequent refreshes stabilize optimization, while overly stale indices degrade performance [2002.08909].

The pre-training configuration is large but straightforward. REALM trains for 200k steps on 64 Cloud TPUs with batch size 512 and learning rate $3 \times 10^{-5}$ using BERT’s optimizer. Document embedding for index refresh is parallelized over 16 TPUs. During pre-training, the model marginalizes over $k=8$ retrieved candidates, including the null document. During QA inference, it considers the top-5 retrieved documents, and top-5 retrieval plus reading fits on a single 12GB GPU [2002.08909].

These engineering choices are integral rather than incidental. REALM depends on cached dense document vectors, approximate nearest-neighbor search for tractability, and exact recomputation of the retrieval distribution on the retrieved top-$k$ set. The method therefore occupies a specific systems niche: it is more complex than a pure parametric language model, but it avoids the need to backpropagate through an explicit search over the entire corpus at each step [2002.08909].

## 5. Fine-tuning for open-domain question answering and empirical results

For open-domain QA, REALM treats the question as the query and predicts extractive spans from retrieved passages. In fine-tuning, the document-side encoder is typically fixed from pre-training, while the query encoder continues to be updated. The reader concatenates question and passage, predicts start and end positions with an MLP, and marginalizes over spans in the retrieved passages. At inference time, the system retrieves top-5 passages via MIPS and computes $p(y \mid x)$ by marginalizing over them [2002.08909].

The reported exact-match test results on three benchmarks are as follows [2002.08909]:

| Model | NQ | WQ / CT |
|---|---:|---:|
| T5 (11B) | 34.5 | 37.4 / — |
| ORQA | 33.3 | 36.4 / 30.1 |
| REALM (target = Wikipedia; unlabeled = Wikipedia) | 39.2 | 40.2 / 46.8 |
| REALM (target = CC-News; unlabeled = Wikipedia) | 40.4 | 40.7 / 42.9 |

Across NaturalQuestions-Open, WebQuestions, and CuratedTrec, REALM improves absolute accuracy by 4–16 points over prior systems, while being roughly 30× smaller than T5-11B [2002.08909]. The paper does not report TriviaQA results [2002.08909].

The qualitative case studies clarify what these gains mean. For the masked sentence “An equilateral triangle is easily constructed … because 3 is a [MASK] prime,” REALM retrieves a passage discussing Fermat primes and constructible regular polygons; the marginalized probability of “Fermat” increases from much less than $10^{-14}$ without retrieval to approximately $0.129$ after marginalization over retrieved documents [2002.08909]. This is not merely an accuracy improvement. It shows that the model can recover a fact by external evidence rather than by pure parameter memorization.

## 6. Interpretability, modularity, limitations, and later refinements

REALM’s most frequently cited qualitative properties are **interpretability** and **modularity**. Because predictions depend on retrieved passages, users can inspect supporting evidence. Because the knowledge store is external, updating the corpus can change model behavior without retraining the full network. The paper illustrates this by swapping a 2018 Wikipedia snapshot with a 2020 snapshot: the masked prompt about “Jennifer [MASK] formed the production company Excellent Cadaver” becomes solvable as “Lawrence” under the updated page set. At the same time, some facts remain memorized in the encoder, so corpus updates do not imply perfectly clean separation between parametric and non-parametric knowledge [2002.08909].

Several limitations are equally explicit. REALM requires building and refreshing a large ANN or MIPS index and repeatedly re-embedding millions of passages, which increases engineering complexity. Performance depends on corpus coverage and phrasing; missing or idiosyncratically expressed facts can still defeat retrieval. Approximate nearest-neighbor search and top-$k$ truncation introduce approximation error. Retrieval and reading add inference latency, although the reported setup keeps $k$ small. Finally, training is sensitive to design choices such as salient masking and refresh frequency; stale indices or weak masking policies degrade learning [2002.08909].

In the broader retrieval-augmented landscape, REALM occupies a specific historical position. Relative to heuristic sparse-retrieval systems such as DrQA, HardEM, GraphRetriever, and PathRetriever, it learns retrieval from MLM signal rather than from a separate heuristic pipeline. Relative to ORQA, it shares the latent-variable perspective and ICT initialization, but adds unsupervised end-to-end retriever pre-training with backpropagation through retrieval at corpus scale. Later retrieval-augmented generation models also marginalize over retrieved documents, but typically use seq2seq generation rather than REALM’s extractive reader; the original novelty lies in unsupervised dense-retriever pre-training through masked LM and scalable marginalization over latent retrieval [2002.08909].

A subsequent study, “Simple and Efficient ways to Improve REALM” [2104.08710], argued that the original fine-tuning regime was substantially undertrained. Without changing the model design, it improved performance by using exact MIPS during fine-tuning, larger batches via distributed training, stronger passage supervision from Natural Questions long answers, and much larger reader-side $k$ at inference. The resulting “REALM++” reached 44.8 EM on NQ, 45.6 on WQ, and 49.7 on CT, roughly a 5.5-point absolute gain over the original baseline configuration [2104.08710]. This later result does not alter the conceptual identity of REALM, but it reinforces a central lesson of the original work: retrieval-augmented pre-training is highly sensitive to the interaction between retriever quality, reader capacity, supervision, and search configuration.

REALM therefore marks a transition in language-model design from purely parametric factual storage toward explicit, trainable access to external corpora. Its lasting significance is not only that it improved open-domain QA, but that it showed retrieval itself can be pretrained end-to-end and unsupervised, using masked language modeling as the learning signal for a latent document-selection policy at corpus scale [2002.08909].

Source: https://www.emergentmind.com/topics/realm-288645db-56b6-4c40-a464-d6eeaba6d260