---
title: Unigram Language Models
url: https://www.emergentmind.com/topics/unigram-language-models
type: topic
---

# Unigram Language Models

A unigram language model is a probabilistic model over a vocabulary of words or subword units in which the tokens are assumed to be drawn independently and identically from a single multinomial distribution. This non-contextual modeling paradigm is foundational both as a baseline in language modeling and as a building block for segmental and subword algorithms in modern natural language processing. Despite its simplicity, substantial methodological innovation has occurred in estimation strategies, smoothing, and neuralization, as well as in its integration into tokenization methods and as a structural prior within neural architectures.

## 1. Theoretical Foundations

Let $\mathcal{V}$ denote a vocabulary (either fixed or, in some formulations, open-ended) and let $N$ be the corpus token count. The core model specifies the probability $p(w)$ of any token $w\in\mathcal{V}$ as context-independent. For a string $x_1, x_2, ..., x_N$, the probability assigned under the unigram model is:
$$
P(x_1, \ldots, x_N) = \prod_{i=1}^N p(x_i)
$$
Maximum likelihood estimation (MLE) from a corpus $C$ proceeds via frequency counts:
$$
p_{\mathrm{MLE}}(w) = \frac{\mathrm{Count}(w)}{N}
$$
This plug-in estimator is standard in both classical and neural NLP pipelines, e.g., as the default in HITgram and for the empirical distribution priors in neural generators [2412.10717, 2212.09686].

However, the empirical unigram faces two severe limitations: (1) it assigns zero probability to out-of-vocabulary (OOV) tokens, and (2) its sample bias inflates estimates of frequent items while under-representing rare or unobserved types, with these biases scaling with corpus size [2106.02289].

## 2. Unigram LM for Subword Tokenization

The unigram LM tokenization method, as implemented in SentencePiece, is a probabilistic generative model over segmentations of sentences into subword units [2004.03720]. Let $V$ be a candidate subword vocabulary and $\theta = \{p(u) : u \in V\}$, with $\sum_{u \in V} p(u) = 1$. For a string $x_t$, all possible segmentations $s \in \mathrm{Seg}(x_t)$ (ways to cover $x_t$ with tokens from $V$) are enumerated:
$$
P(s \mid \theta) = \prod_{u \in s} p(u)
$$
The marginal likelihood for $x_t$ is:
$$
P(x_t| \theta) = \sum_{s \in \mathrm{Seg}(x_t)} \prod_{u \in s} p(u)
$$
An expectation-maximization (EM) procedure is used for optimization:

- **E-step:** Compute posteriors $q_t(s) = \frac{P(s | \theta^{(\mathrm{old})})}{P(x_t | \theta^{(\mathrm{old})})}$
- **M-step:** Update: $p^{\mathrm{new}}(u) = \frac{C(u)}{\sum_{v \in V} C(v)}$, where $C(u)$ is the expected total count.

To fix vocabulary size $k$, an initial oversized set is pruned iteratively, measuring each candidate's removal impact on corpus perplexity and removing the least useful (highest-loss) tokens until $|V| = k$. This global, iterative pruning, driven by a likelihood objective, contrasts sharply with greedy BPE [2004.03720].

At inference, the optimal segmentation $s^*$ is found via Viterbi-style dynamic programming:
$$
s^{*} = \underset{s \in \mathrm{Seg}(x)}{\mathrm{argmax}} \ \sum_{u \in s} \log p(u)
$$
This ensures segmentations align with both maximum likelihood and final vocabulary.

## 3. Advanced Smoothing and Priors

Smoothing is critical for robust unigram modeling. HITgram implements additive (Laplace/add-$k$) smoothing so that even unseen words receive nonzero probabilities:
$$
P_{\mathrm{Lap}}(w) = \frac{\mathrm{Count}(w) + \alpha}{N + \alpha |\mathcal{V}|}
$$
For finer control, $0 < \alpha < 1$ (add-$k$). More sophisticated approaches, such as adapter-generator models, interpolate the empirical 1-gram with a character-level generator. Goldwater et al.'s construction [2106.02289] combines a Pitman–Yor process (PYP) smoothing prior over clusters (adaptors) with a generative character LM for OOV handling:
$$
p(w_n) = \underbrace{\frac{c_{w} - a n_w}{N + b}}_{\text{smoothed 1-gram}} + \underbrace{\frac{aK + b}{N + b} p_{\mathrm{gen}}(w)}_{\text{backoff for OOVs}}
$$
This framework ensures power-law type/token distributions and addresses the empirical estimator's over-/under-confidence.

## 4. Extensions and Neuralization

Neuralization of the unigram LM refers to replacing the character-level generator with modern RNNs (e.g., a 3-layer LSTM), yielding a fully differentiable model that alternates cluster assignment sampling (E-step: Gibbs for clusters) and generator updating (M-step: SGD on cross-entropy) [2106.02289]. This two-stage model learns to interpolate between the token-level and type-level distributions automatically, as controlled by the PYP hyperparameters.

Empirically, such models consistently outperform both token- and type-LSTM baselines in held-out cross-entropy across diverse languages, accurately handle the frequency spectrum (stratified by rank), and generalize to OOV types via the generator component.

## 5. Integration in Neural Language Generation and Pretraining

Neural language generators (e.g., sequence-to-sequence Transformers) inherently drift towards learning unigram distributions in early training, prior to acquiring semantic or syntactic knowledge [2212.09686]. Initializing the decoder’s output bias $b$ as $\log p_{\mathrm{unigram}}$ (i.e., $\log\,\frac{\mathrm{count}(w)}{N}$ for each $w$) encodes the empirical frequency prior directly in the softmax layer:
$$
b_i = \log \left( \frac{\mathrm{count}(w_i)}{N} \right)
$$
This initialization may accelerate convergence and enables the context encoder to specialize in nonfrequency aspects. Across machine translation benchmarks, initializations with this log-unigram prior yield an early BLEU AUC improvement of 2–5 points and small improvements in final BLEU scores [2212.09686]. The effect persists across languages and under various data regimes.

Ablations reveal that (1) the bias remains stable throughout training (KL divergence stays near zero), and (2) removing the bias at evaluation disentangles context from frequency, a property less evident in randomly-initialized models. A plausible implication is that neural decoders with unigram-initialized bias more effectively leverage model capacity for modeling token interdependencies.

## 6. Practical Implementations and Algorithmic Considerations

HITgram demonstrates that unigram language models can be efficiently implemented as hash maps for token count storage, with linear corpus scaling and incremental updates via merges [2412.10717]:
- Maximum tokenization throughput: 50,000 tokens/sec on standard hardware.
- Corpus management supports incremental learning, data replacement, and low-frequency pruning for memory conservation.
- Smoothing and context-sensitive weighting (e.g., $w(w) = \log(1+\mathrm{Count}(w))$) are supported, the latter stabilizing the influence of highly frequent tokens.

For practical applications—autocomplete, word clouds, pedagogical demos—the model’s simplicity and efficiency are key strengths. Limitations include the incapacity to model token order and syntactic dependencies, as well as scaling considerations for rapidly-growing vocabularies.

## 7. Empirical Comparison and Impact

The unigram LM tokenization method yields subword vocabularies with superior morphological alignment relative to BPE. Empirical results [2004.03720] show, on 20k-vocabulary settings:
- English (vs. gold-standard CELEX): Unigram LM F1 $30.3\%$, BPE F1 $19.3\%$.
- Japanese (vs. MeCab): Unigram LM F1 $77.2\%$, BPE F1 $73.8\%$.

Pretrained masked language models using unigram LM segmentation achieve equal or superior downstream performance (e.g., SQuAD 1.1 EM, MNLI-m, Japanese TyDi QA) compared to otherwise-identical BPE-based models.

Illustrative segmentations confirm that the unigram LM more frequently preserves morphological stems and affixes, avoiding the "dead-zone" intermediate tokens typical of BPE.

| Metric / Task        | BPE   | Unigram LM |
|----------------------|-------|------------|
| English CELEX F1     | 19.3% | 30.3%      |
| Japanese MeCab F1    | 73.8% | 77.2%      |
| SQuAD 1.1 EM         | 80.6  | 81.8       |
| Japanese TyDi QA F1  | 42.1  | 54.4       |

A plausible implication is that unigram LM approaches not only offer theoretical advantages in vocabularization but also translate into tangible improvements in LLM pretraining and downstream efficacy.

---

In summary, unigram language models are foundational in both theoretical and applied NLP, with relevance extending from simple empirical estimation to advanced neural inference and subword modeling strategies. They offer mathematically precise estimation, direct interpretability, compatibility with EM and Bayesian smoothing, and robust performance within broader neural and non-neural systems [2004.03720, 2412.10717, 2212.09686, 2106.02289].

Source: https://www.emergentmind.com/topics/unigram-language-models