---
title: 'Unigram LM: Fundamentals & EM Training'
url: https://www.emergentmind.com/topics/unigram-language-model-unigram-lm
type: topic
---

# Unigram LM: Fundamentals & EM Training

A unigram language model (Unigram LM) is a probabilistic model that assigns a probability to a string based on the assumption that each token (typically a subword or word segment) is generated independently from a fixed vocabulary distribution. This non-contextual model plays a central role in modern subword tokenization algorithms—most notably in the Unigram LM tokenizer—which underpins many NLP pipelines for both training and inference, especially in neural language model pretraining. Crucially, the Unigram LM formalism, objective, and estimation procedure support morphology-sensitive vocabulary discovery and robust segmentation, setting it apart from purely heuristic algorithms such as Byte Pair Encoding (BPE) [2004.03720, 2512.12641].

## 1. Probabilistic Formulation and Objective

The Unigram LM defines a probability distribution $\theta$ over a vocabulary $V$ of subword types. Given a segmentation $w_{1},\ldots,w_{n}$ of a text, the joint probability is

$$
p(w_1, \ldots, w_n \mid \theta) = \prod_{i=1}^n \theta_{w_i}
$$

where $\theta_{t} \geq 0$ for $t \in V$ and $\sum_{t \in V} \theta_{t} = 1$ [2004.03720, 2512.12641]. The model's objective is to maximize the likelihood of an unlabeled corpus $D$:

$$
\theta^* = \arg\max_\theta \sum_{x \in D} \log p(x \mid \theta)
$$

where

$$
p(x \mid \theta) = \sum_{S \in \mathcal{S}(x)} \prod_{t \in S} \theta_{t}
$$

and $\mathcal{S}(x)$ denotes all valid segmentations of $x$ into tokens in $V$ [2512.12641].

At decoding time, segmentation of an unseen string $x$ seeks the covering $S^*$ that maximizes $p(S\mid \theta)$, tractably computed via Viterbi-style dynamic programming [2004.03720].

## 2. Expectation–Maximization Training and Vocabulary Pruning

Direct maximization of the Unigram LM objective is intractable due to the exponential number of segmentations. Parameter estimation proceeds by Expectation–Maximization (EM):

- **E-Step**: Compute the expected count $C_{t}$ for each token $t$ over all segmentations weighted by their posterior probability under current $\theta^{\text{old}}$.
- **M-Step**: Update $\theta_{t}^{\text{new}} = C_{t} / \sum_{u \in

Source: https://www.emergentmind.com/topics/unigram-language-model-unigram-lm