---
title: SentencePiece Unigram Model
url: https://www.emergentmind.com/topics/sentencepiece-unigram-model
type: topic
---

# SentencePiece Unigram Model

The SentencePiece Unigram model is a probabilistic subword tokenization algorithm central to modern neural text processing. It estimates a probability distribution over candidate subword units and segments input text by maximizing marginal likelihood. Unlike greedy approaches such as Byte-Pair Encoding (BPE), the Unigram model jointly optimizes both the vocabulary and subword probabilities via an Expectation-Maximization (EM) paradigm, often coupled with iterative pruning. This framework yields robust tokenization efficacy in morphologically complex and low-resource languages and forms the foundation for extensible, language-independent tokenization systems such as the open-source SentencePiece library [1808.06226, 2512.12641].

## 1. Probabilistic Model Definition

The SentencePiece Unigram model formalizes subword segmentation as a latent variable problem. Let $V=\{s_1,\dots,s_{|V|}\}$ denote the candidate vocabulary of subword pieces, and $S(w)$ the set of all possible segmentations of string $w$ into sequences of pieces from $V$. Each segmentation $S=[s_{i_1},\dots,s_{i_K}]$ concatenates to $w$. The model assigns probabilities $P(s)$ to subwords with normalization $\sum_{s\in V}P(s)=1$.

The probability of a segmentation $S$ is:
\[
P(S)=\prod_{k=1}^K P(s_{i_k})
\]
The marginal probability of $w$ is obtained by summing over all segmentations:
\[
P(w)=\sum_{S\in S(w)}P(S)
\]
Training maximizes the corpus log-likelihood:
\[
L=\sum_{n=1}^N \log P(w_n)=\sum_{n=1}^N \log\sum_{S\in S(w_n)}\prod_{s\in S}P(s)
\]
This formulation allows for modeling complex segmentations under uncertainty and is more expressive than greedy, parse-based alternatives [1808.06226, 2512.12641, 2512.18399].

## 2. EM Training and Vocabulary Pruning

Direct maximization of $L$ is intractable due to exponentially many segmentations. SentencePiece employs EM:

- **E-step:** For each string $w_n$, construct a segmentation lattice where nodes index positions and arcs represent candidate subwords matching segments. Using forward–backward recurrences,
  - $\alpha_n(0)=1$
  - $\alpha_n(j)=\sum_{s\in V,\ s\text{ ends at }j}\alpha_n(j-|s|)\cdot P(s)$
  - $\beta_n$ is analogous for the backward pass.
Calculate expected counts for each piece $s$ via arc posteriors:
\[
r_n(s,i) = \frac{\alpha_n(i-|s|)\cdot P(s)\cdot \beta_n(i)}{P(w_n)}
\]
Summing $r_n(s,i)$ over all $n$ and matching $i$ yields $C(s)$.

- **M-step:** Update probabilities by normalizing:
\[
P(s)\leftarrow \frac{C(s)}{\sum_{t\in V}C(t)}
\]

- **Pruning:** After each M-step (or every few iterations), prune subwords with lowest expected impact on likelihood or smallest expected counts. Pruning can be via likelihood-drop heuristics or simply by top-probability thresholding [1808.06226, 2512.12641].

Iterative EM and pruning continue until $|V|$ meets the target vocabulary size. Empirical findings indicate that reducing EM sub-iterations, omitting the digamma transform, and applying Final-Style Pruning (FSP) can accelerate training with negligible quality loss [2512.12641].

## 3. Decoding and Subword Regularization

Given trained subword probabilities, tokenization of new text amounts to finding the most probable segmentation $S^*$:
\[
S^* = \arg\max_{S \in S(w)} P(S) = \arg\max_{S} \sum_{s \in S}\log P(s)
\]
This is efficiently solved via Viterbi dynamic programming on the segmentation lattice, with time complexity $O(M \cdot L)$ for an input of length $M$ and maximum piece length $L$. The model also supports subword regularization: segmentations can be sampled according to the marginal probability $P(S|w)$, optionally with temperature scaling, exposing downstream models to segmentation variability during training [1808.06226].

## 4. Practical Implementation and Parameterization

Key implementation steps:

- **Seed Vocabulary Construction:** Build $V$ from all substrings up to a threshold length occurring in the corpus. Pretoken-based seed extraction yields superior loss and token compression outcomes compared to suffix-array aggregation [2512.12641].
- **Parameters:** Empirical tuning of vocabulary size factor ($\gamma$), EM iterations ($k_{EM}$), pruning shrink factor ($\alpha$), early prune threshold, and overshoot ratio impacts convergence, compression, and likelihood minimally for most settings. FSP variant further simplifies training by keeping only the top-$n$ pieces, often matching or exceeding greedy BPE on compression [2512.12641].
- **OOV Handling:** Ensure that every Unicode character has coverage in $V$ to avoid mapping input to <unk>. Additive smoothing is unnecessary due to the pruning mechanism.
- **Integration:** The trained model consists of $V$, subword probabilities, and normalization FSTs, all stored in a self-contained format for reproducible deployment [1808.06226].

## 5. Extensions for Non-Latin and Morphologically Rich Languages

Morphologically rich and low-resource languages such as Arabic and Dzongkha benefit from the Unigram model’s global likelihood criterion, which better accommodates complex affixation and inflection than greedy merge-based techniques. Empirical results for Arabic (AraToken) with a comprehensive normalization pipeline yield fertility reduction from 1.35 (BPE) to 1.199 (SentencePiece normalized), and compression gains from 4.60 to 5.03 chars/token [2512.18399]. The normalization pipeline addresses Unicode decomposition, orthographic unification (e.g., Alif variants), numeral and punctuation mapping, tatweel removal, and diacritic handling. For Dzongkha, SentencePiece achieves optimal subword fertility (0.79), minimal proportion of continued words (0.09), and superior normalization compared to WordPiece and BPE [2509.15255].

The Language Extension Pipeline (LEP) integrates new vocabularies into existing models (Qwen3-0.6B), leveraging mean subtoken embedding initialization, gradient masking for old embeddings, and selective unfreezing of transformer layers. This enables efficient adaptation to new scripts with under 0.01% of pretraining cost, demonstrated by evaluation loss reduction from 8.28 to 2.43 after 800 steps on Arabic text [2512.18399].

## 6. Comparative Evaluation and Metrics

Performance assessment utilizes metrics tailored for subword quality:

| Metric                         | Ideal Value   | Definition                                                                                       |
|--------------------------------|--------------|--------------------------------------------------------------------------------------------------|
| Fertility                      | 1.0          | tokens/word; lower indicates better compression                                                  |
| Compression                    | high         | chars/token; higher is better                                                                    |
| Proportion of Continued Words  | 0.0          | fraction of words split into multiple tokens; lower implies less fragmentation                   |
| Normalized Sequence Length     | —            | sequence length vs. baseline; lower is more efficient                                            |
| Execution Time                 | —            | wall-clock runtime per input loop                                                                |

SentencePiece Unigram frequently achieves lowest fertility and highest compression in comparative studies. For Arabic with diacritic dropping, SentencePiece normalized yields fertility 1.199 and compression 5.03, outperforming BPE and WordPiece [2512.18399]. For Dzongkha, SentencePiece achieves fertility 0.79, continued word proportion 0.09, and fastest inference time (131 ms/loop) [2509.15255]. Trade-offs between compression and likelihood (e.g., FSP vs. standard pruning) are well characterized: FSP may yield 1.4% fewer tokens at 0.5–1.5% higher loss out-of-domain [2512.12641].

## 7. Implications and Limitations

The SentencePiece Unigram model provides a language-independent framework for subword tokenization with effective modeling of morphological phenomena and extensibility to non-Latin scripts. Its probabilistic EM-based inference avoids the over-fragmentation seen in greedy approaches and directly optimizes corpus likelihood. Language-specific normalization, targeted vocabulary pruning, and efficient integration strategies underpin superior performance in morphologically rich and low-resource contexts. Practical simplifications (e.g., FSP) yield substantial implementation efficiency without sacrificing quality.

A plausible implication is that the Unigram model's principled segmentation and robust pruning make it especially suitable for rapid extension of LLMs to new scripts or resource-scarce settings. However, its computational cost in training (especially EM iterations and likelihood calculations) may exceed that of BPE, though tokenization at inference remains efficient. Overall, the Unigram model is technically preferred where morphological alignment and compression are critical [2512.12641, 2512.18399, 2509.15255, 1808.06226].

Source: https://www.emergentmind.com/topics/sentencepiece-unigram-model