---
title: Unigram Tokenization Algorithm
url: https://www.emergentmind.com/topics/unigram-tokenization-algorithm
type: topic
---

# Unigram Tokenization Algorithm

The Unigram Tokenization Algorithm, often referred to as "UnigramLM," is a probabilistic subword segmentation methodology utilized in neural text processing. Distinguished from deterministic or greedy tokenization schemes, UnigramLM underlies the widely adopted SentencePiece toolkit and has become a standard for language-agnostic, lossless, and robust preprocessing in neural systems, particularly in machine translation and large-scale language modeling [1808.06226][2512.12641].

## 1. Mathematical Framework and Objective

Let $V = \{x_1, ..., x_{|V|}\}$ denote the vocabulary of subword tokens. For an input string $X$, a segmentation is a sequence $x = (x_{i_1}, ..., x_{i_k})$ of tokens from $V$ such that their concatenation exactly reconstructs $X$. The UnigramLM makes a strong conditional independence assumption over subwords, leading to the segmentation probability:

$$
P(x) = \prod_{j=1}^k p(x_{i_j})
$$

where each $p(x_i) > 0$ and the probabilities sum to one: $\sum_{i=1}^{|V|} p(x_i) = 1$. 

Unlike approaches that optimize token sequence likelihoods directly, UnigramLM treats the segmentation as latent—optimizing for the marginal likelihood over all valid segmentations $x \in S(X)$:

$$
L(V, p) = - \frac{1}{\sum_{X \in C} |X|} \sum_{X \in C} \log \left[ \sum_{x \in S(X)} P(x) \right]
$$

where $C$ is the corpus. This negative average per-byte log-likelihood is minimized by adjusting both $V$ and $p$ [2512.12641].

## 2. EM-Based Training and Vocabulary Pruning

UnigramLM training proceeds by initial "overgeneration" of the vocabulary through extracting frequent substrings (seed vocabulary $V^0$), followed by iterative pruning while fitting token probabilities using the Expectation-Maximization (EM) algorithm. SentencePiece and subsequent references formalize the process as follows:

- **Seed Vocabulary Construction:** $|V^0| = \psi \cdot n$ (typical seed_ratio $\psi = 10$), where $n$ is the target final vocabulary size. Suffix array and longest common prefix (LCP) interval algorithms efficiently generate substrings up to a maximal length [2512.12641].
- **EM Fitting (per pruning iteration):**
    - **E-Step:** For each sentence, the expected counts of each token are computed by forward–backward dynamic programming over all segmentation paths.
    - **Early Pruning:** Tokens with expected count below threshold $\tau_e$ are removed ($\tau_e = 0.5$ effective but robust to changes).
    - **M-Step:** Probabilities are updated as $p(x_i) \gets \psi(c_i) / \sum_{j \in V}\psi(c_j)$, with $\psi(c)$ often the digamma function but can default to identity without material loss ($<$0.01\% effect on loss).
- **Pruning:** Tokens are ranked by their estimated loss increase if removed (computed by retokenizing the corpus without the candidate token). Top $\alpha \cdot |V|$ ($\alpha = 0.75$) are retained, and the process repeats until the vocabulary size reaches the pre-final overshoot factor $\beta \cdot n$ ($\beta = 1.1$). Final pruning is by probability.

The method scales linearly with corpus size and achieves practical runtimes on large datasets [2512.12641][1808.06226]. 

## 3. Algorithm Variants: Final-Style Pruning

Land & Pinter (2025) introduced the "Final-Style Pruning" (FSP) heuristic variant, which replaces loss-based iterative pruning with repeated pruning by lowest token probability:

1. Construct seed vocabulary and perform EM fitting as in standard UnigramLM.
2. After each EM block, remove the lowest-$p(x_i)$ tokens to reduce $|V|$ by factor $\alpha$.
3. Repeat until $|V| \leq \beta \cdot n$, then do one final prune by $p(x)$.

Empirical evaluation shows that FSP modestly increases the negative log-likelihood ($+$0.6–1.2% relative) but gains $-$0.5–1.3% in compression (i.e., produces fewer tokens for the same data), with morphological alignment metrics between standard Unigram and BPE [2512.12641]. 

## 4. Key Hyperparameters and Their Effects

Extensive evaluation has determined best practices for robust training, summarized as follows:

| Parameter        | Default      | Observed Effect                                                        |
|------------------|--------------|------------------------------------------------------------------------|
| seed_ratio $\psi$| 10           | $\psi < 4$ degrades both loss and compression; $\psi \geq 10$ is safe  |
| em_steps $\tau$  | 2            | Invariant to variation (1–5)                                           |
| early_thr $\tau_e$ | 0.5        | No consistent loss/compression effect across 0–10                      |
| prune_ratio $\alpha$ | 0.75    | Higher $\alpha$ slows pruning, lowers loss but increases token count    |
| pre_final $\beta$   | 1.1       | Lower $\beta$ stops pruning earlier, minimal impact                    |
| digamma on/off      | on         | Negligible change ($<$0.01\%)                                          |

Recommended defaults are robust for most settings [2512.12641]. Seed vocabularies must be at least $8 \cdot n$ in size for well-posed optimization; inadequate seeding materially harms final compression [2512.12641].

## 5. Integration in SentencePiece and Practical Usage

SentencePiece provides reference implementations and command-line tools as well as C++/Python APIs. End-to-end pipelines—including Unicode NFKC normalization (optionally extensible via FSTs), id-mapping, encoding/decoding, and lossless detokenization—are supported for both BPE and UnigramLM paradigms [1808.06226].

### Typical Usage

- **Training:**  
    ```bash
    spm_train \
      --input=corpus.txt \
      --model_prefix=spm_unigram \
      --model_type=unigram \
      --vocab_size=32000 \
      --character_coverage=1.0 \
      --seed_sentencepiece_size=320000
    ```
- **Encoding/Decoding (Python):**
    ```python
    import sentencepiece as spm
    sp = spm.SentencePieceProcessor()
    sp.Load('spm_unigram.model')
    tokens = sp.EncodeAsPieces('Sample text.')
    ids = sp.EncodeAsIds('Sample text.')
    text = sp.DecodeIds(ids)
    ```

SentencePiece operates losslessly, preserving spacing via Unicode marker U+2581 ("_"). No pre-tokenization is required; whitespace and rare Unicode symbols are preserved using the `character_coverage` setting [1808.06226][2512.12641].

## 6. Empirical Properties and Impact

Empirical evaluation on neural machine translation benchmarks (e.g., English ↔ Japanese) demonstrates that UnigramLM with SentencePiece delivers BLEU improvements of 1–1.5 points over word-level baselines at a fraction of vocabulary size. Training directly on raw sentences, without external pretokenization or language-specific rules, yields results as good as or superior to pipelines with hand-crafted segmentation [1808.06226].

SentencePiece's raw-Japanese tokenization is approximately 380× faster than word-level pre-tokenization approaches, with throughput exceeding 20,000 sentences per second on modern CPUs, suitable for on-the-fly inference [1808.06226].

## 7. Limitations and Implementational Considerations

The principal limitation is algorithmic complexity: standard UnigramLM requires EM inference and loss-based pruning, implemented efficiently only in select systems like SentencePiece. Corpus duplication alters early pruning, and rare Unicode drop-out can occur if `character_coverage` is set below 1.0 [2512.12641]. Final-Style Pruning is not yet exposed as a SentencePiece command-line option, requiring manual modification for experimental use [2512.12641].

A plausible implication is that while UnigramLM achieves rigorous marginal likelihood objectives and adaptivity across languages, production systems may trade off exact objective minimization for engineering simplicity and greater compression using FSP or similar heuristics.

---

**References:**  
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, 1808.06226]
- [Which Pieces Does Unigram Tokenization Really Need?, 2512.12641]

Source: https://www.emergentmind.com/topics/unigram-tokenization-algorithm