---
title: Bottom-Up Tokenisation Algorithms
url: https://www.emergentmind.com/topics/bottom-up-tokenisation
type: topic
---

# Bottom-Up Tokenisation Algorithms

Bottom-up tokenisation refers to a class of algorithms that construct token vocabularies by starting from atomic units (typically characters or bytes) and iteratively merging adjacent units to form larger “subword” tokens. Unlike traditional top-down or rule-based tokenisers, which operate over fixed wordlists or linguistic heuristics, bottom-up approaches are entirely data-driven and produce open-vocabulary segmentations, enabling robust handling of rare words, neologisms, and morphologically complex languages [2112.10508, 2506.17789].

## 1. Theoretical Foundations and Formal Definition

Bottom-up tokenisation is formally framed as the process of applying a sequence of pairwise merge operations to a dataset $D=\{w_1,\ldots,w_N\}\subset\Sigma^*$, where $\Sigma$ is a finite alphabet. Each merge $m=\langle s_1, s_2\rangle$ consists of concatenating adjacent subwords $s_1$ and $s_2$ into a new symbol $s_1\circ s_2$. A merge sequence $M=[m_1,\ldots, m_\delta]$ induces a vocabulary $V(M)=\Sigma\cup\{s_1\circ s_2:\langle s_1,s_2\rangle\in M\}$ with $|V(M)|=|\Sigma|+\delta$. The tokenisation of a string $w$ is $tok^\uparrow_M(w)=(merge_{m_\delta}\circ \cdots\circ merge_{m_1})(w)$.

The canonical objective is to minimise the total output length,
$$
L(D;M) = \sum_{w\in D} |tok^\uparrow_M(w)|
$$
under a budget of at most $\delta$ merges (i.e., fixed vocabulary size). The central decision problem is, given $(D, \delta, K)$, does there exist a merge sequence $M$ of length $\leq\delta$ such that $L(D;M)\leq K$? This is the “bottom-up tokenisation” decision problem [2412.15210, 2511.15709].

## 2. Algorithmic Instantiations: Byte-Pair Encoding, WordPiece, LiB, and Variants

The most widely used bottom-up tokenisation algorithm is Byte-Pair Encoding (BPE), which iteratively selects the most frequent adjacent symbol pair in the corpus and merges it into a new token. Each merge expands the vocabulary by one and the process repeats until the desired vocabulary size is reached. The segmentation process employs a longest-match-first strategy over the constructed vocabulary [2112.10508, 2506.17789].

**WordPiece** modifies BPE by selecting merges that maximise an n-gram language model likelihood increase rather than pure frequency, aligning merges more closely with statistical regularities [2112.10508].

**Less-is-Better (LiB)** tokenisation, a cognitive-scientifically motivated extension, alternates between a memoriser phase (adding n-gram units that yield net token savings exceeding the type penalty) and a forgetter phase (removing units whose presence increases the total cost). The cost function to be minimised is $C(V) = \alpha T(D,V) + \beta M(V)$, where $T(D,V)$ and $M(V)$ are the number of tokens and types, respectively. Empirically, LiB yields compression and vocabulary balance improvements over BPE [2403.00417].

**Partition-cover** approaches, such as GreedTok, reformulate bottom-up tokenisation as maximum coverage set function optimisation. Here, the goal is to select a vocabulary $S$ of size at most $k$ maximising $F(S)=\sum_{W} c_W\; \mathrm{cover}(W,S)$, i.e., covering as many adjacent pairs as possible with non-overlapping substrings/tokens [2501.06246].

## 3. Computational Complexity and Hardness Results

Bottom-up tokenisation is computationally intractable in the worst case. Both “Tokenisation is NP-Complete” [2412.15210] and “Tokenisation over Bounded Alphabets is Hard” [2511.15709] rigorously prove that, even under restrictive conditions (e.g., $\Sigma$ binary and merges limited), the problem is NP-complete and does not admit a polynomial-time approximation scheme unless P=NP. The proofs employ reductions from MAX-2-SAT and VERTEX COVER; for the partition-cover variant, NP-hardness is shown directly via a VERTEX COVER reduction [2501.06246].

Moreover, even approximating the minimal token sequence within a factor of $1+\epsilon$ for any constant $\epsilon>0$ is NP-hard—a fundamental barrier that explains why all practical bottom-up tokenisation algorithms are heuristic and greedy [2412.15210, 2511.15709, 2501.06246]. In special cases where the merge budget or alphabet is fixed and small, brute-force enumeration is possible, but this is of theoretical rather than practical significance [2412.15210].

## 4. Empirical Performance and Cross-Linguistic Considerations

Empirically, merge-based tokenisers such as BPE and its variants show consistently strong performance in language modeling and translation tasks across a variety of languages—particularly for those with high morphological complexity or limited annotated data [2112.10508, 2506.17789]. Bottom-up tokenisation yields open vocabularies, allowing any out-of-vocabulary sequence to be decomposed to the atomic level without mapping to “unk.”

Comparative studies across 17 Indian languages reveal nuanced trade-offs: at small vocabulary sizes, Unigram LM (a top-down probabilistic model) slightly outperforms BPE in aligning with morpheme boundaries and minimising word fragmentation, particularly in highly inflected languages. However, as vocabulary size increases (128K+), BPE closes this gap and can even surpass Unigram LM on fertility and word fragmentation rate. Larger vocabularies reliably yield fewer tokens per word and increased subword length, decreasing the sequence length required by the language model but increasing memory and compute cost in the embedding and output layers [2506.17789].

In multilingual settings, bottom-up tokenisation on pooled corpora typically biases vocabulary construction toward high-resource languages. Cluster-based or language-adaptive vocabulary strategies mitigate this effect, improving segmentation fairness and downstream performance for low-resource languages. Zero-shot transfer of tokenisers from high-resource to low-resource related languages is empirically viable, with reasonable fertility and compression retained [2506.17789].

## 5. Byte-Level and Partition Approaches

Byte-level bottom-up tokenisation, as instantiated by UTF8Tokenizer, bypasses traditional subword merging entirely: each UTF-8 byte is treated as its own atomic token, mapping directly to its numerical value in $[0,255]$. No out-of-range or auxiliary IDs are used. C0 ASCII control bytes (0x00–0x1F, 0x7F) serve as unambiguous markers for special functions (padding, sequence boundaries, segment demarcation). The embedding is a simple $256 \times d$ matrix, trivial to share across models. Enhancements such as bit-biased embeddings exploit internal byte structure at training without increasing inference cost. This approach yields 14× faster tokenisation and 8× lower host-to-device memory requirements than standard BPE-style methods, and yields modest improvements in language modeling convergence [2510.16987].

Partition-cover approaches, typified by GreedTok, select tokens (substrings) to maximise non-overlapping coverage under a size constraint. GreedTok empirically achieves 3–5% better compression than BPE at comparable vocabulary sizes and approaches the theoretical $(1 - 1/e)$ maximum coverage approximation ratio. The selected token sets diverge substantially from those of BPE beyond the first few hundred merges, reflecting the restrictive local optimality of frequency-based merging [2501.06246].

| Algorithm     | Greedy/Heuristic | Objective Optimised          | Notable Empirical Result                       |
|---------------|------------------|-----------------------------|------------------------------------------------|
| BPE           | Yes              | Most frequent pair merging  | 30k–50k merges optimal for English [2112.10508]|
| LiB           | Yes              | $\alpha T + \beta M$        | Lowest bits-per-character on CTB8/BRphon [2403.00417]|
| GreedTok      | Yes              | Partition-cover set function| 3–5% fewer tokens/word vs. BPE [2501.06246]    |
| UTF8Tokenizer | Not applicable   | None—bytes are tokens       | 14× speedup, 8× less memory [2510.16987]       |

## 6. Practical Implications and Best Practices

The computational intractability of optimal bottom-up tokenisation necessitates empirical, greedy, or approximation-based solutions. BPE, LiB, partition-cover relaxations, and other heuristic models all operate under this constraint. For LLM pretraining, partition-cover approaches such as GreedTok provide measurable reductions in token counts and vocabulary sizes, while also supporting integration of domain- or language-specific substrings. Bottom-up frameworks naturally support open vocabulary, straightforward handling of neologisms, and compositional robustness.

Practitioners should tune vocabulary size to the morphological properties of the target language; e.g., 30–50k merges suffice for English, while lower sizes yield better generalisation in polysynthetic languages. For multilingual applications, cluster-based or language-adaptive variants are recommended. In non-Latin scripts lacking explicit word boundaries, cognitively motivated models such as LiB efficiently recover “implicit” word and multiword units without additional annotation [2403.00417, 2506.17789].

Pure byte-level tokenisation (e.g., UTF8Tokenizer) is particularly suited for hardware-constrained or streaming scenarios, achieving maximal speed and minimal memory cost at the expense of longer sequences—an acceptable trade-off for many robust LLMs [2510.16987].

## 7. Connections to Compression and Morphological Segmentation

Bottom-up tokenisation is deeply connected to dictionary-based and grammar-based compression, as well as unsupervised morphological segmentation. The reduction of the tokenisation problem to straight-line programs (context-free grammars in Chomsky normal form) and set cover/maximum coverage instantiates these links formally [2412.15210, 2501.06246]. Bottom-up tokenisers can be seen as greedy grammar compressors under a rule-budget. Empirical segmentation quality, however, depends on the linguistic fit—pure unsupervised morph-based segmentation (e.g., Morfessor) can yield linguistically coherent units but may not improve downstream model performance over simpler BPE-style merges [2112.10508].

A plausible implication is that advances in text compression algorithms and submodular optimisation could inform the development of more effective approximation strategies for bottom-up tokenisation, within the provable performance and efficiency limits delineated by the NP-completeness and inapproximability results.

Source: https://www.emergentmind.com/topics/bottom-up-tokenisation