Morphologically-Aware Tokenization

• Morphologically-aware tokenization is a technique that segments text using unsupervised morphological models like Morfessor to improve morpheme boundary precision.
• It leverages algebraically derived subword embeddings and dynamic programming to optimize token alignment with word semantics.
• Efficiency is enhanced by distilling complex segmentation into bigram models, yielding better part-of-speech tagging and tractable inference.

Morphologically-aware tokenization is the family of techniques that segment text into tokens by explicitly incorporating information about morphologically meaningful units such as stems, prefixes, and suffixes, in contrast to purely frequency-driven subword methods such as Byte Pair Encoding (BPE) and Unigram LM. Recent research demonstrates that integrating morphological structure into tokenization workflows can markedly improve segmentation plausibility, enhance language modeling for tasks sensitive to morphology, and elevate efficiency metrics such as token boundary alignment and Rényi efficiency—though not always with clear improvements in neural translation quality. This article surveys the principal innovations and empirical findings in lexically grounded and morphologically-informed tokenization, focusing on unsupervised morphological pre-tokenization, embedding-based segmentation, and distilled statistical models (Libovický et al., 2024).

1. Unsupervised Morphological Pre-tokenization

The initial phase introduces Morfessor, an unsupervised morphological analyzer, as a pre-tokenization stage before subword vocabulary induction. Morfessor learns a lexicon of morph types and a segmentation S of a corpus D that minimizes a Minimum Description Length (MDL) objective: $$J(\mathcal{L}, S) = -\log P(\mathcal{L}) - \log P(S \mid \mathcal{L})$$ where $\mathcal{L}$ is the morph lexicon and $P(S \mid \mathcal{L})$ is the probability of the corpus given segmentation into morphs. This approach ensures that morphologically plausible boundaries are preserved, preventing subsequent subword vocabulary learning (BPE or Unigram) from violating true morpheme splits. The result is a hybrid pipeline: raw text $\rightarrow$ Morfessor morphs $\rightarrow$ subword segmentation (Libovický et al., 2024).

2. Algebraic Derivation of Subword Embeddings

A central innovation is the derivation of subword (substring) embeddings grounded in the pretrained word embedding space. Formally, given a pretrained word embedding matrix $E \in \mathbb{R}^{|V| \times d}$, output embeddings $W \in \mathbb{R}^{d \times |V|}$, and a word co-occurrence matrix $C$, the skip-gram model is: $\textrm{softmax}(E W) \approx \textrm{norm}(C)$ Candidate subword embeddings $E_s$ for substrings $S$ are computed by extending co-occurrence statistics (with a segmentation incidence matrix $A$) and solving

$E_s = \log (\textrm{norm}(A C)) W^\dagger$

where $W^\dagger$ is the right pseudo-inverse of $W$. Thus, substrings obtain vector representations directly comparable to words, enabling similarity-based scoring for downstream tokenization (Libovický et al., 2024).

3. Lexically Grounded Subword Segmentation

Utilizing the subword embeddings $E_s$, segmentation becomes a word-specific scoring problem: $$\textrm{Score}(x \rightarrow s_1...s_n) = \sum_{i=1}^{n} \textrm{cos}(E(x), E_s(s_i)) - \alpha n$$ with $\textrm{cos}(\cdot,\cdot)$ as cosine similarity, and $\alpha > 0$ penalizing longer segmentations. This scoring function is efficiently maximized via dynamic programming, identifying the segmentation that maximizes alignment between subwords and their containing word’s semantics. An iterative refinement alternates between computing segmentations and updating the subword candidate set, shrinking vocabulary toward more meaningful substrings (Libovický et al., 2024).

4. Efficient Subword Bigram Model

To offset inference cost, the embedding-based segmentations are distilled into a bigram statistical model $p(s_i|s_{i-1})$, estimated from corpus counts with Laplace smoothing: $$p(s_i|s_{i-1}) = \frac{\textrm{count}(s_{i-1},s_i) + 1}{\textrm{count}(s_{i-1}) + |S|}$$ Decoding is performed left-to-right using beam search, making the method tractable for deployment at scale. This model nearly replicates the embedding-based splits but with far lower computational overhead, making lexically grounded tokenization applicable for large-scale inference (Libovický et al., 2024).

5. Evaluation Metrics and Empirical Results

The intrinsic evaluation focuses on morpheme boundary precision, recall, and F1—comparing predicted token boundary sets to gold morphological boundaries—and on Rényi efficiency, quantifying the effective use of the subword vocabulary: $$\textrm{Precision} = \frac{|P \cap G|}{|P|} \qquad \textrm{Recall} = \frac{|P \cap G|}{|G|}$$

$$H_\alpha(p) = \frac{1}{1-\alpha} \log\left(\sum_i p_i^\alpha\right)$$

Experiments on the SIGMORPHON 2018 dataset (8 languages, 32k vocabulary) demonstrate that Morfessor pre-tokenization consistently improves boundary precision by 4–8 percentage points over word-level BPE/Unigram, with further gains from embedding-based approaches and negligible loss when transitioning to bigram-based inference. In downstream part-of-speech tagging, morphologically plausible splits from Morfessor and embedding/bigram segmentation lead to +0.5–0.7 percentage point improvements in accuracy over standard subword models, indicating a substantial benefit for morphologically sensitive tasks. Machine translation quality, measured by chrF and BLEU, is less affected, with improvements limited to ±1 point (Libovický et al., 2024).

6. Practical Implications and Significance

Lexically grounded morphologically-aware tokenization directly addresses the semantic and grammatical fragmentation inherent in frequency-based statistical segmentation. By aligning token boundaries to meaningful morphs and constructing subword vocabularies and embeddings with explicit reference to lexical structure, these methods improve morphological plausibility, increase token distribution efficiency, and yield better part-of-speech tagging—while maintaining tractable inference via distilled statistical models. Although gains in machine translation are modest, the methodology offers a linguistically principled foundation for tasks requiring fine-grained morphological competence, and demonstrates the value of integrating unsupervised morphological analysis and semantic information into the tokenization process (Libovický et al., 2024).