--- title: Subword Units in Language Processing url: https://www.emergentmind.com/topics/subword-units type: topic --- # Subword Units in Language Processing A subword unit is a linguistic or orthographic sequence—shorter than a word but usually longer than a character—used as an atomic symbol in various statistical and neural language technologies. Subword units enable open-vocabulary modeling, efficient parameter sharing across related forms, and mitigation of sparsity and out-of-vocabulary (OOV) problems in speech and language processing. Subword vocabularies are typically constructed via unsupervised, linguistically motivated, or data-compression–oriented algorithms, such as Byte-Pair Encoding (BPE), unigram language models, or model-based morphological segmenters, and are now ubiquitous in state-of-the-art machine translation, speech recognition, and linguistic representation systems. ## 1. Theoretical Rationale and Types of Subword Units Subword units address the tradeoff between fully word-based and fully character-based modeling. Word units capture longer context but are limited by fixed vocabularies and heavy sparsity, while character units offer open-vocabulary coverage but greatly increase sequence length and impair the modeling of morphological/semantic information. Subword units—typically character n-grams, phone n-grams, frequent morphemes, or statistically-derived segments—strike an efficient compromise [1508.07909, 1704.08352, 1904.07994]. Three main types of subword units are prevalent: - **Character n-grams:** All overlapping n-length substrings of a word (e.g., “cats”: “cat”, “ats”), including the full word. - **Morph-based units:** Segments that correspond to actual morphemes or morphological boundaries, usually induced by models like Morfessor [2003.03131] or supervised with gold morphological annotations [1704.08352, 1904.07994]. - **Data-driven units (e.g., BPE, Unigram LM):** Variable-length segments optimized for frequency, compression, or segmentation likelihood. BPE merges the most frequent adjacent pairs; unigram models prune a large vocabulary via EM [1508.07909, 1804.10959, 2003.03131]. These units often do not align perfectly with true morphemes but yield superior performance in large-scale neural systems [1806.05482, 1807.04978, 1807.09639]. Subword vocabularies are typically constructed to ensure that every word can be fully segmented: all single characters are retained, guaranteeing a fallback for unseen words [1508.07909, 1712.06855]. ## 2. Algorithms and Segmentation Principles Two foundational algorithms for subword segmentation are BPE and the Unigram LM model. - **Byte-Pair Encoding (BPE):** Each word is decomposed into its sequence of characters, and frequent pairs are iteratively merged. The number of merges is a hyperparameter; typical vocabulary sizes are 1–100k. The merge procedure enables a flexible unit set from sub-character (singletons) to word-level units [1508.07909, 1712.06855, 1806.05482]. Extensions introduce context-sensitive or compression-driven merges, such as Accessor Variety (AV) and Description Length Gain (DLG), providing finer control and improved performance in morphologically complex languages [1807.09639]. - **Unigram Language Model (Unigram LM):** Begins with a large candidate subword vocabulary. The EM algorithm is used to estimate the probability for each subword, and low-probability items are pruned to a target size. SentencePiece [2203.02632] and Morfessor EM+Prune [2003.03131] are prominent implementations, the latter incorporating MDL or Bayesian priors [1804.10959]. The Unigram LM can probabilistically sample multiple valid segmentations per word, enabling subword regularization [1804.10959, 2008.04034]. Linguistically motivated algorithms, such as Morfessor and semi-CRF segmenters, induce morph boundaries based on likelihood or gold labels but are frequently outperformed by data-driven methods in neural systems [1806.05482, 1704.08352, 1904.07994]. ## 3. Subword Units in Neural Architectures and Representation Learning In neural architectures, subword units are employed as the tokens over which all modeling—embedding, prediction, output—operates. - **Machine Translation (NMT):** Token sequences from BPE or Unigram segmentation replace word tokens as the input and output of neural encoder/decoder models, enabling open vocabulary translation via composition and facilitating parameter sharing for rare word forms [1508.07909, 1804.10959, 1807.09639, 2005.06606]. Subword units have been shown to yield BLEU improvements of 0.5–2 points over word-based or character-based models, especially on OOV and rare words [1508.07909, 1804.10959]. - **Speech Recognition (ASR):** Subword units (characters, grapheme n-grams, or phone-based BPE units) serve as the output labels of CTC, RNN-T, or attention-based models. Appropriately chosen subword vocabularies enable both efficient modeling and high OOV word recall, especially in morphologically rich languages [1807.04978, 2004.04290, 2005.13827, 2104.09106]. Phone-based or acoustically matched subwords often outperform character-based subwords for ASR [2004.04290, 2104.09106]. - **Word Representation Learning:** Word vectors are constructed as (typically normalized) sums of the embeddings of their composing subword units—character n-grams, BPE units, or morphemes [1904.07994, 1808.09500]. Positional embeddings and attention-based composition functions provide additional capacity for capturing sequential and morphological information [1904.07994]. - **Named Entity Recognition and Classification:** Neural NER models operationalize subword sequences (e.g., char or phone LSTMs) to encode morphological and phonological cues for improved OOV and low-frequency entity tagging [1808.07364]. ## 4. Subword Regularization and Ambiguity Mapping text to subword units is inherently ambiguous, with many valid segmentations for a given surface form. Subword regularization is the policy of explicitly sampling and training on multiple segmented variants, leveraging this ambiguity to regularize the model and increase its robustness to OOVs and domain shifts [1804.10959, 2008.04034]. - **Mathematics:** For Unigram LM segmenters, the segmentation probability is \( p(y | w) \propto \prod_{i=1}^M p(y_i) \), with the best segmentation obtained via dynamic programming (Viterbi), but training can sample alternative segmentations using a temperature parameter α and N-best list [2008.04034]. - **Effects:** Empirical studies report 2–8% relative WER reduction in ASR [2008.04034], and 0.5–2.0 BLEU improvement in NMT, especially with low-resource or out-of-domain data [1804.10959]. Unseen-word recognition recall is significantly improved by regularization [2008.04034]. Regularization is realized in practice by sampling a segmentation for each training example on-the-fly, using a smoothed selection from N-best segmentations, and training the model to minimize the expected loss over these segmentations [1804.10959, 2008.04034]. ## 5. Empirical Evaluations, Task-Specific Recommendations, and Limitations Empirical studies establish key best practices and identify important tradeoffs: - **ASR (Keyword/OOV):** Single-character subwords achieve highest OOV recall and maximum term-weighted value in OOV keyword search for Arabic and Finnish [2005.13827]. Interpolating n-gram and RNNLM-derived subword LMs, with variable-order n-gram approximation, yields further improvements [2005.13827]. Acoustically-driven subword discovery better aligns segmentation with phonemic transitions and yields lower WER than BPE in time-synchronous models [2104.09106]. - **NMT:** BPE with a "zero-suffix" marker on non-final tokens and shared source-target vocabulary recovers most of the BLEU degradation relative to more complex segmenters in morphologically rich pairs [1806.05482]. Subword regularization via Unigram LM consistently adds 0.5–1.5 BLEU, especially in low-resource and OOD conditions [1804.10959]. - **Representation Learning and Transfer:** Combining character n-grams, morphemes, and phone n-grams in compositional embeddings substantially improves NER F1 and rare-word transfer, notably in low-resource settings [1808.09500, 1904.07994]. - **Morphology:** Subword units based on data-compression–motivated principles (BPE/Unigram) only weakly approximate true morphological boundaries and rarely outperform oracle morphological analysis for language modeling [1704.08352, 1904.07994]. However, linguistically motivated segmenters show their advantages in rare word and semantic similarity tasks [1904.07994]. - **Unit Vocabulary Size:** Optimal vocabulary size balances sequence length and data sparsity; too large yields many singleton units, too small increases sequence and model size, degrading performance [1508.07909, 1712.06855]. Envelope values are task- and data-size–dependent; tuning is always recommended [1807.09639, 1806.05482, 1712.06855]. Limitations arise from over-segmentation (overly fine-grained, redundant units), language-specific orthographic or phonological alternations unrecoverable by frequency-driven merges, and duplication of hypotheses in beam search under probabilistic segmentation [1804.10959, 2008.04034, 1806.05482]. ## 6. Extensions and Emerging Paradigms Recent advances include: - **Dynamic Programming Encoding (DPE):** Treats segmentation as a latent variable; marginalizes all valid segmentations during training using a mixed character–subword transformer, yielding further BLEU gains on top of BPE and regularization [2005.06606]. - **Acoustic Data-Driven Subword Modeling (ADSM):** Aligns subword discovery directly to acoustic boundaries by iterating supervised alignment, vocabulary refinement, and subword merging; achieves state-of-the-art WER and robust coverage in both time-synchronous and label-synchronous ASR [2104.09106]. - **Visually Grounded Discovery:** CNNs trained on speech–image paired data discover interpretable diphone events at intermediate layers, which tightly align with phone boundaries without any supervision, suggesting novel unsupervised pathways for subword discovery [1902.08213]. Hybrid models (e.g., joint phone-BPE and char-BPE systems in ASR) and hierarchical composition for word representations offer further robustness and phonological or script generalization [2004.04290, 1904.07994]. --- **References:** - [1508.07909]: Neural Machine Translation of Rare Words with Subword Units - [1704.08352]: From Characters to Words to in Between: Do We Capture Morphology? - [1712.06855]: Subword and Crossword Units for CTC Acoustic Models - [1804.10959]: Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates - [1806.05482]: Morphological and Language-Agnostic Word Segmentation for NMT - [1807.04978]: Hybrid CTC-Attention based End-to-End Speech Recognition using Subword Units - [1807.09639]: Finding Better Subword Segmentation for Neural Machine Translation - [1808.07364]: Neural Named Entity Recognition from Subword Units - [1808.09500]: Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations - [1902.08213]: Towards Visually Grounded Sub-Word Speech Unit Discovery - [1904.07994]: A Systematic Study of Leveraging Subword Information for Learning Word Representations - [2003.03131]: Morfessor EM+Prune: Improved Subword Segmentation with Expectation Maximization and Pruning - [2004.04290]: An investigation of phone-based subword units for end-to-end speech recognition - [2005.06606]: Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation - [2005.13827]: Subword RNNLM Approximations for Out-Of-Vocabulary Keyword Search - [2007.06949]: Deep Transformer based Data Augmentation with Subword Units for Morphologically Rich Online ASR - [2008.04034]: Subword Regularization: An Analysis of Scalability and Generalization for End-to-End Automatic Speech Recognition - [2104.09106]: Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognition - [2203.02632]: Extracting linguistic speech patterns of Japanese fictional characters using subword units Source: https://www.emergentmind.com/topics/subword-units