Subword Learning Dynamics

• Subword learning dynamics is the process where neural models optimize subword segmentations and representations during training, enhancing linguistic generalization.
• Dynamic frameworks like SSLM use joint training and efficient marginalization to balance morphological alignment with flexible tokenization strategies.
• Techniques such as BPE-dropout introduce controlled noise to boost model robustness and translation performance, particularly in low-resource and morphologically rich languages.

Subword learning dynamics refer to the temporal and structural evolution of subword segmentations, representations, and compositionality within neural LLMs—especially during training when the subword vocabulary and segment boundaries are themselves subject to optimization. The field covers both the explicit learning of subword segmentation (such as dynamic tokenization during pretraining and finetuning) and the implicit compositional processes by which LLMs internalize, manipulate, and utilize subword information for robust linguistic and semantic generalization. This dynamic is of particular consequence in morphologically diverse and low-resource language settings, as well as for the generalization, robustness, and compositional coverage of LLMs and word representations.

1. Frameworks for Subword Learning and Segmentation

Traditionally, subword segmentation is fixed at preprocessing using heuristics such as Byte Pair Encoding (BPE) or unigram-based methods, resulting in a static vocabulary and deterministic segmentation procedure. However, dynamic subword learning frameworks such as the Subword Segmental LLM (SSLM) allow the segmentation to be optimized during LLM training itself (Meyer et al., 12 Nov 2025). Classical LMs condition sequence modeling on a single tokenization $T$:

$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$

whereas SSLMs marginalize across all feasible segmentations $\mathcal{T} \in \pi(D)$:

$$p_\theta(D) = \sum_{\mathcal{T} \in \pi(D)} p_\theta(D, \mathcal{T})$$

Efficient O$(|D|K)$ computation of this marginalization is enabled by dynamic programming, with $\alpha_k$ representing the cumulative marginal likelihood up to character position $k$:

$$\alpha_0 = 1 \qquad \alpha_k = \sum_{j=f(c,k)}^k \alpha_{j-1} \cdot p(t = c_{j:k} \mid c_{<j})$$

where $f(c, k)$ prevents illegal subword spans. At each checkpoint, the most likely segmentation (Viterbi decoding) $T^* = \arg\max_{T \in \pi(D)} p_\theta(D, T)$ exposes the evolving inventory of subwords.

In the Subword Information Skip-Gram (sisg) model (Bojanowski et al., 2016), each word $p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$0 is represented as the sum of its character $p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$1-gram embeddings $p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$2:

$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$3

This construction allows efficient parameter sharing, accelerated learning for rare words, and robust handling of out-of-vocabulary items.

2. Stages and Metrics of Subword Learning Dynamics

Empirical studies using SSLMs across morphologically diverse languages (Setswana, English, isiXhosa) reveal a multi-stage dynamic in subword learning (Meyer et al., 12 Nov 2025):

1. Discovery Phase (0–20%): Rapid restructuring of random boundaries to morpheme-aligned segmentations, with morphological boundary F$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$4 surging from near zero.
2. Over-Segmentation (20–40%): Temporary spike in subword fertility (average subwords per word), with increased recall but decreased precision of morphological boundaries.
3. Consolidation (40–90%): Plateauing of fertility (e.g., Setswana ≈1.65, English ≈1.8, isiXhosa drifting toward ≈3.0), flattening of productivity (generalizability of subwords) and idiosyncrasy (association with few high-frequency words).
4. Task-Specific Specialization (Finetuning): Segmentation becomes finer-grained, increasing fertility and improving adaptation to task-specific distributions (e.g., named entities becoming single characters).

Key metrics to quantify these dynamics include:

Metric	Description	Operationalization
Morphological Alignment	Overlap of subword boundaries with gold morpheme boundaries	Precision, recall, F$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$5
Productivity	Breadth of subword generalization	$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$6 (distinct words with subword $p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$7)
Idiosyncrasy	Frequency skew per subword	$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$8
Fertility	Mean subwords per word	$p_\theta(D, T) = \prod_i p_\theta(t_i \mid t_{<i})$9 indicates finer segmentation

All three metrics are tightly coupled; early pruning of rare subwords accelerates morphological alignment, reduces oversegmentation, and enhances the compositional coverage relevant for downstream tasks (Meyer et al., 12 Nov 2025, Bojanowski et al., 2016).

3. Subword Representations and Compositionality in Neural Models

The compositional processes by which LLMs construct word-level and higher-level meaning from subwords exhibit significant variation across architectures and training regimens (Peng et al., 25 Aug 2025). An analysis of six popular 7–9B parameter LLMs reveals three broad strategies for subword composition:

• Isometric Adders (Aya-expanse, Gemma2): Maintain a nearly linear sum-of-parts subspace from embeddings to output; structural similarity between composed and holistic word representations remains high (Precision@1 ≈70–80%).
• Abstractors with Late Re-Introduction (Falcon, Qwen2.5): Early subword structure retention, mid-network abstraction, later re-injection of form cues—manifested as U-shaped or partially recovering P@1 across layers.
• Immediate Abstractions (Llama3, Llama3.1): Abrupt collapse of linear subword structure after embedding, favoring holistic or memorized representations; geometric alignment lost, though semantic decomposability persists.

Semantic content (e.g., root vs. non-root), as assessed by F$\mathcal{T} \in \pi(D)$0 of a logistic classifier, remains robustly encoded (F$\mathcal{T} \in \pi(D)$1 ≳80%) even when geometric alignment is compromised. Surface form retention (e.g., word length prediction) is strongest in isometric models, weakest in immediate abstraction models, and displays a characteristic dip-and-rebound across depth.

4. Effects of Subword Regularization and Noise

Fixed segmentation methods such as standard BPE tend to yield a single deterministic segmentation path, potentially hindering the model's ability to generalize compositionally or resist segmentation errors (Provilkov et al., 2019). BPE-dropout introduces stochasticity at segmentation time—each merge step is dropped with probability $\mathcal{T} \in \pi(D)$2—yielding a distribution over $\mathcal{T} \in \pi(D)$3 possible segmentations per word.

This regularization mechanism:

• Forces the model to learn more robust, compositional subword representations.
• Increases the occurrence and salience of frequent subword units and 4-grams, equalizing the embedding space for rare and frequent tokens.
• Enhances model robustness to noise, typographical errors, and domain shift, as evidenced by up to +2.3 BLEU improvement and better misspelling tolerance.
• Is most beneficial for low-resource settings and when applied to the training (not inference) phase only; high $\mathcal{T} \in \pi(D)$4 settings can underperform due to divergence from test-time segmentation (Provilkov et al., 2019).

5. Limitations and Cognitive Adequacy

Subword learning dynamics expose critical limitations in fixed-tokenization subword models when probed for word-level competence and learning sequence (Bunzeck et al., 18 Feb 2025). In controlled lexical decision tasks, character-level models rapidly and robustly distinguish real words from non-words (97–99% accuracy), while subword models perform poorly without context (as low as 35.6% accuracy for GPT-2-BPE).

Subword models:

• Rely heavily on syntactic or semantic context to recognize words, reflecting weak context-independent lexical representations.
• Exhibit entwined, not staged, acquisition of lexical and syntactic ability—unlike character models, which mirror staged, human-like language acquisition.
• Appear competent in surprisal-based tasks (close to character models in context), but this is due to their reliance on context rather than genuine lexical knowledge. Surprisal thus masks deficiencies in word learning (Bunzeck et al., 18 Feb 2025).

The implication is that while subword tokenization supports efficient modeling and open-vocabulary capacity, its learning dynamics may not parallel human psycholinguistics or provide robust lexical generalization absent contextual cues.

6. Implications for Language Diversity and Model Robustness

Dynamic subword learning and compositionality are particularly consequential for morphologically rich and low-resource languages (Meyer et al., 12 Nov 2025, Bojanowski et al., 2016). Learned segmentation enables adaptation to language-specific morphological regimes, supporting more balanced trade-offs between stem and affix coverage and significantly enhancing open-vocabulary text generation and cross-lingual transfer (up to 6 BLEU improvement observed for isiXhosa). BPE-dropout further increases compositional generalization, character-n-gram awareness, and error resilience in translation and representation learning (Provilkov et al., 2019).

A plausible implication is that model and tokenizer co-design—allowing for subword adaptation, architectural incentives for compositionality, and regularization supporting multiple segmentations—offers strategic levers for advancing robustness, transparency, and linguistic fidelity, especially in underrepresented linguistic contexts.

7. Open Problems and Future Directions

Current findings raise outstanding questions regarding the universality of composition-strategy groupings across model scales and languages, the effect of deeper or larger architectures (e.g., 70B LLMs), and the prospect for fine-tuning or inductive biases that bridge gaps between form–content transparency and semantic richness (Peng et al., 25 Aug 2025). Future work may explore:

• Injecting explicit linear or hierarchical biases for interpretable subword addition.
• Deeper, multi-phase training objectives that jointly encourage both abstraction and faithful surface-form reconstruction.
• Adaptive segmentation methods tuned to specific task or language morphology, rather than universal static BPE.

Systematic probing and augmentation of subword learning dynamics thus remains a central research axis for the continual development of language modeling technology and its alignment with linguistic structure.