MorphGPT Language Model

• MorphGPT is a GPT-style causal language model that deterministically segments words into canonical morphemes using the MorphPiece tokenization scheme.
• It achieves substantial perplexity reductions and significant improvements in zero-shot natural language understanding tasks compared to standard GPT-2 models.
• Its explicit morphological decomposition enhances interpretability and rare-word coverage, underscoring the benefits of integrating linguistic inductive bias into language modeling.

MorphGPT is a GPT-style causal LLM that incorporates a linguistically motivated tokenization scheme, MorphPiece, designed to infuse explicit morphological information into the neural language modeling pipeline. Through deterministic segmentation of known vocabulary into canonical morphemes, MorphGPT demonstrates improved performance across a range of natural language understanding and embedding tasks when compared to standard statistically-driven tokenization approaches and retro-fitted morphological schemes. The integration of MorphPiece remains lightweight, relying on a nearly identically sized vocabulary and the same autoregressive transformer architecture as GPT-2 Base, yet exhibits marked gains in downstream metrics after only half as many pre-training updates (Jabbar, 2023).

1. MorphPiece Tokenization and Morphological Segmentation

MorphPiece extends beyond contemporary subword tokenizers by introducing deterministic morphological analysis for English. Its key differentiation lies in splitting known words, as defined in a 134,943-entry MorphTable (sourced from MorphyNet), into linguistically faithful stems, prefixes, suffixes, and compound markers. The resulting tokens use explicit markers: a trailing ‘#’ denotes a prefix, a leading ‘#’ a suffix, a stand-alone ‘#’ a compound boundary, and stem forms remain unmarked. For example, “paratrooper” segments as “para#”, “troop”, “#er”.

The segmentation pipeline begins with standard byte-pair encoding (BPE) pre-tokenization (whitespace, punctuation regex). Each pretoken is then mapped via MorphTable to its canonical morpheme sequence if present; out-of-vocabulary words are tokenized with a trained BPE model. This scheme yields 18,304 morpheme-based tokens (minimum 5 corpus occurrences) and 32,000 BPE subwords trained on the residual lexicon, producing a final vocabulary of 50,006, comparable to GPT-2’s 50,257.

MorphPiece’s operation is deterministic, but a (pseudo-)probabilistic perspective can be formulated:

$$\operatorname{segmentation}(w) = \begin{cases} \text{MorphTable}[w], & \text{if}\ w \in \text{MorphTable} \ \underset{\text{BPE}}{\arg\max}\ P_\text{BPE}(\text{bpe tokens}\mid w), & \text{otherwise} \end{cases}$$

Standard statistical tokenizers such as BPE, WordPiece, and Unigram lack explicit morphological structure. FLOTA, a competing approach, greedily approximates morphemes at fine-tuning time using available BPE/WordPiece splits, but does not build new atomic units or integrate morphological segmentation during pre-training.

2. Model Architecture and Pre-training Paradigm

MorphGPT precisely reproduces the OpenAI GPT-2 Base architecture: 12 transformer decoder layers with hidden size 768, 12 attention heads, and an intermediate dimension of 3072 in the feed-forward layers. The autoregressive context window size is 1024 tokens, and the network uses causal self-attention and post-attention layer normalization.

Training follows standard procedures for language modeling objectives:

$$L(\theta) = -\sum_{t=1}^{T} \log P_\theta(x_t\mid x_{<t})$$

using a standard cross-entropy loss over the 50,006-token MorphPiece vocabulary with no auxiliary morphological prediction. The model is trained on OpenWebText for 200,000 steps (approximately half the duration of GPT-2’s reported 400,000–500,000 steps), with batch size 512, Adam optimizer ($\beta_1=0.9$, $\beta_2=0.995$, $\epsilon=10^{-8}$), and a one-cycle learning rate schedule: 2,000-step warmup to 1e–3, followed by cosine decay to a final 1e–5.

3. Empirical Evaluation and Benchmarking

MorphGPT is assessed on a diverse set of language benchmarks, always in direct comparison with GPT-2 Base, GPT-2 Large, and in certain tasks, the FLOTA morphologically-inspired tokenization retrofit.

3.1 Language Modeling Perplexity

Dataset	GPT-2 Base	MorphGPT	GPT-2 Large
Penn Treebank	61.86	38.69	37.93
OpenAI-250K	25.54	17.50	16.74
LAMBADA	55.87	42.85	37.21

MorphGPT achieves substantial perplexity reductions, with Penn Treebank and OpenAI-250K results approaching those of GPT-2 Large despite substantially shorter training.

3.2 Cloze and Zero-Shot Natural Language Understanding

On LAMBADA cloze tasks, MorphGPT achieves 58.5% accuracy for fragmented targets (+25% over GPT-2) and 36.6% for unfragmented words. In zero-shot GLUE evaluation using PromptSource templates, MorphGPT consistently surpasses GPT-2 Base on most tasks, especially RTE (+54% gain), QQP (+28%), and QNLI (+46%).

3.3 Massive Text Embedding Benchmark (MTEB)

MorphGPT demonstrates robust improvements across embedding benchmarks, including +93% V-Measure on clustering, +17% classification accuracy, and gains from +48% to +103% on STS (Spearman/Pearson).

3.4 FLOTA Comparison

On arXiv paper‐title classification, GPT-2 plus FLOTA yields 6–10% improvement over baseline, whereas MorphGPT achieves +26–28% (clean) and +40–88% (noisy) margins, showing significant benefits of integrating morphological splits at the pre-training stage.

4. Qualitative Analysis and Tokenization Dynamics

MorphGPT’s tokenization increases average split length (“fertility”) by approximately 17% relative to BPE, resulting in longer tokenized sentences. Nevertheless, the explicit morphological decomposition facilitates more interpretable input representations and improved rare-word coverage. Canonical segmentation, such as “bat”+“ing” (versus BPE’s “bat”+“ting”), mitigates spurious surface-form errors. Consistent treatment of negation prefixes and productive affixes further aids model interpretability.

5. Comparative Assessment and Linguistic Inductive Bias

MorphGPT’s primary innovation is the injection of linguistic inductive bias directly through its tokenization layer, in contrast to post-hoc morphologically-informed schemes (e.g., FLOTA) and purely statistical approaches. This explicit morphological structure enables better handling of compositional and rare vocabulary, as evidenced in both quantitative and qualitative evaluations. The findings support the conclusion that a lightweight, linguistically grounded modification to the tokenizer can yield strong performance gains even with reduced pre-training resources (Jabbar, 2023).

6. Future Directions and Open Challenges

The authors highlight several avenues for development: expanding MorphTable to cover more languages and lexical families, reducing sequence length by morpheme fusion, scaling model size, and releasing code and checkpoints. Prospects include a multilingual detokenization framework and deeper integration of morphology into LLMs, beyond tokenization. A plausible implication is that further morphological integration at various stages of the model pipeline could yield additional gains across multilingual and compositional generalization benchmarks.