- The paper introduces Morpheus, a neural model that unifies lossless morphology-aware tokenization with root-level word embeddings for Turkish.
- It leverages a deep, differentiable architecture using multi-scale convolutions and self-attention to achieve 100% reversible tokenization and robust morphological alignment.
- Empirical results demonstrate improved tokenization fidelity, efficient GPU usage, and superior lexical retrieval compared to traditional subword and rule-based methods.
Morpheus: Morphology-Aware Neural Tokenization and Embedding for Turkish
Motivation and Context
Turkish's agglutinative morphology introduces complexity for NLP systems: single roots generate hundreds of distinct surface forms via ordered morphemic affixation. For modern LMs, tokenization and embedding are bottlenecks. Corpus-driven subword models such as BPE, WordPiece, and Unigram fragment semantically loaded morphemes and often fail reversibility, either through lossy normalization or diacritic stripping. Rule- and dictionary-based systems such as Zemberek or TurkishTokenizer offer improved alignment but either lack scalability, fail to decode surface forms, or remain dictionary-bound. Meanwhile, standard representation learning decouples tokenization (meaningless token IDs) and embedding (large, context-heavy encoders), which duplicates the modeling of morphological signals.
Methodology
Morpheus unifies three functionalities in a standalone neural model:
- Lossless Morphology-Aware Tokenization: Given a Turkish word as a character sequence, Morpheus predicts morpheme boundaries using a deep, surface-preserving differentiable mechanism, ensuring decode(encode(w)) = w by construction.
- Structured, Morpheme-Derived Embedding: The same forward pass produces an embedding per word, organized at the root-family level.
- Neural Differentiability and OOV Generalization: The pipeline, trained end-to-end, accommodates any Turkish string, including nonce forms and rare surface fusions.
The segmentation process centers on a Poisson-binomial dynamic program that softly assigns character memberships to morphemes given per-character boundary probabilities. The mechanism supports end-to-end gradient flow so the boundary detector is directly shaped by both supervised (Morfessor-derived) labels and various distributional objectives (skip-gram, contrastive root-family learning, character-level MLM). The architecture employs multi-scale convolutions, RoPE-augmented self-attention, and segment-wise attention pooling in a 320-dimensional latent space.
Data are sourced from carefully cleaned, register-diverse Turkish corpora (Wikipedia, news, academic, colloquial). Preprocessing emphasizes strict surface preservation and comprehensive coverage of rare morphology.
Key Results and Numerical Evaluation
Tokenization Integrity
- Reversibility: Morpheus achieves 100% roundtrip reconstruction over >30,000 inflected wordforms, matching corpus-based BPE/Unigram/Byte-level methods. Both TurkishTokenizer (95.4%) and WordPiece (58.2%) are non-reversible due to canonicalization and diacritic stripping, rendering them invalid for generative use.
- Surface Fidelity: In curated OOV evaluation, Morpheus exhibits 0% corruption when matching segment boundaries to surface forms; unlike rule-based systems, no morph string is lost or rewritten.
- Morphological Alignment: Morpheus attains a MorphScore macro-F1 of 0.61 (UD_Turkish-Kenet), nearly double the subword baselines, and comparable to TurkishTokenizer (0.65), but with perfect surface string fidelity (no normalization-induced mismatch).
Downstream Language Modeling and Efficiency
- Bits-per-Character (BPC): On parameter-equalized 58M GPT LLMs, Morpheus yields the lowest BPC (1.425) among reversible tokenizers, slightly surpassing BPE and Unigram.
- Token Fertility: Morpheus emits 1.73 tokens per word, higher than subword schemes (~1.5). This deliberate trade-off optimizes morphological faithfulness at the expense of sequence length.
- GPU Memory: Morpheus demands ~19% less GPU memory during autoregressive generation than 64K-vocab subword configurations.
Embedding Quality
- Root-Level Retrieval and Deduplication: Frozen Morpheus embeddings outperform both BERTurk and BGE-M3 in lexical retrieval (root-family MAP 0.85 vs. 0.80/0.49) and same-root verification (ROC-AUC 1.00 vs. 0.98/0.70).
- Context/Inflection-Sensitive Tasks: Morpheus trails heavily contextual encoders in number/case morphological probing (0.59/0.22 vs. BERTurk's 0.95/0.89) and NER (macro-F1 0.48 vs. 0.79). The explicit root-centric training objective explains this: variance across inflectional features is collapsed, and lack of contextualization limits token-level entity resolution.
Implications and Discussion
Practical Impact
Morpheus provides a single, easily deployable component to replace multi-stage Turkish text pipelines:
- For text generation, it is one of the few reversible and morphology-aligned choices, critical for high-fidelity open-ended generation tasks.
- In lexical retrieval, deduplication, and dictionary-style applications, its root-clustered embeddings and lossless segmentation offer significant efficiency and faithfulness gains.
- The architecture lowers inference memory and features fast encoding/decoding throughput, narrowing practical gaps with established baselines.
Theoretical Insights
The integration of morph-boundary detection and embedding in one mechanism exposes an information-theoretical advantage in agglutinative language modeling. Coupling segmentation with representation allows for mutual reinforcement—morpheme boundary learning and word geometry are both optimized under end-task supervision, rather than separately and potentially adversarially.
Trade-Offs
Morpheus's uniform treatment of OOV/nonce forms via a neural model requires shipping a neural artifact (PyTorch checkpoint) rather than a static vocab table. Higher token counts modestly impact autoregressive speed. Its embedding geometry, favoring the root level, makes it ill-suited to tasks requiring fine-grained inflection or context comprehension, such as high-accuracy NER.
Future Directions
- Extending Morpheus's method to other agglutinative or morphologically rich low-resource languages could universalize its architecture.
- Hybrid or hierarchical contextualization strategies may bridge the root-level geometry with deeper, inflectionally nuanced representations.
- Investigating integration with RAG pipelines: use Morpheus for the lexical/keyword retrieval index and dense contextual encoders for semantic resolution.
Conclusion
Morpheus addresses longstanding deficiencies in Turkish NLP by providing a lossless, morphology-aware tokenizer that simultaneously delivers high-quality, root-structured embeddings in a single neural architecture. Among reversible tokenizers, Morpheus achieves the strongest scores in both intrinsic (BPC, morphological F1) and lexical downstream metrics, at a measured, explicit cost in sequence length and dense contextualization. Its deployment streamlines Turkish text pipelines for both generative LMs and information retrieval, offering a robust, theoretically principled, and practically validated alternative to frequency-driven or rule-based approaches. The decoupling of lossless morph segmentation from surface-destructive normalization measurably advances the state of Turkish tokenization and word representation.