- The paper introduces HeceTokenizer, a deterministic syllable-based tokenizer that outperforms larger morphology-driven models on retrieval tasks.
- It employs a greedy, right-to-left syllabification algorithm to create a fixed vocabulary of approximately 8,000 tokens, eliminating OOV errors.
- A lightweight BERT-tiny model trained on this segmentation achieves 50.3% Recall@5 on TQuAD, demonstrating robust semantic retrieval.
Syllable-Based Tokenization for Turkish: HeceTokenizer and Its Efficacy in Semantic Retrieval
Overview and Motivation
"HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval" (2604.10665) presents a resource-light tokenization framework leveraging the deterministic six-pattern phonological structure of Turkish. Unlike prevalent BPE or WordPiece schemes, or recent morphology-aware systems, HeceTokenizer segments text into syllabic units, yielding a closed vocabulary of approximately 8,000 unique types—rendering OOV tokens impossible by design. The paper demonstrates that a lightweight BERT-tiny (1.5M parameters) encoder trained on HeceTokenizer segmentation competes with, and in fact outperforms, much larger morphology-driven retrieval baselines on the TQuAD benchmark.
The agglutinative, morphologically rich nature of Turkish generates a vast space of surface forms from a modest root inventory due to productive suffix concatenation. Frequency-based tokenizers conventionally introduce many spurious splits and OOV fragments, since they lack explicit linguistic awareness of morphemic or phonological boundaries.
Prior work has addressed these pathologies in two principal directions. Morphology-driven tokenizers, such as TurkishTokenizer [3], employ curated lexicons for segmentation and mapping to canonical forms, with fallback to subword (BPE) units for OOV. Such systems substantially improve linguistic alignment but retain dependency on external resources and remain theoretically susceptible to OOV when coverage is incomplete. Several works have inserted morphological constraints into BPE or WordPiece schemes, yielding improvements in compositionality and coverage [10,12]. However, these strategies typically scale vocabulary size and resource requirements.
Syllable segmentation has been occasional in transparent phonological systems, but has not been systematically applied for Turkish retrieval. The deterministic nature of Turkish syllabification—accounted for by six patterns (V, CV, VC, CVC, VCC, CVCC)—enables exhaustive, unsupervised decomposition of the entire language surface form space into a stable, closed set of tokens. The primary contribution of HeceTokenizer is to empirically validate that this purely phonologically-motivated approach substantially closes the gap with, and in fact exceeds, large morphology-informed systems on hard retrieval tasks.
Methodology
Syllabification Algorithm and Tokenizer Construction
HeceTokenizer implements syllabification via a right-to-left, greedy pattern-matching algorithm, collapsing contiguous substrings matching one of the six canonical syllable types. This approach requires neither morphological parsing nor any form of lexical knowledge—yielding significant scalability and applicability in low-resource or open-domain settings. The resulting token set forms a fixed, OOV-free closed vocabulary, empirically measured at approximately 8,000 unique syllable types in a large-scale corpus.
Preliminary analysis showed that standard BPE tokenizers (trained on Turkish Wikipedia subsets) suffered severe embedding collapse (mean cosine similarity ~0.998 between passage embeddings), leading to degenerate representations. In contrast, syllable-based tokenization produced well-separated passage embeddings (mean cosine similarity ~0.257), confirming the inductive value of phonological units for modeling.
Retrieval Model and Benchmark
A BERT-tiny model was trained from scratch using the MLM objective on 26,000 Turkish Wikipedia articles, entirely syllabified. The model configures only 1.5M parameters (with hidden size 128 and 2 transformer layers), in contrast to the 300M parameter sizes typical in prior TurkishTokenizer experiments.
Retrieval quality was measured on the TQuAD development set (892 QA pairs), using Recall@5. Rather than full-passage retrieval, HeceTokenizer systematically varies the chunk size—retrieving from smaller, overlapping syllabic chunks. Embeddings are compared via cosine similarity, with a top-5 evaluation.
Empirical Results
The study's central empirical findings are:
- HeceTokenizer achieves 50.3% Recall@5 at optimal chunk size (8 syllable tokens), surpassing TurkishTokenizer's morphology-driven BERT-Large (46.92% Recall@5 at 300M parameters) and all BPE-based variants, despite using a model nearly 200 times smaller.
- The optimal chunk size (~8 syllables, or ~2.6 words) aligns closely with typical length of answer spans, entities, or technical terms in QA retrieval, maximizing locality of lexical overlap while preserving contextual coherence.
- Token density of HeceTokenizer is higher (3.22 tokens/word) than competitive baselines, indicating finer segmentation granularity. This implicates careful context window sizing for downstream tasks but does not diminish retrieval efficacy.
- The closed vocabulary and lack of OOV tokens, together with phonological segmentation, yield more robust embedding distributions and improved semantic retrieval in Turkish.
Theoretical and Practical Implications
These findings demonstrate that the regular, deterministic phonological structure of Turkish syllables provides a sufficiently strong inductive bias for semantic retrieval, even in the absence of morphological lexical resources. The approach's independence from lexicon curation, frequency estimation, or statistical word boundary induction positions it as a viable solution for (i) low-resource languages with explicit syllable structure, (ii) environments with limited compute or labeled data, and (iii) applications where strict OOV absence is critical (e.g., IR in specialist or domain-shifted corpora).
While retrieval efficacy was demonstrated on TQuAD—with heuristically empirically determined chunk size—the methodology may generalize to other Turkish benchmarks (e.g., TR-MTEB) and even to generative tasks, pending further validation. There is also the open possibility of similar gains in other languages with closed syllabic inventories.
A limitation is the increased inference-time computation due to increased chunk-based retrieval granularity, particularly as chunk partition count increases. However, this cost must be weighed against the significant model size reduction and elimination of OOV errors.
Conclusion
HeceTokenizer establishes that fine-grained, phonologically deterministic syllable tokenization can enable high-performance, OOV-free Turkish textual representation for retrieval, rivaling and even outperforming much larger morphology-driven systems. The approach is lexicon-free, computationally light, and produces robust semantic representations at subword level. Future work should extend validation to additional tasks and languages, and assess generative modeling and broader cross-domain generalizability.