SampoNLP: Corpus-Free Morphology Toolkit

• SampoNLP is an open-source, corpus-free toolkit that induces high-purity morpheme lexicons and evaluates subword tokenizers for Uralic languages.
• The toolkit uses an Iterative Morphological Decomposition Pipeline with MDL-inspired self-referential scoring to refine noisy candidate lexicons.
• It provides empirically grounded recommendations for optimal BPE vocabulary sizes in Finnish, Hungarian, and Estonian by balancing lexical coverage and over-splitting.

Searching arXiv for the cited SampoNLP paper and closely related contextual work. SampoNLP is an open-source, corpus-free toolkit for morphological lexicon creation and for the intrinsic evaluation of subword tokenizers in morphologically rich Uralic languages. It was introduced to address a specific bottleneck: the absence of clean, high-purity morpheme lexicons for Finnish, Hungarian, and Estonian, which makes it difficult to measure whether subword tokenizers preserve meaningful morphological units. The toolkit centers on an Iterative Morphological Decomposition Pipeline (IMDP) built around “MDL-inspired Self-Referential Atomicity Scoring,” and its induced lexicons are then used to analyze Byte-Pair Encoding (BPE) tokenizers across vocabulary sizes from $8k$ to $256k$. The resulting study provides the first empirically grounded recommendations for BPE vocabulary size in these languages and quantifies the limitations of standard BPE under strong agglutinative morphology (Chelombitko et al., 8 Jan 2026).

1. Problem setting and conceptual scope

SampoNLP was introduced for low-resource settings in which corpus-based unsupervised morphology methods are less suitable because they depend on token-frequency information from sizable corpora. Its immediate practical goal is not full linguistic morphology in the classical sense, but the automatic and reproducible creation of cleaner reference morpheme inventories from noisy lexical resources such as Hunspell dictionaries. Those inventories serve as evaluation references for tokenizer analysis rather than as externally validated gold morphology (Chelombitko et al., 8 Jan 2026).

The toolkit is explicitly corpus-free and type-only. It starts from noisy candidate lists assembled from dictionary stems and affixes, then filters those lists through internal structural evidence. The central assumption is that recurrent sublexical units can be detected without corpus frequency, external gold annotation, or an external analyzer, provided that candidate forms are evaluated in relation to other candidates in the same inventory. This makes SampoNLP “self-referential”: a candidate is judged only by whether it can be explained by other candidates in the set.

This design places SampoNLP in a distinct niche. It is not presented as a general-purpose NLP pipeline for POS tagging, dependency parsing, or named entity recognition. Its scope is narrower and more technical: corpus-free lexicon induction for tokenizer evaluation, especially where morphology is rich and clean morpheme lexicons are unavailable.

2. Iterative Morphological Decomposition Pipeline

The core workflow is the Iterative Morphological Decomposition Pipeline. IMDP has three stages: pre-filtering and support-based candidate selection, iterative atomicity refinement, and final thresholding into atomic versus composite forms (Chelombitko et al., 8 Jan 2026).

In Stage 1, the raw candidate list $C_{raw}$ is hard-filtered. Tokens are removed if they contain symbols outside a language-specific valid alphabet, contain non-alphabetic material other than boundary hyphens used for affixes, look like proper nouns or acronyms, or are implausibly short or long, except for whitelisted one-character morphemes. The remaining set $C'$ is then filtered by type support:

$$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$

and a token is kept only if $support(t) \geq m$, with $m=3$. The surviving set is the final candidate pool $C$. Each $t \in C$ receives an initial atomicity score

$S_0(t) = \frac{1}{|t|}.$

This encodes the preference for shorter units as plausible building blocks.

Stage 2 performs iterative refinement. For each token $256k$0, the algorithm searches for its best decomposition into smaller tokens from the same candidate set, allowing any number of components and permitting length-1 segments only if they belong to a language-specific whitelist $256k$1. The Best Explanation Power is defined as

$256k$2

Dynamic programming is used to find the decomposition with maximum summed score. Scores are then updated by comparing a token’s current atomicity to its best internal explanation:

$256k$3

A token is penalized only when its decomposition gives stronger evidence than its current score as an atomic unit. Convergence is defined by a maximum score change below $256k$4, with $256k$5 and $256k$6.

Stage 3 converts the final score distribution into a binary decision. Otsu’s thresholding method selects $256k$7 by maximizing inter-class variance in the score distribution, and tokens with $256k$8 are retained as atomic. This final step makes the cutoff data-driven and reproducible rather than manually tuned.

The pipeline is “MDL-inspired” because it favors compact explanations in which smaller reusable units explain larger forms. It is not a classic MDL morphology model, however, because it has been adapted to a corpus-free setting.

3. Lexicon induction for Finnish, Estonian, and Hungarian

The initial noisy candidate lists were derived from open Hunspell-based dictionaries: LibreOffice dictionaries for Hungarian and Estonian, and hunspell-fi for Finnish. For each language, stems from .dic files and affixes from .aff files were merged into a noisy candidate inventory, and that inventory was then passed through IMDP (Chelombitko et al., 8 Jan 2026).

The reported filtering is severe. Finnish was reduced from 499,647 initial candidates to 3,850 atomic morphemes, a 99.23% reduction and a 129.8x reduction factor. Estonian was reduced from 281,256 to 5,705, a 97.97% reduction and a 49.3x factor. Hungarian was reduced from 103,317 to 3,189, a 96.91% reduction and a 32.4x factor.

These outputs are repeatedly described as “high-purity” and “clean,” but the paper also states an important qualification: they are not guaranteed to be fully linguistically correct morphemes. The induced units are orthographic, type-based, and self-referential. They are therefore best understood as practical approximations of recurrent sublexical units rather than phonologically grounded morphological analyses.

This distinction is central to the toolkit’s epistemic status. SampoNLP does not claim to recover gold morphology. It constructs a usable reference layer for intrinsic tokenizer evaluation, especially where the alternative would be either manual curation or frequency-dependent unsupervised morphology.

4. Tokenizer evaluation methodology

The induced lexicons are used to evaluate standard BPE tokenizers trained from scratch on language-specific Wikipedia corpora. Hugging Face tokenizers and SentencePiece are used for comparison in the experimental setup, although the reported sweep focuses on BPE. Vocabulary size is varied over

$256k$9

with $C_{raw}$0 for merges (Chelombitko et al., 8 Jan 2026).

Two complementary metrics are defined. Lexical Morpheme Coverage (LMC) measures the fraction of reference morphemes from the cleaned lexicon $C_{raw}$1 that appear directly as single vocabulary entries in $C_{raw}$2:

$C_{raw}$3

Over-Split Rate (OSR) measures how often morphemes that occur inside words are nevertheless never realized as a single token:

$C_{raw}$4

Because a useful tokenizer should increase LMC while decreasing OSR, the paper combines them into the Integrated Performance Score (IPS), defined as normalized Euclidean proximity to the ideal point $C_{raw}$5:

$C_{raw}$6

Higher IPS therefore corresponds to better joint behavior on morpheme storage and resistance to over-splitting.

The IPS curve is further summarized through characteristic points. The elbow point $C_{raw}$7, detected with the Kneedle algorithm, marks diminishing returns. The $C_{raw}$8 quality point $C_{raw}$9 is the smallest vocabulary size at which $C'$0 of the maximum observed IPS is reached. Together they define a recommended vocabulary range $C'$1. A “max gain point” $C'$2 is also reported, though it is not formalized further.

5. Empirical findings and recommended BPE vocabulary sizes

The central empirical result is a diminishing-returns pattern in IPS as vocabulary size grows: rapid gains at smaller vocabularies followed by flattening. The recommended BPE ranges are language-specific (Chelombitko et al., 8 Jan 2026).

Language	Recommended $C'$3 range	Key reported points
Hungarian	$C'$4–$C'$5	elbow $C'$6, $C'$7, max IPS about $C'$8
Estonian	$C'$9–$$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$0	elbow $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$1, $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$2, max IPS about $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$3
Finnish	$$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$4–$$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$5	elbow $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$6, $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$7, max IPS about $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$8

The language-specific curves are highly uneven. For Hungarian, LMC rises from 25.15% at $$support(t) = |\{c \in C' \mid t \text{ is a substring of } c\}|$$9 to 79.12% at $support(t) \geq m$0, while OSR falls from 67.72% to about 32.04%. For Estonian, LMC rises only from 11.27% to 26.81%, and OSR remains high, declining from 65.79% to 46.11%. Finnish is the weakest case: LMC rises from 7.85% to only 23.73%, and OSR remains around 60% even at $support(t) \geq m$1.

These results underpin the paper’s critique of standard BPE. Larger vocabularies help, but they do not resolve the underlying morphological mismatch. Hungarian behaves relatively better, which the authors attribute to a more transparent agglutinative structure with fewer morphophonological alternations. Estonian and especially Finnish remain difficult because orthographic morpheme boundaries are obscured by alternations; in Finnish, consonant gradation and stem alternations are singled out as particularly problematic.

A plausible implication is that vocabulary-size tuning should be treated as an optimization problem within a constrained method, not as a substitute for morphology-aware tokenization. The low IPS ceiling for Finnish indicates that increasing $support(t) \geq m$2 alone is insufficient.

6. Limitations, uses, and broader significance

SampoNLP’s release includes the toolkit for corpus-free lexicon induction and the generated morpheme lists for Finnish, Hungarian, and Estonian at the GitHub repository linked in the paper. Intended use cases include creating reference lexicons in low-resource settings, evaluating tokenizer morphology intrinsically, and supporting reproducible research on subword segmentation for Uralic languages (Chelombitko et al., 8 Jan 2026).

The main limitations are stated clearly. The induced lexicons are orthographic, type-based, and self-referential; they are not guaranteed to be fully linguistically correct morpheme analyses. IPS captures the coverage/over-splitting trade-off, but it ignores some qualitative differences in segmentation. The tokenizer experiments are conducted on clean standardized corpora rather than noisy real-world text. The paper also does not provide extensive ablation studies of IMDP itself.

Within those limits, the toolkit’s significance is methodological. It supplies a lightweight, automatic, reproducible alternative to frequency-dependent morphology induction for settings where clean lexicons do not exist. It also converts tokenizer selection from heuristic practice into a measurable trade-off governed by LMC, OSR, and IPS. The paper’s practical guidance is correspondingly specific: standard BPE vocabulary size should not be set heuristically; it should be chosen in the elbow-to-$support(t) \geq m$3 range, and that range is language-specific.

The broader conclusion is more critical than prescriptive. SampoNLP demonstrates that standard BPE has substantial limitations for highly agglutinative languages, particularly Finnish. Future work is framed around richer qualitative segmentation analysis and the development of genuinely morphology-aware tokenization methods, especially where standard BPE appears fundamentally inadequate.