MorphScore: Subword Tokenizer Alignment

Updated 3 July 2026

MorphScore is an intrinsic metric that quantifies how well subword tokenizer boundaries align with linguistically defined morpheme boundaries using precision, recall, and F₁ scores.
It employs methods from single-boundary accuracy to macro-averaged F₁ across datasets from UD and UniMorph, offering robust evaluations over up to 70 languages.
Empirical results show that while higher MorphScore values indicate more linguistically plausible tokenization, they do not directly correlate with improved overall language model performance.

MorphScore is an intrinsic evaluation metric designed to quantify the degree to which subword tokenizer boundaries align with gold-standard morpheme boundaries in a given language. It is used to diagnose and compare the morphological alignment of tokenizers, especially in the context of morphologically complex languages. MorphScore has evolved in form and scope, from single-boundary accuracy in small typological samples to macro-averaged precision/recall/F₁ across diverse boundary configurations and datasets spanning up to 70 languages. It informs the study of tokenizer quality as it relates to both linguistic plausibility and real-world task performance in LLMs.

1. Formal Definitions and Variants

MorphScore has several instantiations, all anchored to the evaluation of how closely tokenizer-induced subword segmentations correspond to linguistically motivated morpheme boundaries.

Single-boundary accuracy (22-language setting)

Given a word $w$ with gold-standard morpheme boundary at character offset $p(w)$ and a tokenizer that produces splits at positions $B(w)$ , define the indicator: $I(w) = \begin{cases} 1 & \text{if } p(w) \in B(w), \ 0 & \text{otherwise}. \end{cases}$ Exclude $w$ if $|B(w)| = 0$ (i.e., word is not segmented), and compute MorphScore as: $\text{MorphScore} = \frac{1}{|W'|}\sum_{w\in W'} I(w),$ where $W'$ is the set of evaluated words (those with at least one token boundary) (Arnett et al., 2024).

Macro-F₁ for multi-boundary words (Turkish case)

Let $B_p$ be the set of predicted boundaries and $B_g$ the gold boundaries. For each $p(w)$ 0: $p(w)$ 1 MorphScore is the macro-average: $p(w)$ 2 (Şakar, 17 Jun 2026).

Expanded (70-language) MorphScore

Let the gold segmentation yield morpheme spans $p(w)$ 3, tokenizer output spans $p(w)$ 4, with boundary sets $p(w)$ 5 as defined. MorphScore reports:

Boundary-level precision: $p(w)$ 6
Boundary-level recall: $p(w)$ 7
Subword-level precision/recall/F₁ (span alignment):

$p(w)$ 8

with associated $p(w)$ 9 (Arnett et al., 8 Jul 2025).

Scoring may be micro- or macro-averaged, with optional frequency weighting and single-token word exclusion.

2. Datasets and Language Coverage

Sources:

Gold-standard morphological boundaries are drawn from:

Universal Dependencies (UD) treebanks, including morphological annotation layers, for a majority of languages
UniMorph inflection tables (for Turkish, Gujarati, Kurdish, Zulu)
J-UniMorph resource for Japanese
Sentence context, POS tags, and inflectional features in expanded versions

Languages:

Original 22-language set (split evenly between agglutinative and fusional, e.g., Turkish, Japanese, Zulu, English, Persian, Greek) (Arnett et al., 2024)
2025 extension: 70 agglutinative or fusional languages, filtering out monomorphemic, irregular, and non-concatenative forms (Arnett et al., 8 Jul 2025)
Turkish-specific: Over 30,000 inflected forms from UD_Turkish-Kenet, all boundary-labeled (Şakar, 17 Jun 2026)

Minimum dataset sizes are enforced (e.g., $B(w)$ 0 items per language).

3. Evaluation Procedures and Practical Computation

Generic Pipeline:

Gold Data Preparation: Collect morphological boundary-annotated word forms (from UD or equivalent resources). For consistency, forms with exactly one boundary are selected in the original version.
Tokenizer Segmentation: Apply the tokenizer to each word, producing a segmentation (token boundary positions).
Boundary Comparison: Compare tokenizer boundary locations to gold standard(s), word by word. For each word, compute I(w), precision/recall, or F₁ as described above.
Aggregation: Average results (arithmetic mean for accuracy/indicator; macro-average for F₁) across the (filtered) test set.
Optional Adjustments: Weight by word frequency, exclude whole-word tokens, and use span-based matching if provided.

No dynamic programming or edit-distance matching is applied at evaluation time; a set intersection over character offsets suffices (Şakar, 17 Jun 2026). The metric design ensures that over-segmentation is penalized through reduced precision, especially in macro-F₁ formulations.

4. Empirical Findings and Comparative Results

Alignment statistics:

Model	MorphScore (macro-F₁)	Reversible?
Morpheus	0.61	Yes
Morfessor	0.59	Yes
BPE	0.32	Yes
Unigram	0.32	Yes
TurkishTokenizer	0.65	No
WordPiece	0.27	No

Notes: Non-reversible tokenizers (i.e., those that cannot decode losslessly) are unsuitable for generative modeling, regardless of MorphScore (Şakar, 17 Jun 2026).

Across the 70-language benchmark, typical transformer-based tokenizers (e.g., XGLM, BLOOM, Llama2/3, Gemma3) achieve low boundary precision and modest recall: for instance, XGLM attains $B(w)$ 1, $B(w)$ 2 across languages, with higher scores for European languages (English $B(w)$ 3, German $B(w)$ 4, Turkish $B(w)$ 5) (Arnett et al., 8 Jul 2025).

Agglutinative languages exhibited higher average MorphScores ( $B(w)$ 6) than fusional languages ( $B(w)$ 7) in the indicator version (Arnett et al., 2024).

Correlation to LM quality:

No significant linear correlation is found between MorphScore and language-model perplexity or downstream accuracy in regression and mixed-effects analyses, after accounting for model size and dataset proportion. The $B(w)$ 8 values are small ( $B(w)$ 9 for recall, $I(w) = \begin{cases} 1 & \text{if } p(w) \in B(w), \ 0 & \text{otherwise}. \end{cases}$ 0 for precision), sometimes negative, indicating (counterintuitively) that better alignment does not entail better model performance (Arnett et al., 2024, Arnett et al., 8 Jul 2025).

5. Interpretation, Limitations, and Recommendations

Interpretation:

High MorphScore indicates that tokenizer subword boundaries coincide with gold morpheme boundaries; low scores imply that segmentations crosscut morphological units, producing semantically less plausible subwords.
Agglutinative languages can have higher intrinsic alignment, challenging the assumption that they are inherently harder for subword-based models.
MorphScore is best employed for intrinsic tokenizer diagnostics, particularly when tasks require linguistically meaningful subwords (e.g., morphological tagging, low-resource scenarios) (Arnett et al., 2024).

Limitations:

Single-boundary versions do not faithfully model multi-morpheme structure; multi-boundary averaging addresses this partially.
MorphScore ignores over-segmentation except via macro/micro-averaged precision. It does not penalize unnecessary splits unless precision is considered.
Small or unbalanced datasets (e.g., few examples for some languages) reduce robustness.
Gold boundaries are not always available, and gold segmentations rely on inflectional, not derivational or non-concatenative, morphology.
Scores are sensitive to frequency effects and inclusion/exclusion of whole-word tokens (Arnett et al., 8 Jul 2025).
All evaluations have focused on monolingual tokenizers; extending to multilingual, vocab-shared tokenizers remains an open avenue.
Morphological alignment trades off with compression; high MorphScore does not guarantee superior overall tokenizer quality.

Recommendations:

Use MorphScore as a tool to guide tokenizer selection or design in contexts where morpheme integrity is desirable.
Consider augmenting tokenization algorithms with morphological analyzers, supervised merge/prune routines (e.g., BPE-knockout, CompoundPiece), or linguistically-motivated subword induction (e.g., Morfessor, MRT) if MorphScore is low and task demands require it (Arnett et al., 2024).
Pair MorphScore evaluation with information-theoretic metrics (e.g., bits-per-character, Rényi entropy) for comprehensive assessment.
Recognize the limitations regarding transferability of high MorphScore to generic model quality, emphasizing paired metrics and sufficient data coverage.

6. Extensions and Future Directions

MorphScore has been expanded to support more nuanced evaluation regimes:

70-language coverage, with explicit boundary and subword-level metrics, and optional frequency-based weighting, addresses earlier limitations (Arnett et al., 8 Jul 2025).
Micro- and macro-variant reporting allows focus on either corpus-wide or per-type alignment.
Annotation and scoring now incorporate more context (POS tags, morphological features), supporting future analyses beyond subword alignment.
Current limitations (e.g., bias toward Indo-European, exclusion of non-concatenative languages, coarse boundary annotation) point to the need for wider typological breadth and richer gold resources.
Future work is expected to:
- Extend MorphScore evaluation to truly multilingual and vocabulary-shared tokenizers (e.g., mBERT, mT5).
- Develop more refined metrics that penalize excessive subwording and capture more complex, non-concatenative morphologies.
- Combine alignment metrics with compression/information criteria for balanced assessments, especially for morphologically rich, low-resource languages.

7. Summary Table of MorphScore Key Properties

Version/Study	Metric Formula & Scope	Languages	Data Source
(Arnett et al., 2024) (2024)	Single-boundary accuracy (indicator mean)	22	UD, UniMorph
(Şakar, 17 Jun 2026) (2026)	Macro-F₁, all boundaries per word	Turkish	UD_Turkish-Kenet
(Arnett et al., 8 Jul 2025) (2025)	Boundary- and span-level precision/recall/F₁	70	UD (filtered)

These formulations and their empirical results provide a typologically broad and methodologically rigorous toolkit for evaluating and comparing subword tokenizers with respect to morphological transparency, while highlighting the nuances and trade-offs inherent in intrinsic and extrinsic evaluation paradigms.