MorphScore Boundary Recall Metric

• The paper introduces MorphScore boundary recall, a metric that quantifies how well a tokenizer aligns with gold-standard morpheme boundaries.
• It employs UD-derived annotations and systematic mapping of predicted subword boundaries, using both micro- and macro-averaging for corpus-level evaluation.
• Empirical results across various languages and tokenizers highlight oversegmentation biases and underscore the need to complement recall with precision metrics.

MorphScore boundary recall is a metric designed to quantify the degree to which the tokenization of a LLM aligns with linguistically valid morpheme boundaries. Specifically, it measures the proportion of true morphological boundaries within words that are correctly captured by a tokenizer, providing a formalized approach to evaluating subword alignment for concatenative morphology (fusional and agglutinative languages) across multiple languages and tokenization algorithms (Arnett et al., 8 Jul 2025).

1. Formal Definition and Calculation

Let each gold-standard segmentation of a word be defined by the set $G$ of true morpheme-boundary positions (character indices marking morpheme ends) and the set $P$ of predicted subword-boundary positions produced by the tokenizer. For a word $w$ of length $N$ characters, assign $N - 1$ possible boundary indices. The following conventions apply:

• $|G|$: number of gold (true) morpheme boundaries
• $|P|$: number of predicted token boundaries
• $|G \cap P|$: count of gold boundaries correctly recovered by the predicted segmentation

Boundary recall for a given word $w$ is defined as:

$\mathrm{recall}(w) = \frac{|G \cap P|}{|G|}$

Corpus-level boundary recall may be reported as either micro-averaged or macro-averaged:

• Micro-averaged recall:

$P$0

• Macro-averaged recall:

$P$1

where $P$2 is the number of eligible words in the corpus.

2. Stepwise Computation Procedure

Boundary recall is computed via a series of steps that systematically extract and align morphemic and predicted boundaries:

1. Gold boundary extraction: Use the Universal Dependencies (UD)-derived annotation to mark each stem and affix position; this constitutes the gold boundary set $P$3.
2. Tokenizer boundary mapping: Tokenize the word to yield subwords $P$4 and map each subword boundary to its corresponding character index, producing $P$5.
3. Correct boundary identification: For each word, count the correctly predicted morpheme boundaries as $P$6.
4. Per-word recall calculation: Compute $P$7 for all eligible words (words containing more than one morpheme).
5. Corpus-level aggregation: Aggregate per-word scores via micro- or macro-averaging as defined above.

3. Aggregation, Weighting, and Variants

Several preprocessing and aggregation options modulate the interpretation of corpus-level boundary recall:

• Frequency scaling: Words are weighted by corpus frequency $P$8, as determined by the UD treebank, modifying recall to:

$P$9

• Single-token word handling: The inclusion or exclusion of single-token words (cases where the tokenizer emits exactly one token) significantly affects recall. For multi-morphemic words with only one predicted token, recall is scored as 0. Single-morpheme words are excluded, as they offer no segmentation points.
• The default approach in (Arnett et al., 8 Jul 2025) combines frequency scaling with the exclusion of one-token predictions, as this configuration maximizes predictive value for downstream performance, though morph-alignment recall alone accounts for only approximately $w$0 of variation in task outcomes.

4. Illustrative Example

Consider the wordform "books" with the gold morphological segmentation [book + s]:

• Gold morpheme boundary: $w$1 (boundary between "book" [0–3] and "s" [4])
• Tokenizer output: [boo + k + s]
  • "boo": characters 0–2 → boundary at 3
  • "k": character 3 → boundary at 4
  • "s": character 4 → boundary at 5 (end of word, not scored)
• Predicted boundaries: $w$2

Calculation:

• $w$3 (boundary at 4 matches)
• $w$4
• $w$5
• Precision: $w$6

Aggregating these figures across all relevant words provides the macro or micro corpus-level score.

5. Empirical Results Across Models and Languages

The accuracy of morphological boundary recovery varies widely by tokenizer and language. Table 1 from (Arnett et al., 8 Jul 2025) reports corpus-average, frequency-scaled, and one-token-excluded recall across five widely used tokenizers and 70 languages:

Tokenizer	Boundary Recall (mean ± std)
BLOOM	0.33 ± 0.00
Gemma 3	0.35 ± 0.00
Llama 2	0.56 ± 0.00
Llama 3	0.45 ± 0.00
XGLM	0.52 ± 0.00

• In non-Latin-script languages (e.g., Korean, Sinhala, Tamil), Llama 2 and others often oversegment at the character or byte level, resulting in near-perfect recall ($w$71.0).
• In fusional, alphabetic languages (e.g., German, Dutch), recall generally ranges from 0.2 to 0.7 depending on tokenizer fragmentation.
• Llama 2 displays consistently high recall due to its strong tendency to oversegment, while BLOOM's block-based scheme yields the lowest typical recall.

6. Patterns, Implications, and Limitations

Several key patterns and caveats shape the practical interpretation of MorphScore boundary recall:

• Oversegmentation bias: Tokenizers that split words into excessively small units (character-level or byte-level strategies) attain artificially high recall, as every morpheme boundary is likely to be marked, but this does not reflect true morphological awareness.
• Predictive limitation: Higher recall weakly correlates, and even negatively correlates, with downstream modeling performance ($w$8; negative slope).
• Precision and F1 necessity: Recall alone is insufficient; it fails to penalize false positive boundaries. Comprehensive evaluation requires paired reporting with boundary precision and, ideally, subword-level F1.
• Dataset scope: Evaluation is limited to concatenative morphological systems. Non-concatenative and isolating languages are not scored, potentially limiting generalizability.
• Single-token cases and frequency scope: Inclusion of single-token words may inflate recall; frequency weighting depends on sometimes limited UD treebank statistics.

A plausible implication is that while boundary recall is interpretable and easy to compute, its diagnostic value for downstream model performance or overall tokenizer quality is limited; it addresses one dimension of morphological alignment but omits considerations such as compression, Rènyi efficiency, vocabulary coverage, and dynamic fine-tuning behaviors (Arnett et al., 8 Jul 2025).

7. Relationship to Broader Tokenization Evaluation

MorphScore boundary recall provides a linguistically motivated, reference-based evaluation criterion, but should be interpreted in the context of a broader suite of metrics. Compression efficiency, vocabulary coverage, and alignment with modeling objectives remain orthogonal concerns not addressed by boundary recall. To avoid rewarding pathological oversegmentation, precision and F1 should always be included in comprehensive tokenization evaluation. The metric is most suitable for diagnosing the alignment of tokenizers with concatinative affixal systems and should be used in concert with orthogonal criteria to form a robust assessment framework (Arnett et al., 8 Jul 2025).