Impact of linguistically aligned tokenization on model performance

Ascertain whether tokenizers that produce linguistically aligned tokens (e.g., morpheme-like or morphologically coherent units) lead to improved downstream language model performance compared to tokenizers that segment text into less linguistically aligned units.

Background

Unigram tokenizers can yield more linguistically aligned segments than BPE, but this paper finds Unigram has worse compression across languages, and prior work offers mixed evidence on performance implications.

Clarifying the effect of linguistic alignment on performance would guide tokenizer selection and design, especially for languages with complex morphology.