Generalization of English-only findings about compression and performance

Ascertain the extent to which findings from English-only experiments on the relationship between tokenization compression and language model performance generalize to other languages, including typologically diverse and low-resource languages.

Background

Some controlled studies limited to English report no relationship between compression and performance, raising the question of whether such results hold for other languages with different scripts, morphology, and tokenization behavior.

Establishing crosslingual generalization is vital for equitable multilingual model development and for understanding whether tokenizer design choices should differ by language.