Compression–performance relationship in language models

Determine how tokenization-driven compression relates to language model performance on downstream tasks across languages and tasks, explicitly assessing whether higher compression (fewer tokens for equivalent content) improves, harms, or has no effect on performance.

Background

The paper reviews conflicting evidence on whether tokenizer compression correlates with downstream performance: some studies report positive correlations, while others find no relationship when carefully controlling tokenizer properties.

Because compression directly affects training and inference costs and may influence the amount of information per sequence, establishing a clear relationship is important for both fairness and efficiency across languages.

References

How compression relates to LLM performance on downstream tasks remains uncertain.

— Explaining and Mitigating Crosslingual Tokenizer Inequities  (2510.21909 - Arnett et al., 24 Oct 2025) in Section 2.3 (Related Work — The Relationship Between Compression and Performance)

Given the superior compression ability of neural compressors, the natural question arises: do the improved compression ratios of neural compressors translate into improved predictive performance via compression distance? This is a new area of research, and there is much that is still unknown.

— An Introduction to Compression-Based Machine Learning  (2609.21309 - Hurwitz et al., 18 Sep 2026) in Section 4.2, “When is compression-based similarity meaningful?”

Grouping digits may harm arithmetic in downstream models, which remains to be tested.

— The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype  (2609.39001 - Serval, 30 Sep 2026) in Section 5, subsection “A French-optimized prototype,” paragraph “Construction”

We state these results as segmentation results only: their effect on model quality and task cost remains to be demonstrated.

— The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype  (2609.39001 - Serval, 30 Sep 2026) in Abstract

The reasons behind the sensitivity of these measurements is an investigation we leave for future work.

— Language-Specific Effects of Tokenizer Choice in Multilingual Language Models  (2610.12144 - Meister et al., 8 Oct 2026) in Appendix, Section “Intrinsic Metrics: Definitions and Additional Results,” subsection “Controlling for confounders,” paragraph “Per-language correlations given the language's training bytes.”