Effect of baseline language choice in parallel-data tokenizer comparisons

Determine whether using baseline languages other than English when comparing tokenizers trained on parallel data changes the observed compression outcomes and conclusions about token premium effects.

Background

Due to data availability, the paper compares each target language only against an English baseline for parallel-data-trained tokenizers, noting this constraint may limit cross-language comparability.

Understanding whether baseline choice affects observed compression differences would strengthen conclusions about the efficacy of training on parallel data.

References

It is unclear whether comparisons with different languages would lead to different results.

— Explaining and Mitigating Crosslingual Tokenizer Inequities  (2510.21909 - Arnett et al., 24 Oct 2025) in Section 8 (Limitations)

Counts were reproduced on two machines, from scratch on a fresh environment, and by an independent AI-assisted reproduction from the released files (final-test counts, CroissantLLM and Claude corrections; the later controls in Table~\ref{tab:controls} remain to be reproduced externally).

— The Invisible Language Tax: Token Premiums of French and Regional Languages in 2026 LLM Tokenizers, and a French-Optimized Prototype  (2609.39001 - Serval, 30 Sep 2026) in Section 5, subsection “A French-optimized prototype,” paragraph “Benchmark status”

We have not tested whether they reproduce under a different extraction pipeline, web snapshot, or corpus family.

— Neurograf v1: An End-to-End Romanian-First Language Model Family  (2610.13134 - Rotaru, 9 Oct 2026) in Section 3.10, “A single corpus pair”