Quantify tokenizer effects across pretraining seeds

Determine whether the reported 1.48-point improvement of the Latent Core Tokenizer over BPE persists after accounting for variance across independent encoder-pretraining seeds.

Background

The downstream comparison uses seven encoders, but each encoder is pretrained only once with seed 42; the reported variation reflects fine-tuning seeds rather than pretraining variation. Consequently, the observed improvement of the Latent Core Tokenizer may partly reflect randomness in the pretraining run rather than a systematic tokenizer effect. Establishing the robustness of the gain requires repeating encoder pretraining with multiple independent seeds.

References

We therefore cannot exclude the possibility that part of the $1.48$-point improvement over BPE reflects pretraining variance. Replicating encoder pretraining across multiple seeds would provide a stronger estimate of the tokenizer effect.

— Latent Core Tokenizer: Compress, but Meaningfully  (2610.12376 - Ali et al., 8 Oct 2026) in Limitations, “Single pretraining seed”