Measuring downstream language modelling performance without training a model

Develop a computationally feasible measure or estimator of downstream language modelling performance that can be used to evaluate or optimise tokenisers without fully training a language model.

Background

Ideally, tokenisers would be selected to maximise downstream performance, but direct evaluation would require fully training models, which is computationally prohibitive.

In practice, researchers resort to proxy objectives (e.g., unigram log-probability, Rényi efficiency, compression). A principled, efficient performance measure would enable direct optimisation aligned with end goals.

References

Unfortunately, we do not know how to measure such performance without fully training a model, making its direct maximisation computationally infeasible.

— Tokenisation is NP-Complete  (2412.15210 - Whittington et al., 2024) in Section 2 (How to Choose a Tokeniser?)

Tokenizers are a critical aspect of LLM design \citep{alqahtani2026stop}, but it is still not known how to measure whether a tokenizer is suitable for a particular model.

— No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance  (2610.21679 - Lopardo et al., 15 Oct 2026) in Section 1, Introduction