Measuring downstream language modelling performance without training a model

Develop a computationally feasible measure or estimator of downstream language modelling performance that can be used to evaluate or optimise tokenisers without fully training a language model.

Background

Ideally, tokenisers would be selected to maximise downstream performance, but direct evaluation would require fully training models, which is computationally prohibitive.

In practice, researchers resort to proxy objectives (e.g., unigram log-probability, Rényi efficiency, compression). A principled, efficient performance measure would enable direct optimisation aligned with end goals.