Characterize what makes a good tokeniser for language modelling

Determine which properties of subword vocabularies and the resulting tokenisations lead to strong downstream language modelling performance; specifically, identify the characteristics of the produced subwords that make a tokeniser an effective starting point for language modelling so that an explicit evaluation objective can be defined.

Background

The paper motivates the need to understand tokenisation quality because LLMs are trained over subword sequences rather than raw characters. Despite tokenisation’s centrality, there is no agreed-upon characterization of what constitutes a good tokeniser in terms of properties of the subwords it produces.

Clarifying these properties would enable the formulation of a principled objective function for evaluating and selecting tokenisers, beyond current practice that relies on proxy objectives.

References

We still do not know, for instance, what makes a good tokeniser \citep{gowda-may-2020-finding,cognetta-etal-2024-two}: which characteristics should its produced subwords $\subwords$ have to be a good starting point for language modelling?

— Tokenisation is NP-Complete  (2412.15210 - Whittington et al., 2024) in Section 1 (Introduction)

We do not measure other downstream task performance, reasoning ability, or long-context behaviour; how our models perform in those different settings is thus left open.

— Objective vs. Search: Decomposing What Makes a Good Tokeniser  (2609.19145 - Yavuz et al., 16 Sep 2026) in Limitations, Evaluation paragraph

Tokenization is unresolved: a syllable vocabulary of 35K types still leaves 15% of tokens unknown (Liu et al., 2022), whereas multilingual tokenizers fragment Tibetan, so that a 27-token English sentence takes 100 or more in Tibetan (Zhang et al., 2025a), and expanding byte-level byte-pair-encoding vocabularies helps only beyond about 3,000 new tokens per language (Han et al., 2026).

— Towards the Roof of the World: NLP Progress in Tibetan Languages  (2610.13398 - Cao et al., 9 Oct 2026) in Section 5.3, “The Transition to LLMs”