Hardness for alternative objectives and tokenisation variants beyond direct and bottom-up

Establish the computational hardness (decision and approximation) of tokenisation when the optimisation objective is not compression and for tokenisation variants other than direct encoding and bottom-up encoding, in particular by classifying the complexity and approximability of these alternative objectives and variants over bounded alphabets.

Background

Throughout the paper the analysis focuses on compression as the objective and on two variants—direct and bottom-up tokenisation. While these choices cover common settings, other objectives (such as those used by probabilistic tokenisers) and other tokenisation paradigms were not analysed.

The authors explicitly state that determining the hardness for both alternative objectives and other tokenisation variants remains an open direction, highlighting a gap in the current complexity landscape of tokenisation.