ConvexTok: Global LP Tokenisation

• ConvexTok is a tokenisation algorithm that frames segmentation as a global optimisation problem using linear programming, offering clear quantifiable optimality guarantees.
• It employs convex relaxation by replacing integer constraints with continuous variables, enabling scalable and near-optimal solutions with minimal integrality gap.
• Empirical results show ConvexTok outperforms traditional greedy methods like BPE in compression metrics and language model performance on large corpora.

ConvexTok is a tokenisation algorithm for NLP based on global convex optimisation, specifically linear programming (LP). Unlike traditional tokenisation methods such as byte-pair encoding (BPE) and UnigramLM which employ greedy, locally optimal merges or pruning strategies, ConvexTok frames tokeniser construction as a global optimisation problem. The algorithm solves an LP defined over a tokenisation graph, providing near-optimal tokenisers with quantifiable optimality guarantees and demonstrating improved compression and representation efficiency for LLMs (Tempus et al., 21 May 2026).

1. Linear Programming Formulation

ConvexTok originates from a formal integer programming (IP) formulation of the tokenisation problem, constructed on a directed acyclic tokenisation graph $G = (V, E)$. Each node represents a position in the concatenated dataset, and each edge corresponds to either a "free" byte step or a "priced" token step. Candidate tokens induce edge groups ("colours") $C$. Decision variables are binary vectors indicating which free edges ($f \in \{0,1\}^{E_\text{free}}$), priced edges ($p \in \{0,1\}^{E_\text{priced}}$), and token types ($c \in \{0,1\}^{|C|}$) are used. Incidence matrices ($A_\text{free}$, $A_\text{priced}$), a supply/demand vector ($d$), and an arc-to-colour assignment matrix ($M$) encode flow and token dependencies.

The integer program is: $$\begin{array}{ll} \min_{\,f,p,c} & \sum_{e\in E_\text{free}} f_e + \sum_{e\in E_\text{priced}} p_e \ \text{s.t.} & \begin{cases} A_\text{free} f + A_\text{priced} p = d \ p - M c \le 0 \ \mathbf{1}^T c \le K \ f \in \{0,1\}^{E_\text{free}},\, p \in \{0,1\}^{E_\text{priced}},\, c \in \{0,1\}^{|C|} \ \end{cases} \end{array}$$

Relaxation of integrality constraints to $C$0 intervals gives the ConvexTok LP: $C$1 This represents a convex hull over feasible discrete tokenisation configurations, indexed by the dataset and token vocabulary budget $C$2.

2. Convex Relaxation and Algorithmic Solution

The LP relaxation in ConvexTok replaces binary decisions with continuous variables in $C$3, forming an optimisation over a high-dimensional convex polytope. No auxiliary regularisers are required; all constraints remain linear. The convex relaxation enables the application of highly scalable first-order methods and convex solvers, in this instance the PDLP solver from NVIDIA's cuOpt library, which implements a primal–dual hybrid gradient method featuring presolving and adaptive step sizes.

On large corpora with vocabulary sizes up to $C$4k, the resulting LP contains approximately $C$5 million constraints and $C$6 million variables. Typical runtime is about $C$7 seconds to $C$8 relative accuracy on a single NVIDIA GH200 GPU, scaling from $C$9 seconds to $f \in \{0,1\}^{E_\text{free}}$0 seconds across six tested vocabulary budgets. For smaller vocabulary budgets, the LP becomes marginally more expensive to solve.

3. Certification of Optimality

ConvexTok's LP provides a lower bound $f \in \{0,1\}^{E_\text{free}}$1 on the best possible tokenisation objective (e.g., compression). After rounding the LP solution (via deterministic or biased rounding schemes), a discrete tokeniser with objective $f \in \{0,1\}^{E_\text{free}}$2 is extracted and evaluated. The integrality gap ratio,

$f \in \{0,1\}^{E_\text{free}}$3

quantifies suboptimality. Empirical results show that, for vocabulary sizes $f \in \{0,1\}^{E_\text{free}}$4k, this ratio remains within $f \in \{0,1\}^{E_\text{free}}$5 ($f \in \{0,1\}^{E_\text{free}}$6) of the LP lower bound, indicating near-optimality of ConvexTok-rounded tokenisers on the data.

Dual multipliers from the LP solution (dual variables for flow and budget constraints) provide an additional certificate of optimality, with the dual LP related by strong duality up to solver tolerances.

4. Empirical Performance and Metrics

Evaluations span both intrinsic and extrinsic criteria.

Intrinsic tokenisation metrics (measured on held-out ClimbMix, Flores+, and similar datasets):

• Vocabulary utilisation: $f \in \{0,1\}^{E_\text{free}}$7 to $f \in \{0,1\}^{E_\text{free}}$8 percentage points over BPE.
• Type-token ratio: $f \in \{0,1\}^{E_\text{free}}$9–$p \in \{0,1\}^{E_\text{priced}}$0.
• Shannon entropy (Rényi $p \in \{0,1\}^{E_\text{priced}}$1) and $p \in \{0,1\}^{E_\text{priced}}$2: mixed, with BPE slightly higher in some cases.
• Average token length, tokens per line, and compression rate: consistently improved.
• Bias rounding achieves best bits-per-byte (BpB) compression and vocabulary utilisation.

LLM performance (bits-per-byte, BpB):

• Decoder-only GPT-style transformers (12/18/24 layers) trained on ClimbMix400B, same hyperparameters across tokenisers.
• 12-layer, $p \in \{0,1\}^{E_\text{priced}}$3k–$p \in \{0,1\}^{E_\text{priced}}$4k: Deterministic rounding outperforms BPE on all but one configuration, with relative BpB improvement of $p \in \{0,1\}^{E_\text{priced}}$5–$p \in \{0,1\}^{E_\text{priced}}$6.
• 18-layer, $p \in \{0,1\}^{E_\text{priced}}$7k–$p \in \{0,1\}^{E_\text{priced}}$8k: Deterministic rounding is best at key vocabulary sizes, closely followed by bias rounding.
• 24-layer, $p \in \{0,1\}^{E_\text{priced}}$9k–$c \in \{0,1\}^{|C|}$0k: Bias and deterministic rounding match or beat BPE for larger vocabularies.

Downstream tasks (CORE, GLUE-style QA, commonsense, BBH, etc.):

• At 12 layers: ConvexTok deterministic rounding generally outperforms BPE, though certain tasks (e.g., BigBench CS–Algorithm) prefer BPE.
• At 18 layers: bias rounding and BPE often tie or show minor differences.
• At 24 layers: BPE has a slight advantage at $c \in \{0,1\}^{|C|}$1k vocabulary, but ConvexTok surpasses at $c \in \{0,1\}^{|C|}$2k and $c \in \{0,1\}^{|C|}$3k.
• Larger vocabularies tend to favour ConvexTok.

5. Comparison with Greedy Tokenisation Algorithms

BPE and UnigramLM implement greedy, local update rules: BPE greedily merges the most frequent byte pair at each step up to the vocabulary budget, and UnigramLM prunes tokens based on individual contributions. ConvexTok, in contrast, optimises a global segmentation cost subject to a vocabulary budget constraint over all candidates simultaneously.

Empirically, BPE-compressed models use approximately $c \in \{0,1\}^{|C|}$4–$c \in \{0,1\}^{|C|}$5 more bytes (higher BpB) and cannot certify their distance to optimality. ConvexTok provides a provable compression lower bound and marginally superior compression (about $c \in \{0,1\}^{|C|}$6 improvement). This comes at a computational cost: BPE executes in seconds, while ConvexTok's LP requires minutes to hours per corpus–vocabulary configuration, though the solution can be reused across experiments.

6. Limitations and Prospects for Advancement

Several limitations constrain the current ConvexTok framework:

• Scalability: The LP formulation scales as $c \in \{0,1\}^{|C|}$7 with dataset size and average sequence length, reaching $c \in \{0,1\}^{|C|}$8 variables on large corpora. High-end GPU resources (e.g., NVIDIA GH200) are necessary, and memory or wall time may become prohibitive.
• Rounding and Integrality: Existing deterministic, biased, and integer rounding approaches do not guarantee feasible or integral solutions without post-processing. Improvements in rounding or incorporation of cutting-plane algorithms may further reduce the integrality gap.
• Optimisation Objective: The present instantiation focuses on compression minimisation. Extensions to alternative objectives (such as unigram log-likelihood or Rényi efficiency) are theoretically straightforward but remain unexplored experimentally.
• Pretokenisation: ConvexTok currently presumes external word- or regex-based pretokenisation, preserving these boundaries. Joint learning or relaxation of such boundaries, as in SuperBPE, is identified as future work.

7. Summary and Research Significance

ConvexTok advances the theory and practice of NLP tokenisation by casting vocabulary selection and segmentation as a large-scale LP. This allows the construction of tokenisers with certifiable near-optimality for compression, outperforming greedy baselines in both intrinsic and bits-per-byte metrics and remaining competitive in downstream task performance. The methodology involves a one-time computational cost that is offset by substantial gains in representational efficiency. Open research questions concern improved scalability, generalisation of objectives, and fully integrated pretokenisation learning (Tempus et al., 21 May 2026).