Tokenisation via Convex Relaxations
Abstract: Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions without considering the resulting vocabulary as a whole. We instead formulate tokeniser construction as a linear program and solve it using convex optimisation tools, yielding a new algorithm we call ConvexTok. We find ConvexTok consistently improves intrinsic tokenisation metrics and the bits-per-byte (BpB) achieved by LLMs; it also improves downstream task performance, but less consistently. Furthermore, ConvexTok allows the user to certify how far their tokeniser is from optimal, with respect to a certain objective, via a lower bound, and we empirically find it to be within 1\% of optimal at common vocabulary sizes.
Summary
- The paper introduces ConvexTok, a method that formulates tokenisation as an integer program and relaxes it to a linear program for tractable optimisation.
- The paper demonstrates that ConvexTok achieves compression performance within a 1% gap of the LP lower bound at larger vocabulary sizes.
- The paper shows that convex relaxation-based tokenisers yield superior vocabulary utilisation and robust out-of-distribution performance compared to traditional greedy methods.
Convex Relaxations for Globally Optimal Tokenisation
Introduction and Motivation
Tokenisation is fundamental to NLP pipelines, transforming byte sequences into token sequences ingested by LMs. State-of-the-art tokenisers such as BPE and Unigram use greedy, locally optimal strategies that fail to consider the global compression or utility of the resultant vocabulary. Empirical research has indicated that tokeniser compression correlates with downstream LM performance, motivating a search for globally optimal or near-optimal tokenisation. However, previous work has established that optimal tokenisation is NP-hard [whittington-etal-2025-tokenisation], prompting the need for scalable approximation algorithms.
The paper "Tokenisation via Convex Relaxations" (2605.22821) proposes a novel formalisation: casting tokenisation as an integer program (IP) and subsequently relaxing it to a linear program (LP) solvable by standard convex optimisation techniques. The resulting approach, ConvexTok, provably upper-bounds tokenisation compression error and supports empirical certification of proximity to optimality.
Problem Formulation: Graph-Theoretic and Polyhedral Perspectives
The authors first recast tokenisation as a segmentation problem on a directed acyclic graph (DAG) constructed from a dataset. Each byte-string corresponds to a path with vertices for every character position, and edges represent possible token boundaries, with edge "colours" associated to possible subword substrings (tokens). Imposing a budget corresponds to selecting token "colours"—i.e., fixed-size vocabularies—while requiring that segmentation paths from start to end use only available tokens or byte-level tokens.
This construction admits a natural integer program: binary variables encode token selection and segmentation choices, subject to flow and coverage constraints ensuring valid decompositions. The path length across the graph equates to encoded token sequence length, i.e., the compression objective.
Convex Relaxation and Rounding
Because the IP is computationally intractable for large-scale data, the relaxation moves all binary variables to the interval, yielding a tractable LP over the "tokenisation polytope". The LP is solved via large-scale convex solvers (notably, NVIDIA's CuOPT implementation of PDLP), returning potentially fractional token assignments. Therefore, the output must be discretised via a rounding scheme to produce a valid tokeniser. The authors propose three rounding methods:
- Deterministic (Det): Top- selections by score;
- Bias: Top- after biasing towards shorter tokens (more robust for OOD strings);
- Integral-only (Int): Selects only variables essentially at 1; others dropped.
This strategy transforms the LP solution into a discrete vocabulary and segmentation procedure.
Experimental Methodology
Experiments employ the ClimbMix400B corpus following nanochat pre-tokenisation and train tokenisers at vocabulary sizes from 8k to 256k. Both intrinsic (compression, vocabulary utilisation, entropy measures, etc.) and extrinsic (bits-per-byte (BpB), and downstream LM performance on CORE tasks) evaluations are performed. LMs follow GPT-style architectures, with hyperparameters controlled across vocabulary sizes and tokenisers.
Analytical Results
Solving the Relaxation and Integrality
The LP relaxation admits solutions exceptionally close to integrality, especially at large vocabulary sizes. At , the gap between convex lower bound and rounded integer solutions is consistently under 1%, empirically demonstrating the tightness of the relaxation. The number of variables fixed at 0 or 1 dominates as increases, reducing ambiguity for rounding.
Compression and Certifiable Bounds
After rounding, Det tokenisers nearly match the LP lower bound at all sizes. Notably, even the greedy BPE baseline achieves high compression efficiency (typically within 0.7–3% of the lower bound), but ConvexTok-Det consistently outperforms or matches BPE across all sizes in raw compression (Figure 1).
Figure 1: Compression performance, in both absolute and relative terms to BPE, across tokeniser algorithms and vocabulary sizes.
Vocabulary Utilisation and Stability
Bias and Det outperform BPE across vocabulary utilisation and type-token ratio metrics (Figures 4, 5), indicating better exploitation of the vocabulary space, which can be advantageous in downstream and OOD settings. However, BPE demonstrates notably higher vocabulary stability to random sampling (higher average Jaccard similarity) due to its frequency-based, highly local merge process (Figure 2).
Figure 3: Vocabulary utilisation of different tokenisers, both absolute and relative to BPE.
Figure 4: Type-token ratio for varying tokenisers, illustrating the improved diversity of ConvexTok vocabularies.
Figure 2: Jaccard similarity of vocabularies across independently sampled data subsets, revealing robustness to sampling variance.
Distributional Measures and OOD Generalisation
ConvexTok variants generally yield more uniform and higher-entropy token distributions (Figure 5) and maintain higher vocabulary utilisation OOD. This is manifest in improved compression and vocabulary coverage in the FLORES+ dataset and other multilingual benchmarks.
Figure 5: Shannon entropy of token distributions, demonstrating linguistic diversity and reduction of degenerate token usage.
LM Performance: Bits-per-Byte and Downstream Tasks
On BpB, ConvexTok-Det always matches or outperforms BPE across model depths and vocabulary sizes. For CORE and other reasoning-focused downstream tasks, ConvexTok and BPE are competitive, with ConvexTok showing greater improvements at larger vocabulary sizes, especially for Det and Bias. Importantly, BpB and intrinsic metrics are more sensitive to tokenisation than certain task metrics, reinforcing the link between compression and LM data efficiency established by prior work [galle-2019-investigating, goldman-etal-2024-unpacking].
Theoretical Implications and Connections
- Polyhedral Characterisation: The formulation enables the use of mature LP techniques and directly inherits theoretical results from combinatorial optimisation and network flow literature [LPtextbook, MaxFlowPolytope]. While tokenisation on general graphs lacks total unimodularity (thus permitting fractional vertices), empirical integrality closes the practical gap.
- Certifiable Near-Optimality: The LP lower bound provides a provable certificate of distance to optimality for any tokeniser, unique among tokeniser learning methods.
- Generalisability: The technique extends naturally to non-compression objectives (e.g., Renyi entropy), custom token candidate sets, and settings where pre-tokenisation is relaxed or absent.
Limitations and Prospects
The framework currently optimises for in-sample compression; further work is required to extend the methodology to OOD generalisation, non-compression objectives (e.g., discriminative or semantic priors), or language-specific constraints as in multilingual LMs. Analysis of the polyhedral structure in settings with high extension complexity may elucidate theoretical limits on what tokenisation gains are extractable by convex relaxation [No-small-LP, extension-complexity-no].
Likewise, practical improvements could be sought in hybrid approaches (e.g., branch-and-bound with LP relaxation [BranchAndBround]), non-pretokenised settings (as in BoundlessBPE [schmidt2025boundlessbpe] and SuperBPE [liu2025superbpe]), and dynamic or multi-granular token vocabularies.
Conclusion
"Tokenisation via Convex Relaxations" rigorously demonstrates that globally optimal or near-optimal tokenisers can be derived via tractable convex relaxations, with rounding yielding tokenisers that consistently outperform strong greedy baselines on compression and intrinsic linguistic metrics. The convex relaxation framework introduces a new class of tokeniser algorithms that are certifiably close to theoretical optima and are extensible to broad new objective functions and settings. The implications are substantial: as the field pushes LMs to new data and task regimes, robust, globally optimal tokenisation may be a pivotal tool in maximising model data efficiency and generalisation (2605.22821).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about “tokenization,” a step used before training and using LLMs. Tokenization breaks text into small pieces called tokens, like splitting a sentence into words or word parts. The most common method today, called BPE (Byte Pair Encoding), makes lots of small, local choices and does not always produce the best set of tokens overall.
The authors introduce a new method called ConvexTok. Instead of making local guesses, they turn tokenization into a math problem that can be solved more globally. They use tools from optimization (a way to find the best solution) to get token sets that compress text better and often help LLMs perform better. ConvexTok also tells you how close your token set is to perfect, which older methods can’t do.
Key Questions
The paper tries to answer simple but important questions:
- How can we build a tokenizer that compresses text as much as possible, so LLMs need fewer tokens to understand it?
- Can we move beyond greedy methods like BPE to find token sets that are closer to the global best?
- Can we measure how far our tokenizer is from the best possible one?
- Do better-compressing tokenizers improve LLM training and downstream tasks?
Methods and Approach
Think of tokenization like crossing a river by stepping on stones:
- Each stone represents a position in a string of text.
- You can take short steps (single characters/bytes) or longer jumps (multi-character tokens).
- A good tokenizer finds a path with the fewest steps to cross all the text—this is like “compressing” the text into fewer tokens.
Here’s how ConvexTok works, step by step:
- Build a graph of the text: The text is turned into a map. Positions between characters become nodes (stones), and edges connect nodes. Short edges are single characters; long edges are candidate tokens.
- Color the edges by token type: Every possible token (like “the” or “ing”) gets a color, and all edges that represent that token share the same color. Picking up to K colors means you choose up to K tokens for your vocabulary.
- Turn it into a global math problem: The authors translate this graph into an “integer program” (IP). That’s a fancy way of saying “choose exact yes/no decisions for edges and colors to minimize the total number of steps.” IPs are very hard to solve exactly for big problems.
- Relax it to a “linear program” (LP): Instead of yes/no, allow fractional choices between 0 and 1 (like “70% chosen”). LPs are much easier to solve fast. This gives a near-best global plan but uses “partial tokens.”
- Round back to real tokens: Because real vocabularies can’t have fractional tokens, they convert fractional scores into actual choices. They test three rounding styles:
- Deterministic (Det): Pick the top K highest-scoring tokens. Simple and stable.
- Biased (Bias): Prefer tokens that are both high-scoring and shorter. Shorter tokens are more likely to be useful everywhere, not just in the training text.
- Integral-only (Int): Only keep tokens that the LP almost fully selected (values ~1). This can return fewer than K tokens but picks the “most certain” ones.
- Certify optimality: The LP gives a lower bound on the best possible compression. That means you can say, “No tokenizer can beat this bound,” and measure how close your actual tokenizer is to perfect.
Main Findings
The authors run experiments across many vocabulary sizes (from about 8k to 256k tokens) and compare ConvexTok to BPE. They train small-to-mid-size GPT-style models on a high-quality dataset (ClimbMix400B) and evaluate intrinsic metrics (like compression) and downstream tasks.
Key results:
- Better compression and bits-per-byte: ConvexTok usually compresses text more than BPE and often improves the bits-per-byte (BpB) achieved by LLMs. Det rounding tends to perform best on BpB.
- Close to optimal: ConvexTok provides a guaranteed lower bound on the best compression. Their tokenizers are typically within about 1% of this bound at common vocabulary sizes. Even BPE is surprisingly close but not quite as good.
- Rounding trade-offs:
- Bias rounding consistently wins on intrinsic tokenization metrics (like compression and entropy).
- Det rounding wins on model-level BpB and often helps downstream tasks.
- Int rounding, though it may return fewer tokens than allowed, still does fine at large vocabulary sizes because many “critical” tokens have already been selected.
- Stability: When training on different random samples of the data, BPE’s vocabularies are more stable than ConvexTok’s. This likely happens because BPE favors very frequent merges, which don’t change much with resampling. Int rounding is more stable than Bias or Det because it keeps only the most certain tokens.
- Scaling helps “integrality”: As vocabulary size grows, the LP solutions look more like exact (0/1) choices, meaning the problem becomes less ambiguous and rounding matters less.
Why This Matters
- More efficient models: If your tokenizer compresses better, your model processes fewer tokens for the same text. That can make training and inference faster and cheaper, or let you pack more content into a fixed context window.
- Better performance: Improving bits-per-byte often correlates with better language modeling. ConvexTok boosts BpB and sometimes improves downstream tasks, which matters for real-world applications.
- Measurable progress: ConvexTok tells you how close you are to the best possible tokenizer for your data. This “certificate of optimality” is rare and valuable—it lets researchers and engineers trust that they’re near the theoretical best, instead of guessing.
- Flexible and general: The method can include or exclude candidate tokens based on rules (like respecting boundaries in text or focusing on frequent pieces). It can also adapt to different objectives if future research finds better ones than compression.
Simple Takeaway
Traditional tokenizers like BPE make good local moves but don’t always get the best global result. ConvexTok uses smarter math to look at the whole picture, finds nearly optimal token sets, and proves how close they are to perfect. This often leads to better compression and efficiency for LLMs, and it gives researchers tools to understand and improve tokenization in a principled way.
Knowledge Gaps
Unresolved knowledge gaps, limitations, and open questions
Below is a concise list of what remains missing, uncertain, or unexplored in the paper, phrased to guide actionable future work:
- Objective misalignment with downstream performance: The method optimizes compression only; it is unclear how to formulate and solve LPs for alternative objectives (e.g., LM negative log-likelihood, Rényi efficiency) or multi-objective trade-offs that better predict downstream task performance.
- Limited baselines: Comparisons are primarily to standard BPE; there is no empirical comparison to Unigram (SentencePiece), WordPiece, or recent non-greedy BPE variants (e.g., trimming, scaffold, boundless), leaving the relative advantage of ConvexTok underexplored.
- Downstream inconsistency unexplained: ConvexTok improves BpB but yields less consistent gains on downstream tasks; the causal factors (e.g., token distribution shifts, training hyperparameters, tokenization granularity) are not diagnosed.
- Pre-tokenization dependence: All experiments rely on a regex pre-tokenizer; the impact of removing or varying pre-tokenization (and allowing tokens to cross boundaries) on both compression and downstream performance is not evaluated.
- Generalization beyond training distribution: Tokens are optimized on a specific dataset; robustness to out-of-domain text, other domains (code, math, noisy web), and multilingual/cross-script data remains untested.
- Rounding scheme design and guarantees: Only three simple rounding schemes (Det, Bias, Int) are explored; there are no theoretical approximation guarantees, analyses of worst-case behavior, or evaluations of more advanced rounding (e.g., randomized, pipage, dependent rounding).
- Sensitivity of Bias rounding: The length-normalized score used by Bias is heuristic; the choice of normalization, tie-breaking, and potential hyperparameters are not ablated for their effects on compression and downstream performance.
- Int rounding under-utilizes budget: Int can return fewer than K tokens, but there is no strategy to fill the remaining budget (e.g., fallback selection rules) or evaluation of the associated performance trade-offs.
- Polytope/integrality theory: The paper provides empirical integrality gaps but no theoretical analysis of the tokenization polytope (e.g., facet structure, conditions for total unimodularity, bounds on integrality gaps, or data conditions under which the LP tends to be integral).
- Solver tolerances and numerical accuracy: The LP is solved with a relatively loose primal-dual gap tolerance (reported as 10,000), raising questions about how solver precision affects the lower bound, rounding decisions, and final performance.
- Scalability and memory footprint: The LP instance is extremely large (≈106M variables, ≈99M constraints), solved on high-end GPUs; scalability to larger corpora, bigger candidate sets, and typical hardware is not characterized, nor are memory/time trade-offs.
- Candidate set selection and pruning: While the framework permits restricting priced edges (e.g., by frequency or boundary constraints), the sensitivity of results to candidate pruning strategies is not studied; column generation or on-the-fly candidate discovery is not explored.
- Stability vs. performance trade-off: ConvexTok vocabularies are less stable than BPE across resampled training subsets; it is unknown how to regularize the LP (e.g., L1/L2 penalties, entropy, diversity constraints, robust optimization) to improve stability without sacrificing performance.
- PathPiece for unseen text: For inputs outside the training dataset, encoding uses PathPiece; the impact of this mismatch (LP-trained tokens + PathPiece inference) on compression and downstream performance is not quantified.
- Constraints beyond compression: The LP can encode constraints (e.g., no crossing grapheme boundaries, reserved percentages for languages/scripts), but such fairness or structural constraints are not instantiated or empirically evaluated.
- Joint optimization with LM training: The method is offline; there is no exploration of bilevel/joint training where LM gradients inform token choices (e.g., alternated LP solving and LM updates), nor analysis of compute efficiency and convergence.
- Compute-fair training setup: Model hyperparameters are fixed and chosen using a BPE baseline; how retuning per-tokenizer or per-vocabulary size affects downstream performance comparisons is not examined.
- Tokenization speed and runtime costs: Although inference is said to be “similar to Unigram,” there are no quantitative tokenization latency/throughput measurements, nor assessments of memory use and deployment overhead.
- Cross-language and morphology sensitivity: Experiments focus on ClimbMix400B (English-heavy); effects on languages with complex morphology, different scripts, and diverse token length distributions are unknown.
- Conditions for BPE near-optimality: Empirically, BPE is close to LP lower bounds; the dataset properties (e.g., power-law token distributions, pre-tokenization constraints) that make greedy merges near-optimal are not characterized or theoretically justified.
- Optimal K selection: A “saturation effect” is suggested (diminishing returns beyond ~128k), but the interaction between vocabulary size, model size/compute budget, and downstream performance is not systematically profiled to guide optimal K choices.
- Special tokens and semantics: How to handle special tokens with non-textual semantics (e.g., control tokens) within the LP (e.g., costs, reserved slots) and how such handling affects performance is not detailed.
- Exact solutions for small instances: Although a branch-and-bound approach is mentioned in the appendix, there is no systematic use of exact solutions on small datasets to validate the LP’s tightness and rounding’s optimality.
- Robustness to domain and temporal shift: The method’s performance under domain shifts (e.g., pretrain vs. downstream domains) and over time (continual data updates) is untested; strategies for periodic re-optimization or stable updates are unaddressed.
- Multi-objective extensions: Incorporating penalties for poor vocabulary utilization, token length distributions, or entropy targets directly into the LP (and evaluating their downstream implications) is not explored.
Practical Applications
Immediate Applications
The following items describe concrete, deployable uses of the paper’s findings, along with sector tags, potential tools/workflows, and feasibility notes.
Industry
- Optimized tokenizer replacement for current BPE/Unigram in LLM training and serving (software/AI)
- What: Swap in ConvexTok (Det or Bias rounding) to reduce bits-per-byte (BpB) and tokens per input, improving throughput and lowering compute and API costs in pretraining, finetuning, and inference.
- Tools/Workflows: Use the released ConvexTok code and Hugging Face tokenizers; inference uses shortest-path segmentation (Unigram-like), so runtime is a drop-in replacement; integrate into data/tokenizer build steps before model training.
- Assumptions/Dependencies: Access to the training corpus and to GPU or CPU LP solver capacity for one-time build; benefits vary by domain and vocabulary size; downstream win is not guaranteed even when BpB improves.
- Domain-specific tokenizers with enforceable boundary rules (software, code, legal, finance, healthcare)
- What: Define priced/free edges to constrain tokens (e.g., prevent crossing Unicode, grapheme, morpheme, or PII boundaries; handle numeric/decimal/currency formats; preserve code structure). Improves robustness and interpretability while maintaining near-optimal compression.
- Tools/Workflows: Add boundary filters (Unicode/grapheme/morpheme/regex) to candidate-edge generation; solve LP; apply Det/Bias rounding.
- Assumptions/Dependencies: Requires curated rule sets per domain; modest engineering to generate/prune candidate edges.
- Tokenizer certification and benchmarking in ML production (software/AI, MLOps)
- What: Use the LP value as a provable lower bound to certify how close a tokenizer is to compression-optimal and to establish “tokenizer quality SLAs” during model release or vendor evaluation.
- Tools/Workflows: Build a tokenizer QA dashboard that logs LP lower bound, integrality gap, vocabulary utilization, Rényi entropy, and stability; attach a certification report to model cards.
- Assumptions/Dependencies: The LP is solved on the actual training corpus (or a representative sample); numerical solver tolerances are documented.
- Cost/latency trade-off selection of vocabulary size (AI infra, product)
- What: Use near-optimal compression certificates to choose vocabulary size that minimizes tokens per input subject to embedding/memory constraints; helps product teams pick vocab sizes for different SKUs (edge vs cloud).
- Tools/Workflows: Add an automated sweep over K (vocab budget), solve LP once per K, plot integrality gaps vs memory cost; pick Pareto-optimal points.
- Assumptions/Dependencies: Hardware constraints (embedding table memory) and serving targets; benefits saturate at larger vocabularies.
- Safer tokenization for sensitive data (privacy/security; healthcare/finance)
- What: Exclude PII patterns (phone numbers, IDs) from candidate tokens so they don’t become single tokens; reduces memorization risk and makes redaction easier.
- Tools/Workflows: Regex/PII-detector-guided pruning of priced edges; re-run ConvexTok; validate coverage with held-out sensitive templates.
- Assumptions/Dependencies: Quality of PII detection; potential small compression loss due to constraints.
Academia
- Reproducible tokenizer studies with provable optimality gaps (NLP/ML)
- What: Use LP lower bounds to report and compare tokenizers with certificates; enables rigorous ablations on pretokenization, rounding, and candidate set choices.
- Tools/Workflows: Integrate ConvexTok into experimental pipelines; publish gaps, stability (Jaccard), BpB, and Rényi metrics alongside downstream tasks.
- Assumptions/Dependencies: Compute to solve LPs on representative corpora; clear reporting of solver tolerances.
- Teaching module connecting NLP and polyhedral optimization (education)
- What: Classroom labs using the tokenization polytope to illustrate integer programming, relaxations, and rounding schemes with measurable impacts on NLP.
- Tools/Workflows: Provide curated mini-corpora, solver notebooks (PDLP or CPU LP solvers), and visualization of graph segmentation and rounding.
- Assumptions/Dependencies: Scaled-down datasets for classroom compute budgets.
Policy and Governance
- Documentation standards that include tokenizer optimality certificates (government, enterprise procurement)
- What: Require vendors to disclose tokenizer integrality gaps and stability metrics in AI system documentation to support energy/cost claims and reproducibility.
- Tools/Workflows: Add a “Tokenizer Certification” section to model cards and procurement templates; audit with reproducible LP runs on provided samples.
- Assumptions/Dependencies: Standardized corpora or representative samples for verification; agreed solver settings.
- Language equity and accessibility constraints (public sector, localization)
- What: Enforce grapheme/morpheme boundary constraints per language to avoid unfair compression disparities; monitor vocabulary utilization across languages.
- Tools/Workflows: Language-specific boundary rule sets; ConvexTok runs per language; publish cross-lingual metrics.
- Assumptions/Dependencies: Availability of robust language-specific boundary detectors; potential minor compression trade-offs.
Daily Life and Developer Productivity
- Better community tokenizers for open-source finetuning and prompts (software, creators)
- What: Drop-in ConvexTok tokenizers for common domains (coding, math, legal writing) to reduce token counts and costs for hobbyist finetuning and API usage.
- Tools/Workflows: Use the provided Hugging Face tokenizers; pick Det for BpB gains or Bias for stronger intrinsic metrics; validate with your dataset slice.
- Assumptions/Dependencies: Local GPU not required for inference; small one-time build if customizing.
- Domain-tailored assistants with improved parsing of numbers, code, and symbols
- What: Constraint-aware tokenizers for numerics and code make small assistants more reliable and cheaper to run for note-taking, budgeting, and scripting.
- Tools/Workflows: Simple rule sets for numbers/currency/code formatting; run ConvexTok; deploy in local tools or plugins.
- Assumptions/Dependencies: Some user effort to define boundary rules; modest gains depend on domain skew.
Long-Term Applications
These opportunities likely need further research, scaling, or engineering before broad deployment.
Industry
- Adaptive or per-tenant tokenization (software/SaaS, enterprise)
- What: Periodically re-optimize tokenizers on each tenant’s corpus to reduce costs and improve quality; serve with a standardized segmentation algorithm.
- Potential Products: “Tokenizer-as-a-Service” that computes LP lower bounds, retrains with constraints, and delivers signed artifacts and reports.
- Dependencies/Risks: Operational complexity of multiple vocabularies; caching and model-embedding synchronization; governance for user data.
- Multi-objective and constrained tokenization (software, safety, compliance)
- What: Extend the LP to include side constraints (e.g., fairness across languages, stability regularizers, safety filters) or multi-objective formulations blending compression with Rényi efficiency or domain coverage.
- Potential Tools: Constraint authoring DSL, rule libraries, and solver backends with trade-off visualization.
- Dependencies/Risks: Formulating tractable constraints; proving guarantees; potential loss in compression if objectives conflict.
- Co-optimization of tokenizer and model during training (AI research/infra)
- What: Periodic ConvexTok refreshes as the corpus or loss evolves, or joint scheduling of vocabulary size with scaling laws to minimize total training/inference cost.
- Potential Workflows: Alternating optimization cycles (solve LP → train → rescore → update constraints); continuous validation on downstream tasks.
- Dependencies/Risks: Training instability when changing vocabularies; embedding remapping; checkpoint compatibility.
Academia
- Theoretical guarantees and new rounding schemes with performance bounds
- What: Design rounding methods with provable approximation ratios for compression and perhaps for downstream proxies; analyze integrality gaps under realistic corpora.
- Potential Outcomes: Benchmarks of gap behaviors, datasets where LP is integral, and principled choices among Det/Bias/Int or hybrids.
- Dependencies/Risks: Hardness persists; trade-offs between guarantees and practical performance.
- Generalized sequence tokenization beyond text (bio, time-series, logs)
- What: Apply the graph/LP framework to protein sequences, event logs, and sensor streams to find compressive “subsequence tokens,” enabling better modeling and compression.
- Potential Products: Domain token packs for bioinformatics and observability platforms.
- Dependencies/Risks: Pre-tokenization analogs and boundary rules are domain-specific; mapping to downstream metrics may differ from text.
Policy and Governance
- Standardized certification and auditing protocols for tokenizer efficiency
- What: Third-party services that verify integrality gaps, rule adherence (e.g., PII constraints), and cross-lingual equity, issuing compliance badges for foundation models.
- Potential Tools: Open verification suites, public leaderboards that include certified gaps.
- Dependencies/Risks: Agreement on benchmark corpora and solver settings; confidentiality of training data.
- Guidance on language-specific boundary policies to reduce bias
- What: Establish baseline boundary rules per language and domain to mitigate tokenization-induced disparities in cost and quality (e.g., for under-resourced languages).
- Potential Products: Open rule repositories and policy templates.
- Dependencies/Risks: Community consensus; ongoing maintenance; measuring impact on fairness and usability.
Daily Life and Developer Productivity
- On-device and edge LLMs with tokenization tailored for energy/latency
- What: Use near-optimal tokenizers to reduce token counts and energy per query on phones, robots, or cars where latency and battery matter.
- Potential Products: Edge model bundles with certified tokenizers for speech-to-text or copilots.
- Dependencies/Risks: Balancing larger embedding tables vs fewer tokens; hardware-specific profiling.
- Intelligent prompt budgeting and compression-aware authoring tools
- What: Editors that estimate token savings given different tokenizers and suggest rewrites or formatting to reduce token counts ahead of API calls.
- Potential Tools: IDE/editor plugins that use tokenizer statistics and LP-derived bounds to forecast costs.
- Dependencies/Risks: UX integration; user adoption; smaller per-user benefits may require clear value messaging.
Notes common to many applications:
- Rounding choice matters: Det often best for BpB; Bias often best on intrinsic metrics; Int is conservative and may underuse the budget but performs well at large vocabularies.
- Stability is lower for ConvexTok than BPE under resampling; mitigation includes larger datasets, rule-based candidate pruning, or stability-aware constraints in future work.
- Gains saturate at very large vocabularies; embedding memory and hardware constraints must be considered.
- LP solving at scale benefits from GPU-accelerated PDLP (e.g., NVIDIA CuOPT), but smaller corpora or candidate pruning can enable CPU-only runs.
Glossary
- Biased rounding (Bias): A rounding heuristic that prioritizes tokens based on LP scores adjusted by token length, favoring shorter tokens when scores are similar. "one of our rounding schemes (Bias) consistently outperforms all other tokenisers."
- bits-per-byte (BpB): A compression metric measuring the average number of bits required per input byte after tokenization/modeling. "the bits-per-byte (BpB) achieved by LLMs"
- BoundlessBPE: A BPE variant that constrains merges based on linguistic boundaries to avoid crossing certain units (e.g., unicode/grapheme). "analogously to BoundlessBPE;"
- byte-edge: An edge in the tokenisation graph connecting adjacent byte positions, representing single-byte transitions. "we connect each adjacent vertex within a byte-string with what we call a #1{byte-edge}."
- byte-pair encoding (BPE): A greedy tokenization algorithm that iteratively merges the most frequent token pairs to build a vocabulary. "the de-facto standard tokenisation algorithm is byte-pair encoding"
- colour-partition: A partitioning of token-edges by the byte sequence they represent, grouping all edges corresponding to the same candidate token. "We can then define a colour-partition"
- compression-optimal: Achieving the best possible compression (minimal tokenized length) for a given objective and vocabulary. "compression-optimal for a fixed vocabulary."
- convex optimisation: Optimization over convex sets/functions, here used to solve the LP formulation efficiently. "solve it using convex optimisation tools"
- convex relaxation: Replacing discrete constraints with continuous ones to obtain a solvable convex (linear) program that approximates the original integer problem. "use a convex relaxation to transform it into an LP"
- deterministic rounding (Det): A rounding method that selects the top-K tokens by their LP scores deterministically. "a deterministic rounding scheme (Det) consistently performed best on bits-per-byte (BpB)"
- directed acyclic graph (DAG): A directed graph with no cycles; used to model tokenization as shortest paths from start to end. "shortest path problem in a directed acyclic graph"
- edge-colour matrix: A binary matrix indicating which priced edges belong to which colour (token) classes. "we define an edge-colour matrix"
- flow-constraints: Constraints enforcing conservation of flow across vertices so that selected edges form a valid path/segmentation. "Finally, we introduce flow-constraints"
- flow-difference vector: A vector specifying net flow at vertices (−1 at start, +1 at end, 0 elsewhere) to enforce a single path. "we define a flow-difference vector"
- free edges: Edges always available without counting against the token budget (e.g., byte-edges), though still contributing to path length. "a set of free edges"
- generalised tokenisation graph: An extension of the tokenisation graph that distinguishes “free” and “priced” edges and partitions priced edges by colour. "We define a generalised tokenisation graph"
- graph-segmentation: A path from start to end in the tokenisation graph using only free edges and edges from selected colours. "a graph-segmentation as a path"
- graph-vocabulary: The subset of colours (tokens) chosen from the colour-partition for use in segmentation. "we define a graph-vocabulary"
- incidence matrix, free: The vertex–edge incidence matrix for free edges, encoding edge directions relative to vertices. "a #1{free incidence matrix}"
- incidence matrix, priced: The vertex–edge incidence matrix for priced edges, encoding edge directions relative to vertices. "a #1{priced incidence matrix}"
- integral-only rounding (Int): A rounding strategy that keeps only those tokens whose LP variables are essentially 1, possibly yielding fewer than K tokens. "Finally, #1{integral-only rounding (Int)} keeps only the colours"
- integer program (IP): An optimization problem with integer (often binary) decision variables; here, the exact tokenization formulation. "We first identify an integer program (IP)"
- Jaccard similarity: A set similarity metric defined as intersection over union; used to compare vocabularies. "compare their vocabularies using the Jaccard similarity"
- Kleene plus: In formal language theory, the set of all non-empty concatenations over an alphabet. "$\alphabet^+$ denotes its Kleene plus"
- Kleene star: In formal language theory, the set of all finite-length strings (including empty) over an alphabet. "$\alphabet^*$ denotes the Kleene star of $\alphabet$"
- linear program (LP): An optimization problem with linear objective and constraints over continuous variables; the relaxed form of the IP. "we relax this problem into a linear program (LP)"
- NVIDIA CuOPT: A GPU-accelerated optimization library used to solve large-scale linear programs efficiently. "implemented in the NVIDIA CuOPT library"
- PathPiece: A decoding/segmentation algorithm that is compression-optimal for a fixed vocabulary on unseen strings. "using PathPiece"
- PDLP: A primal–dual algorithm for linear programming used by CuOPT to solve the LP efficiently. "We solve the LP using the PDLP method"
- polyhedral techniques: Methods that translate combinatorial problems into geometric properties of polyhedra/polytopes for optimization. "showing how to solve tokenisation using polyhedral techniques."
- polytope: A bounded polyhedron; the feasible region of the LP defined by linear inequalities/equalities. "A polytope is defined as the finite closed intersection of halfspaces."
- power-law nature: A heavy-tailed distribution property common in language, affecting token frequency stability. "due to the power-law nature of natural language distributions"
- priced edges: Candidate token edges that count against the budget K and are selectable by colour. "a set of priced edges"
- primal dual gap: The difference between primal and dual objective values, used as a convergence/stopping criterion for LP solvers. "default stopping criteria of a primal dual gap of \num{10000}"
- Rényi entropy: A family of entropy measures parameterized by order α, used as an intrinsic tokenization metric. "including compression rate, vocabulary utilisation, and Rényi entropy."
- rounding: The process of converting fractional LP solutions into discrete token selections. "typically known as rounding."
- shortest path algorithm: An algorithm to compute the minimal-length path in a graph; used to compute optimal segmentations given selected tokens. "use a standard shortest path algorithm"
- shortest tokenisation problem: The problem of finding, under a token budget, the shortest segmentation path in the tokenisation graph. "The #1{shortest tokenisation problem} is to find"
- token-edge: An edge in the tokenisation graph connecting non-adjacent positions, representing a multi-byte token. "we connect each non-adjacent vertex within byte-strings with a #1{token-edge};"
- tokenisation graph: A graph structure whose paths correspond to segmentations of strings into tokens under a vocabulary. "We define a #1{tokenisation graph} to be the 3-tuple"
- tokenisation polytope: The feasible region of the LP relaxation capturing all fractional segmentations/vocabularies consistent with constraints. "We term this relaxation of $\polytope$ the #1{tokenisation polytope}."
- Type-Token Ratio: A lexical diversity metric defined as the number of distinct token types divided by the total tokens. "Type-Token \ Ratio (\textuparrow)"
- UnigramLM: A tokenization/modeling approach based on a unigram LLM; referenced for similar decoding runtime characteristics. "similar to UnigramLM's;"
- vocabulary utilisation: The fraction of the vocabulary that appears in the tokenized data; an intrinsic tokenization metric. "including compression rate, vocabulary utilisation, and Rényi entropy."
Collections
Sign up for free to add this paper to one or more collections.
