Tokenization with Split Trees (ToaST)

• The paper introduces ToaST, a novel subword tokenization method that builds full binary split trees to minimize token counts via an integer programming framework.
• ToaST separates tree construction, recursive inference, and vocabulary optimization, enabling deterministic and vocabulary-agnostic tokenization without merge-rule dependencies.
• Empirical results demonstrate that ToaST reduces token counts by over 11% and improves compression and language model performance compared to BPE, WordPiece, and UnigramLM.

Tokenization with Split Trees (ToaST) is a subword tokenization method that combines a vocabulary-agnostic binary decomposition of each pretoken with a deterministic recursive inference rule and a global vocabulary-selection objective. Introduced in "Tokenization with Split Trees" (Schmidt et al., 21 May 2026), it builds a full binary split tree for every pretoken from byte $n$-gram counts, then tokenizes by recursively descending each tree and emitting the first in-vocabulary node reached on each path. Vocabulary learning is posed as an Integer Program that minimizes total corpus token count under this inference procedure. In the reported English-language experiments, ToaST reduces token counts by more than $11\%$ relative to BPE, WordPiece, and UnigramLM at vocabulary sizes $40{,}960$ and above, substantially reduces the use of common single-byte tokens, and yields the highest CORE score in $1.5$B-parameter language-model experiments (Schmidt et al., 21 May 2026).

1. Methodological position and core abstraction

ToaST was proposed as an alternative to two established design patterns in subword tokenization. BPE and WordPiece are bottom-up and merge-based, while UnigramLM is top-down, begins with a large vocabulary, and prunes by likelihood-based scoring. ToaST instead separates three components: tree construction, inference, and vocabulary optimization. The split trees are constructed first, using only byte $n$-gram frequency counts and no knowledge of the eventual vocabulary; inference is then defined for any vocabulary $V \subseteq T$ that contains all valid single-byte UTF-8 tokens; and vocabulary learning is finally cast as an explicit optimization problem over that fixed candidate set (Schmidt et al., 21 May 2026).

The central object is the per-pretoken full binary split tree. A pretoken is the byte substring produced by regex-based pretokenization, and each pretoken type is weighted by its aggregate corpus count. Every node in a tree corresponds to a byte substring, leaves correspond to bytes, and internal nodes correspond to recursively chosen binary splits. The global candidate-token set is

$$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$

where $B$ is the set of $243$ single-byte tokens that must always remain available (Schmidt et al., 21 May 2026).

This decomposition yields several structural consequences. Tree construction is vocabulary-independent, so the candidate token set does not depend on which tokens are later selected. Any vocabulary containing the $243$ valid single-byte UTF-8 tokens is valid, so no merge-rule list is required. Removing a token from the vocabulary does not trigger global resegmentation; it only forces recursive descent below that node. The paper presents this as a key distinction from merge-based inference and shortest-path-like segmentation schemes (Schmidt et al., 21 May 2026).

2. Split-tree construction from byte $11\%$0-gram statistics

The reported implementation begins by pretokenizing a $11\%$1GB English corpus from CulturaX with a length-limited variant of the GPT-4o regex. Pretokens are aggregated with counts, and the top $11\%$2 pretokens cover $11\%$3 of all pretoken occurrences. Each selected pretoken is then converted into a full binary split tree (Schmidt et al., 21 May 2026).

Tree construction depends on corpus-wide byte $11\%$4-gram counts computed within pretoken boundaries. Only $11\%$5-grams with count at least $11\%$6 are stored. Under this threshold, the reported system contains $11\%$7 unique $11\%$8-grams, stored as a dict[bytes, int] requiring about $11\%$9MB. These counts are the only statistics used to define the tree structure (Schmidt et al., 21 May 2026).

For a pretoken $40{,}960$0, every split point $40{,}960$1 with $40{,}960$2 defines a left substring $40{,}960$3 and right substring $40{,}960$4. Let their stored counts be $40{,}960$5 and $40{,}960$6; if absent from the dictionary, a default low value such as $40{,}960$7 is used. The split score is

$40{,}960$8

The chosen split is the one maximizing $40{,}960$9, with ties broken by the leftmost split. If no split has both sides present in the count dictionary, the implementation falls back to splitting at the longest prefix that is in the dictionary. Recursing until single bytes produces a full binary tree (Schmidt et al., 21 May 2026).

The paper’s worked example is the pretoken “␣Kentucky”. Among candidate splits, the best root split is “␣Kent”$1.5$0“ucky”, whose $1.5$1 value is $1.5$2. Recursive application of the same rule generates internal nodes such as “␣Kent”, “Kent”, and “ent”, with byte leaves at the bottom of the tree (Schmidt et al., 21 May 2026).

Although the main experiments operate at the byte level, the construction is explicitly modular. The appendix proposes a split-preference hierarchy of superwords, morphemes, characters, and bytes, with the same $1.5$3 scoring applied inside the highest available level. In the reported experiments, only the multi-byte-character constraint is enforced: splits never occur inside a multi-byte Unicode character (Schmidt et al., 21 May 2026).

3. Recursive inference and segmentation semantics

Given a fixed split tree and a vocabulary $1.5$4, ToaST tokenization is defined by recursive descent. If the current substring is in the vocabulary, or if its length is $1.5$5, tokenization emits that substring and stops descending on that branch. Otherwise the algorithm follows the precomputed split and recurses on the left and right children. In code form, the core logic is $11\%$42 where best_split(s) is fixed by the training-time tree construction (Schmidt et al., 21 May 2026).

Operationally, the inference rule can be described as follows: for each root-to-leaf path, emit the highest node on that path whose string is in the vocabulary. Because each byte position lies on a unique path and each leaf byte is guaranteed to be in the vocabulary, every path emits exactly one token. The resulting tokenization is therefore unique and deterministic (Schmidt et al., 21 May 2026).

This semantics induces a precise coverage property. Let $1.5$6 be the leaves of split tree $1.5$7, and let $1.5$8 be the ancestors of leaf $1.5$9. Then exactly one node on each root-to-leaf path must be selected: $n$0 This equality is both the combinatorial meaning of inference and one of the optimization constraints used during vocabulary learning (Schmidt et al., 21 May 2026).

Several consequences follow immediately. Because trees are independent of the vocabulary, ToaST does not require merge rules. Because tokenization always stops at the first in-vocabulary node on a path, removing a vocabulary item forces refinement only below that node. The paper characterizes this as the absence of global cascading effects. A common misconception is therefore to treat ToaST as a BPE variant with a different merge criterion; in fact, its inference model is tree-based and path-local rather than merge-sequence-based (Schmidt et al., 21 May 2026).

The empirical analysis also classifies output tokens by role: Root, Unavoidable Leaf, Leaf, and Non-Leaf Subword. This typology is later used to compare ToaST with BPE, WordPiece, and UnigramLM, especially in the treatment of single-byte leaf tokens (Schmidt et al., 21 May 2026).

4. Vocabulary selection as an Integer Program

Vocabulary learning in ToaST is formulated as an Integer Program over global vocabulary variables and per-tree usage variables. Let $n$1 indicate whether candidate token $n$2 is selected, and $n$3 indicate whether node $n$4 in tree $n$5 is used in tokenization. If $n$6 is the aggregate count of pretoken $n$7, the objective is

$n$8

which is exactly the total token count over the corpus under split-tree inference (Schmidt et al., 21 May 2026).

The optimization is subject to four main constraints. First, the vocabulary has prescribed size: $n$9 Second, all valid UTF-8 single bytes are mandatory: $V \subseteq T$0 Third, each root-to-leaf path must be covered exactly once: $V \subseteq T$1 Fourth, a node can be used only if its string is in the vocabulary: $V \subseteq T$2 Together with binary constraints on $V \subseteq T$3 and $V \subseteq T$4, these equations define the training problem (Schmidt et al., 21 May 2026).

The LP relaxation replaces integrality by

$V \subseteq T$5

Empirically, this relaxation is exceptionally tight. Across $V \subseteq T$6 instances obtained from $V \subseteq T$7k, $V \subseteq T$8k, $V \subseteq T$9k, and $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$0k split trees and $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$1 vocabulary sizes from $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$2 to $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$3, the worst case has only $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$4 fractional $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$5 variables and $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$6 fractional $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$7 variables, i.e. less than $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$8 and less than $$T=\left(\bigcup_{j\in J,\;k\in K_j} t_{jk}\right)\cup B,$$9 respectively. The maximum absolute gap between the integer-rounded solution $B$0 and the LP optimum $B$1 is $B$2 tokens out of $B$3 billion tokens,

$B$4

and in $B$5 of $B$6 cases the LP solution is entirely integral (Schmidt et al., 21 May 2026).

The paper presents this near-integrality as an empirical finding rather than a proved theorem. It conjectures that the tree structure is responsible, because higher nodes dominate lower nodes in token-count reduction along a path. This suggests that the LP’s tightness is structural, but a general integrality characterization is not stated (Schmidt et al., 21 May 2026).

A simple rounding heuristic is used for the remaining fractional cases. With a tolerance $B$7, variables with $B$8 are fixed to $B$9, those with $243$0 are fixed to $243$1, and the remaining budget is allocated to the fractional tokens with largest

$243$2

The resulting binary vocabulary is then re-evaluated by re-running split-tree inference (Schmidt et al., 21 May 2026).

Model size is linear in total byte length. Each full binary tree satisfies $243$3, so the LP has $243$4 global $243$5-variables, $243$6 $243$7-variables, and $243$8 constraints. Empirically, total build-plus-solve time scales quadratically in the number of split trees over the range $243$9k–$243$0k. With HiGHS dual simplex on an AWS r7i.48xlarge, solving the largest vocabulary size $243$1 takes $243$2–$243$3 hours depending on the number of trees; subsequent warm-started re-solves for smaller $243$4 take less than one hour each (Schmidt et al., 21 May 2026).

5. Empirical behavior: compression, token usage, and language modeling

Compression is evaluated on a $243$5GB English validation set from CulturaX using bytes per token, equivalently corpus byte length divided by total token count. Under this metric, ToaST improves compression by $243$6 over the best baseline at vocabulary size $243$7, by $243$8 at $243$9, and by more than $11\%$00 over BPE, WordPiece, and UnigramLM at vocabulary sizes $11\%$01 and larger. The paper also reports a theoretical upper bound induced by the fixed pretokenization: if each pretoken were represented by a single token, validation compression would be $11\%$02 bytes per token. ToaST approaches this bound more closely than the baselines (Schmidt et al., 21 May 2026).

The token-category analysis highlights the reduction of single-byte usage. At vocabulary size $11\%$03, ToaST uses $11\%$04–$11\%$05 fewer Leaf tokens than all baselines and correspondingly uses substantially more Root tokens. The reported interpretation is that ToaST does not need to preserve as many intermediate subwords as “stepping stones” toward frequent words; since the objective directly minimizes token count under the tree inference rule, allocating vocabulary mass to whole pretokens is often preferable (Schmidt et al., 21 May 2026).

The paper further evaluates Rényi efficiency with $11\%$06, following prior work cited there. Because ToaST dramatically reduces high-frequency leaf tokens, it achieves substantially higher Rényi efficiency than BPE, WordPiece, and UnigramLM; Shannon efficiency is also higher. In a Zipf analysis at vocabulary size $11\%$07k, ToaST has fewer very high-frequency tokens and fewer extremely rare tokens. The minimum token count on validation is $11\%$08 for ToaST, compared with $11\%$09 for UnigramLM and $11\%$10 for BPE and WordPiece. Only $11\%$11 ToaST tokens are unused over the $11\%$12GB validation set, versus $11\%$13–$11\%$14 unused tokens for the baselines; the unused ToaST tokens are rare single bytes included for completeness (Schmidt et al., 21 May 2026).

Inference speed is reported at about $11\%$15k tokens/s in Python on a MacBook Pro. For comparison, BPE in the Hugging Face Rust implementation achieves about $11\%$16M tokens/s, so ToaST is about $11\%$17 slower in that comparison. The paper notes that performance benefits from the frequent case in which the pretoken root itself is in the vocabulary, eliminating recursion (Schmidt et al., 21 May 2026).

Downstream evaluation uses four $11\%$18B-parameter nanochat depth-24 LLMs that differ only in tokenizer and are pretrained on ClimbMix-400B with a matched budget of $11\%$19B tokens. Because ToaST uses about $11\%$20 fewer tokens to represent text, its model sees roughly $11\%$21 more raw text under the same token budget. On CORE, the reported base-model means are $11\%$22 for ToaST, $11\%$23 for BPE, $11\%$24 for Unigram, and $11\%$25 for WordPiece. Relative to each baseline, ToaST is $11\%$26 versus BPE, $11\%$27 versus Unigram, and $11\%$28 versus WordPiece; Welch’s $11\%$29-tests over four seeds give $11\%$30 for ToaST versus BPE and $11\%$31 for ToaST versus Unigram, while the difference versus WordPiece is not significant. ToaST has the highest mean on $11\%$32 of the $11\%$33 CORE base tasks (Schmidt et al., 21 May 2026).

The supervised fine-tuning results are more mixed. The paper reports that ToaST’s SFT CORE is slightly below Unigram but similar to BPE and WordPiece, and attributes part of this outcome to the experimental protocol: SFT is not token-matched, so a more compressive tokenizer receives fewer gradient steps on SFT data. This frames one of the paper’s main trade-offs: under a fixed token budget, ToaST benefits in pretraining exposure but may be disadvantaged in later phases that are not normalized by tokens (Schmidt et al., 21 May 2026).

6. Relation to split-tree theory, conceptual background, and limitations

The term split tree predates ToaST and has an established meaning in probabilistic combinatorics. In that literature, split trees are random trees of logarithmic height introduced by Devroye in $11\%$34, generated by a trickle-down process of balls through buckets on an infinite $11\%$35-ary skeleton. They include binary search trees, $11\%$36-ary search trees, quad trees, and related structures. In "Embedding small digraphs and permutations in binary trees and split trees" (Albert et al., 2019), split trees are used to study permutation occurrences, generalized path-length parameters, and embeddings of fixed DAGs, while "Fringe subtrees of split trees and fractional split trees" (Holmgren et al., 23 Mar 2026) studies additive functionals such as numbers of nodes, leaves, and fringe trees. ToaST uses the same phrase for a different object: a deterministic full binary decomposition of a pretoken obtained from byte $11\%$37-gram counts (Schmidt et al., 21 May 2026).

This suggests a structural rather than model-identical relationship. The older split-tree literature supplies a vocabulary for local tree patterns, fringe subtrees, additive counts, and logarithmic-depth recursions (Albert et al., 2019, Holmgren et al., 23 Mar 2026). A plausible implication is that these concepts are useful for formalizing statistics of tree-based tokenizers, but the ToaST paper’s optimization framework, recursive “first in-vocabulary” inference rule, and LP near-integrality are specific to the tokenizer construction rather than inherited from the random-tree results (Schmidt et al., 21 May 2026).

The current ToaST formulation has several stated limitations. All reported experiments are English-only. Training uses only the top $11\%$38 most frequent pretokens, such as $11\%$39k pretokens in the main experiments, and the coverage behavior of this truncation may differ in languages with more uniform distributions or larger character sets. Training requires solving large LPs or near-MIPs with millions of variables and constraints, which is substantially heavier than BPE training. The paper also notes that although the LP relaxation is exceptionally tight in practice, this is an empirical observation rather than a proved general property (Schmidt et al., 21 May 2026).

Future extensions are framed mainly through tree construction and weighting. The appendix proposes richer split hierarchies involving superwords, morphemes, characters, and bytes. Multilingual tokenization can be incorporated by rescaling the linear objective weights $11\%$40 so that different languages contribute differently to vocabulary selection. More generally, any vocabulary-independent binary splitting rule could replace the current $11\%$41 heuristic without changing the downstream IP/LP formulation (Schmidt et al., 21 May 2026).

Within the tokenization literature, ToaST therefore occupies a distinct design point: fixed vocabulary-agnostic trees, deterministic path-local inference, and explicit global minimization of token count over that structure. Its empirical claims are strongest on compression and token-usage balance, while its theoretical distinctiveness lies in recasting vocabulary learning as a nearly integral linear optimization problem over a tree-constrained segmentation space (Schmidt et al., 21 May 2026).