SuperBPE: Cross-Boundary Tokenization

• SuperBPE is a tokenization algorithm that extends BPE by allowing merges across pretoken boundaries to form superwords, improving encoding efficiency.
• It utilizes a two-phase process—first learning subwords with strict pretokenization, then merging across boundaries—to reduce token counts and computational cost.
• Empirical results show that SuperBPE enhances language model accuracy, lowers inference FLOPs, and ensures more uniform tokenization across diverse languages.

SuperBPE defines a class of tokenization algorithms extending byte-pair encoding (BPE) by enabling merges that cross pretokenization (typically, whitespace) boundaries. This facilitates the learning of "superwords"—multi-pretoken (often multi-word) units—improving encoding efficiency, capturing linguistic phrases, and mitigating inequities introduced by word-boundary-based pretokenization. Empirical results demonstrate gains in average LLM accuracy, reduced computational cost, and more uniform crosslingual tokenization compared to standard BPE. The method has been widely adopted in both monolingual and multilingual LLM pretraining pipelines (Schmidt et al., 6 Apr 2026, Liu et al., 17 Mar 2025, Arnett et al., 24 Oct 2025).

1. Theoretical Foundations and Notation

SuperBPE generalizes BPE tokenization by operating on "pretokens"—the smallest atomic units obtained after pretokenization. In space-delimited scripts (e.g., English), pretokens correspond roughly to words or punctuation. In non-space-delimited scripts (e.g., Chinese, Thai), each Unicode character may be a pretoken to preserve text integrity (Schmidt et al., 6 Apr 2026).

Key definitions:

• Pretoken $w$: Atomic unit post-pretokenization.
• Pretoken frequency: $f(w) = \sum_{d\in D} f_d(w)$, where $D$ indexes the corpus and $f_d(w)$ is the count of $w$ in document $d$.
• Merge: Classical BPE operation, $f(r_1, r_2)=\sum_{d\in D} f_d(r_1, r_2)$.
• Supermerge: Composition of two or more consecutive pretokens, forming a "superword" token; frequency $$f(w_i,\dots,w_{i+k-1}) = \sum_{d\in D} f_d(w_i,\ldots,w_{i+k-1})$$.

SuperBPE evaluates both regular merges (within boundaries) and supermerges (across them) based on aggregate frequencies, iteratively adding the highest-scoring candidate to the vocabulary, whether it is a standard merge or a supermerge (Schmidt et al., 6 Apr 2026, Arnett et al., 24 Oct 2025).

2. Algorithmic Structure

SuperBPE is operationalized as a two-phase or "curricular" extension to standard BPE:

Phase 1 (Subword learning): BPE is applied to data with strict pretokenization, ensuring that merges do not cross original boundaries. This phase accumulates robust subword units (Liu et al., 17 Mar 2025).

Phase 2 (Superword learning): The pretokenization constraint is lifted. Merges are allowed to span boundaries, enabling the construction of superwords from frequent adjacent pretokens or subwords—including multiword expressions—improving both byte-per-token efficiency and downstream LM effectiveness (Liu et al., 17 Mar 2025, Arnett et al., 24 Oct 2025).

A schematic pseudocode for SuperBPE:

$f_d(w)$0

The transition point $t$ is a tunable hyperparameter: common settings are $t =$ 80k, 160k, or 180k for a final $f(w) = \sum_{d\in D} f_d(w)$0k (Liu et al., 17 Mar 2025).

3. Implementation Strategies and Optimizations

Naive implementations necessitate tracking of all pretoken sequences, resulting in prohibitive memory and compute costs. Recent research demonstrates that aggregation of supermerge candidates (i.e., recording only unique sequences and their counts) obviates the need for full-corpus memory residency (Schmidt et al., 6 Apr 2026).

Optimizations include:

• Pretoken and supermerge aggregation: Collect unique candidates and their frequencies.
• Max-heap for merge selection: Maintain a priority queue for regular and supermerges via their scores.
• Efficient updates: Implement all merges by updating only affected unique entries, not the entire corpus.
• Fast implementations: Reference Python and Rust codebases exist, with Rust achieving $f(w) = \sum_{d\in D} f_d(w)$1600$f(w) = \sum_{d\in D} f_d(w)$2 speedup versus original approaches for training on 1GB data—593s for SuperBPE versus 4.7 CPU days (Schmidt et al., 6 Apr 2026).

Greedy n-gram splitting (via an Apriori-style bound) can further accelerate aggregation at the cost of slight count approximation. Training remains single-threaded, but parallelism is possible (Schmidt et al., 6 Apr 2026).

4. Empirical Performance and Comparative Metrics

SuperBPE demonstrates marked improvements in encoding and model metrics. The relevant efficiency metrics are:

• Bytes per token (BPT):

$f(w) = \sum_{d\in D} f_d(w)$3

SuperBPE achieves $f(w) = \sum_{d\in D} f_d(w)$4 BPT at 200k vocabulary, compared to BPE's $f(w) = \sum_{d\in D} f_d(w)$5 (Liu et al., 17 Mar 2025).

• Token-count reduction: For fixed text, SuperBPE yields up to 33% fewer tokens than BPE (Liu et al., 17 Mar 2025).
• Downstream accuracy: For 8B Transformer LMs pretrained on identical data and compute, SuperBPE gives a +4.0 percentage point (pp) average accuracy gain over BPE on 30 benchmarks, with a +8.2 pp gain on MMLU. Inference FLOPs decrease by 27% due to shorter token sequences (Liu et al., 17 Mar 2025).
• Token uniformity: SuperBPE reduces per-token bits-per-byte (BPB) variance, leading to more uniform difficulty in next-token prediction (Liu et al., 17 Mar 2025).
• Crosslingual compression: SuperBPE systematically lowers average corpus token counts (CTC) by 5–10% and halves variance across 97 languages, ameliorating token premium inequities—especially in languages with high whitespace density (Arnett et al., 24 Oct 2025).

A summary table from (Liu et al., 17 Mar 2025):

Model	Avg. $f(w) = \sum_{d\in D} f_d(w)$6@30 tasks	MMLU	Inference FLOPs	Inference savings
8B, BPE	39.8%	36.5%	$f(w) = \sum_{d\in D} f_d(w)$7	–
8B, SuperBPE (t=180k)	43.8% (+4.0)	44.7% (+8.2)	$f(w) = \sum_{d\in D} f_d(w)$8	–27%
11B, SuperBPE	42.9%	41.9%	$f(w) = \sum_{d\in D} f_d(w)$9	0%

5. Crosslingual Token Premiums and Equity

SuperBPE addresses cross-linguistic token inequities formalized as "token premiums": for language $D$0 relative to reference $D$1,

$D$2

where $D$3 is the corpus token count for language $D$4 at vocabulary size $D$5 (Arnett et al., 24 Oct 2025).

With standard BPE, languages with many short whitespace-delimited words suffer disproportionately high token premiums. SuperBPE, by allowing merges over boundaries, reduces both mean token counts and variance—demonstrated by CTC measures over parallel corpora such as FLORES-200. For $D$6, monolingual BPE had mean CTC ≈ 56,000 (std 2,300), SuperBPE ≈ 52,000 (std 1,700) (Arnett et al., 24 Oct 2025).

Moreover, the dependence of token count variance on language whitespace density (up to $D$7) vanishes under SuperBPE, confirming the neutralization of whitespace-driven artifacts (Arnett et al., 24 Oct 2025).

Determining language-specific "optimal" vocabulary sizes (to minimize token premium) is possible by fitting power-law CTC curves; SuperBPE achieves better uniformity and lower minimum CTC than BPE (Arnett et al., 24 Oct 2025).

6. Practical Implementation and Applications

SuperBPE is architecturally minimal, requiring only a two-phase pretokenization "curriculum" and merge process on top of canonical BPE. It is compatible with standard LM training and decoding, without necessitating design changes elsewhere.

Key recommendations and uses:

• Employ robust regular expressions for pretokenization; for non-spaced scripts, split into Unicode characters to preserve semantics (Schmidt et al., 6 Apr 2026).
• Apply greedy n-gram splitting for performance as necessary.
• Reference Python and Rust implementations are available, with the latter suitable for high-throughput or large-scale settings (Schmidt et al., 6 Apr 2026).
• Applications include rapid retraining for evolving domains, on-the-fly vocabulary adaptation, and improved tokenization in non-Latin or mixed-script scenarios (Schmidt et al., 6 Apr 2026).

7. Limitations and Trade-Offs

SuperBPE requires two-phase training (slightly more compute for candidate aggregation), and effectiveness is maximized for large vocabularies; most empirical studies use $D$8 (Arnett et al., 24 Oct 2025). There is a hypothetical risk of "over-compression" where very long tokens could, in principle, hinder model learning dynamics, but such effects have not been observed in downstream tasks to date (Arnett et al., 24 Oct 2025). The method does require hyperparameter selection for the transition point $D$9, though automatically-tuned two-phase variants (e.g., BoundlessBPE) mitigate this necessity (Schmidt et al., 6 Apr 2026).

---

SuperBPE embodies a minimal yet impactful augmentation over classical BPE, yielding substantial benefits in encoding efficiency, downstream model performance, and linguistic equity across diverse scripts and languages. Its approach is modular, scalable, and compatible with existing language modeling infrastructure (Schmidt et al., 6 Apr 2026, Liu et al., 17 Mar 2025, Arnett et al., 24 Oct 2025).