SupraTok: Advanced Superword Tokenization

• SupraTok is a tokenization framework that redefines subword segmentation by discovering contiguous multi-word expressions called superwords.
• It employs multi-phase curriculum learning with cross-boundary merges and entropy-driven data curation to optimize vocabulary utilization and tokenization efficiency.
• Empirical results show over 30% efficiency gains and a 31% reduction in sequence length, boosting model performance on benchmarks like HellaSWAG and MMLU.

SupraTok is a tokenization framework that reconfigures subword segmentation for LLMs by discovering and encoding “superword” tokens—multi-word expressions that transcend traditional word boundaries yet function as single semantic units. SupraTok generalizes Byte-Pair Encoding (BPE) through cross-boundary pattern learning, entropy-driven data curation, and phased curriculum learning, resulting in substantial gains in tokenization efficiency and downstream model performance. SupraTok’s innovations are complementary to existing architectural advancements and suggest a new frontier in reducing the informational bottleneck posed by conventional tokenizers (Tănase et al., 16 Aug 2025).

1. Superword Tokens and Generalization of BPE

SupraTok defines “superword” tokens as contiguous sequences of characters that may span any number of orthographic word boundaries, such as whitespace or punctuation, and are treated as atomic semantic units (e.g., “New_York”, “machine_learning”). This practice diverges from traditional BPE, which limits merge operations to within-word boundaries. In SupraTok, initial subwords are learned via canonical BPE; subsequent phases allow merges across whitespace, constructing vocabulary elements grounded in multi-word co-occurrences and contextual coherence.

SupraTok’s vocabulary expansion is guided by empirical frequency, segment diversity, and statistical association metrics, generalizing the BPE mechanism as follows:

• Phase 1: Standard within-word BPE.
• Phases 2 & 3: Controlled merges across word boundaries, enabled by statistical criteria such as pointwise mutual information (PMI) and branching entropy.

This approach allows the tokenizer to produce multi-word expressions with high joint probability and consistent contextual usage, resulting in a more compressive and semantically faithful encoding.

2. Cross-Boundary Pattern Learning

The discovery of multi-word units in SupraTok utilizes the Advanced Cross-Boundary Pre-Tokenizer (ACBP), which operates in two major stages:

• Phase 2 (100k–200k merges): Sequences meeting PMI $> 2.0$ and frequency $\geq 100$ are considered for merging, with redundancy constraints to maintain vocabulary diversity. PMI is calculated as:

$$\mathrm{PMI}(w_1,\dots,w_n) = \log_2\left(\frac{P(w_1,\dots,w_n)}{\prod_{i=1}^n P(w_i)}\right)$$

• Phase 3 (200k–256k merges): Candidates undergo left-branching entropy filtering:

$$H_{\mathrm{left}}(u) = -\sum_{c} P(c|u)\,\log_2 P(c|u)$$

where $u$ is a candidate multi-word unit and $c$ iterates over left contexts; low entropy indicates that $u$ is reliably an atomic expression. Additionally, a lightweight transformer LLM scores candidates for internal consistency and external unpredictability.

The segmentation process maximizes:

$$\mathcal{S}(V) = \sum_{t \in V} \log P(t) - \lambda |V|$$

with merges accepted if they increase $\mathcal{S}$ and satisfy PMI and entropy thresholds. This yields an inventory of superwords with high compressive and semantic fidelity.

3. Entropy-Driven Data Curation

SupraTok improves corpus quality by selectively sampling documents based on character bigram entropy:

$$H(\mathrm{doc}) = -\sum_{b \in \mathcal{B}} p(b)\log_2 p(b)$$

where $\mathcal{B}$ is the set of character bigrams, and $p(b)$ their relative frequency. Documents are retained at rates dependent on entropy:

$H(\mathrm{doc})$	Keep probability
$< 3.0$	$0.10$
$3.0 \leq H \leq 4.5$	$0.50$
$> 4.5$	$0.90$

This prunes $\approx 35\%$ of low-entropy (uninformative or boilerplate) material, boosting diversity and the informativeness of token patterns learned.

4. Multi-Phase Curriculum Learning

SupraTok merges token patterns over three explicit curriculum phases:

1. Phase 1 (0–100k merges): Standard BPE with merges restricted within word boundaries; cross-boundary bigram frequencies are collected for subsequent phases.
2. Phase 2 (100k–200k merges): Controlled cross-boundary merges based on PMI $>2.0$ and frequency $\geq 100$, halting when candidate count drops below 1,000 per merge iteration.
3. Phase 3 (200k–256k merges): Merges use branching entropy and transformer LM heuristics to capture complex formulaic expressions; convergence at 256k vocabulary size.

This staged approach stabilizes the acquisition of increasingly intricate units and guards against premature absorption of unreliable patterns. Vocabulary grows selectively, ensuring semantic robustness and compression.

5. Comparative Tokenization Efficiency

SupraTok’s cross-boundary merging yields significant efficiency improvements. On WikiText-103 (English; 256k vocabulary):

Tokenizer	Characters/token	Relative improvement over SupraTok
SupraTok	5.91	—
OpenAI o200k	4.51	31.0%
Google Gemma 3	4.53	30.4%
LLaMA 3.2	4.50	—
Qwen 3	4.34	—

Notably, 42% of SupraTok's vocabulary consists of cross-boundary superwords. Vocabulary utilization increases to 3.33% (vs. 1.52% for o200k), indicating greater active use of token types.

6. Integration with LLMs and Empirical Results

SupraTok was substituted for the standard BPE tokenizer in a GPT-2 style model (124M parameters, 12 layers, 768 hidden units, 12 attention heads) trained on 10B tokens from FineWeb-Edu. Training used:

• Learning rate: $6\times 10^{-4}$ (cosine decay)
• Batch size: 512
• Context length: 1,024 tokens
• AdamW optimizer ($\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$)

Fine-tuning for HellaSWAG and MMLU used:

• Learning rate: $5\times 10^{-5}$ (linear warmup over 500 steps)
• Batch size: 32
• Early stopping (patience: 3 epochs)
• Variance estimated over three seeds

SupraTok integration yielded:

Benchmark	SupraTok accuracy	BPE baseline	Relative gain
HellaSWAG	34.87%	32.14%	+8.4%
MMLU	27.75%	25.34%	+9.5%

SupraTok reduces sequence length by 31%, supporting more effective commonsense reasoning and broad knowledge tasks without architectural modifications.

7. Limitations, Scaling, and Architectural Complementarity

SupraTok’s results are validated at the 124M parameter scale; its behavior at 1B+ scale is yet to be determined. Entropy-driven data selection may unevenly affect languages with low average document entropy, risking loss of distinctive features in agglutinative or low-entropy scripts. SupraTok’s reliance on multi-word units may complicate autoregressive generation in scenarios where prediction of long tokens precedes context completion.

A progressive evaluation framework is in place for scale-up (1B, 7B, 13B parameters) with proportional dataset and compute increases. The reduction in sequence length is hypothesized to yield larger speedups and resource savings at scale, given quadratic attention complexities.

SupraTok’s compression and pattern learning are orthogonal to model architecture. It can be integrated with Mixture-of-Experts, sparse attention, retrieval-augmented models, and long-context transformers. SupraTok shifts some representational burden upstream, permitting models to concentrate on reasoning rather than reconstructing fragmented token patterns.

In summary, SupraTok employs statistically principled curriculum learning and entropy-driven selection to discover superword tokens, attaining over 30% tokenization efficiency gain and notable improvements on major benchmarks—all without modifying LLM architectures. These findings indicate that smarter tokenization represents a vital axis of advancement alongside scale and architectural innovation (Tănase et al., 16 Aug 2025).