Byte-Level Tokenization

• Byte-level tokenization is the process of converting normalized UTF-8 text into a fixed sequence of 256 raw byte IDs, ensuring open-vocabulary coverage.
• It serves as the basis for token-free models like ByT5, providing robust multilingual and noise-resistant processing through techniques such as BBPE.
• Techniques such as greedy merging, dynamic byte compression, and careful architectural integration help mitigate inefficiencies from longer input sequences.

Byte-level tokenization refers to the direct mapping of text into sequences of raw bytes—typically encoded in UTF-8 or, less commonly, UTF-16 or UTF-32—thus entirely bypassing morphological, word, subword, or even Unicode code-point segmentation in the preprocessing pipeline. This paradigm underlies a growing class of LLMs, termed “token-free” or pure Byte LLMs (BLMs), and also serves as the foundation for byte-level variants of classical subword tokenization methods, notably Byte-level Byte-Pair Encoding (BBPE). The approach guarantees open-vocabulary coverage, eliminates OOV errors, simplifies cross-lingual modeling, and exhibits unique robustness and pathologies, all of which have been rigorously studied and quantified in recent literature.

1. Definitional Frameworks and Core Algorithms

At its core, byte-level tokenization is the identity map from the normalized, UTF-8–encoded text string $S$ to a sequence of integers $b = [b_1,\ldots,b_n]$ where each $b_i \in \{0, \ldots, 255\}$ represents one raw byte. Formally,

$$T = (t_1, \ldots, t_n )\,,\quad \text{where}\ t_i = \operatorname{UTF8Encode}(S)_i\,,\ \forall i.$$

UTF8Tokenizer (Moryossef et al., 19 Oct 2025) embodies this minimalism, mapping text to byte-IDs without out-of-range IDs or auxiliary tokens, resulting in exactly 256 categories. ByT5 (Xue et al., 2021) adopts a near-identical approach but appends three specially reserved IDs for <pad>, <eos>, and an unused <unk>, and reuses high byte-IDs for sentinels in masked span corruption tasks.

When used as the atomic unit of Byte-Pair Encoding (BPE), bytes serve as the initial alphabet $\Sigma_0=\{0,1,\ldots,255\}$. Iterative merges are performed to build up multi-byte tokens, producing a fixed vocabulary $V$ and associated merge ordering $D$ (Berglund et al., 2023, Jang et al., 2024). The essential steps are as follows:

1. Normalization & Encoding: Text is normalized (e.g., NFC or NFKC) and then encoded as a UTF-8 byte sequence.
2. Pre-tokenization (optional): For BBPE, a lightweight presegmentation splits text into chunks to facilitate parallel BPE merges. Peek2 (Zai, 9 Jan 2026) replaces fragile regex-based pretokenizers with a static-category, $O(n)$ peeking table.
3. Greedy Merging: At each step $t$, the most frequent adjacent byte-pair $(a,b)$ in the training corpus is merged into a new token $c$, and every instance of $(a,b)$ is replaced with $c$.
4. Tokenization: At inference, longest-match ("maximal-munch") greedy lookup is applied against $V$ to segment any byte sequence (Berglund et al., 2023, Jang et al., 2024).

Alternative byte-level schemes include pure one-hot byte input for training (e.g., CANINE (Velayuthan et al., 2024)), bit-level BPE for subbyte compression (Moon et al., 9 Jun 2025), and UTF-16–based BBPE16 for uniform code-unit granularity (Kim et al., 2 Feb 2026).

2. Embedding and Architectural Integration

Models ingesting byte-tokenized input typically employ a single embedding matrix $E \in \mathbb{R}^{256 \times d}$, where $d$ is the model/dimension, and optionally employ a parametrizable projection to expose the per-byte bit structure (“bit-bias embeddings” (Moryossef et al., 19 Oct 2025)). In ByT5-type architectures (Xue et al., 2021), byte-IDs are embedded via $E[b_i]$, learned positional encodings are applied, and the resulting sequence is processed by a Transformer backbone without further architectural modifications, aside from depth rebalancing (3:1 encoder:decoder). Control codes, e.g., ASCII C0, may be reserved in the embedding table for padding, BOS, EOS, and structure demarcation (Moryossef et al., 19 Oct 2025).

Innovative variants dynamically compress byte sequences (e.g., MrT5’s “soft” token deletion/retention mechanism (Kallini et al., 2024)), use hierarchical byte→chunk compression-decompression pipelines (Bao et al., 1 Feb 2026), or organize byte streams into dynamically sized entropy-driven patches (BLT (Pagnoni et al., 2024)) or word-aligned “patches” using spacelike boundary heuristics (SpaceByte (Slagle, 2024)).

3. Computational, Storage, and Efficiency Characteristics

The computational footprint of byte-level models scales directly with the length $N_b$ of the byte sequence input, which is typically a factor $c\approx 4.1$ longer than SentencePiece tokenized input for the same text (English, per (Xue et al., 2021)). Per-sequence FLOPs for a Transformer are:

$$\operatorname{FLOPs}_{\text{byte}} \approx 1.2\times \operatorname{FLOPs}_{\text{subword}}$$

when model dimension, depth, and parameter budget are matched. In practice, this translates to modestly higher resource consumption for training and inference.

For storage and host-device transfer, pure byte-IDs allow representation as $uint8$ (Moryossef et al., 19 Oct 2025), reducing traffic and memory by $8\times$ compared to standard $int64$-tensor token representations. Tokenization is also typically an $O(n)$ operation and is $14\times$ faster with zero-copy NumPy/PyTorch implementations than HuggingFace’s ByT5Tokenizer.

For subword BBPE models, techniques such as Peek2 (Zai, 9 Jan 2026) realize linear-time, regex-free pretok pipelines, offering up to $1.11\times$ throughput improvement over legacy Regex-based implementations in GPT-3, LLaMa-3, and Qwen pipelines.

4. Empirical Performance, Robustness, and Limitations

Byte-level models have demonstrated advantageous empirical properties regarding open-vocabulary coverage, multilingual handling, robustness, and efficiency:

• Open-Vocabulary and Multilinguality: Universality of the byte alphabet allows language-agnostic processing (ByT5 (Xue et al., 2021), UTF8Tokenizer (Moryossef et al., 19 Oct 2025)). In multilingual translation, pure byte-level tokenizers eliminate OOV failure modes, facilitate cross-lingual token sharing, and mitigate over-segmentation, especially for low-resource or morphologically complex languages (Sreedhar et al., 2022, Kim et al., 2 Feb 2026).
• Noise Robustness: ByT5, BLT, and similar models degrade 2–8$\times$ less under synthetic noise (drop, repetition, random-case, uppercase), and outperform token-based models on character-level orthographic tasks (transliteration, G2P, inflection) by 2–5$\times$ (Xue et al., 2021, Pagnoni et al., 2024).
• Compression and Sequence Length: A principal drawback is sequence length inflation: a CJK character (3 UTF-8 bytes) yields 3 byte tokens vs. 1–2 subword tokens, leading to computational inefficiency. Mitigations include BBPE16 (Kim et al., 2 Feb 2026) (reducing CJK sequence length by up to $10.4\%$), bit-level BPE (Moon et al., 9 Jun 2025) (3–6% reduction of CJK tokens), dynamic token merging (MrT5, up to $75\%$ reduction (Kallini et al., 2024)), entropy-driven patch segmentation (BLT, $n/L$ global steps (Pagnoni et al., 2024)), and word-aligned capacity scaling (SpaceByte (Slagle, 2024)).
• Vulnerabilities: Byte-level BPE necessarily admits the existence of “incomplete tokens” (tokens that are not valid UTF-8 subsequences) (Jang et al., 2024, Firestone et al., 5 Nov 2025). “Improbable bigrams” constructed from these can induce hallucination rates up to $79\%$ with LLMs (vs. $0-9\%$ for baseline pairs) (Jang et al., 2024). Altering tokenizations to avoid incomplete tokens cuts hallucination risk by $93\%$. Ill-formed UTF-8 output is algebraically unavoidable unless the tokenizer vocabulary is restricted to code points (Firestone et al., 5 Nov 2025).
• Empirical Task Outcomes: Byte-level LLMs match or exceed token-based models in open-ended text generation, multilingual transfer, and robust spelling tasks, but may lag for large models on English classification with massive subword vocabularies (Xue et al., 2021). In machine translation, explicit local byte aggregation (LOBEF: n-gram convolutional or word-restricted self-attention) yields consistent BLEU gains ($+0.4$–$1.4$) over vanilla byte and subword baselines (Sreedhar et al., 2022). In large-scale scaling, BLT matches or surpasses tokenization-based Llama models, with $5$–$20\%$ lower bits-per-byte (BPB) and up to $50\%$ FLOP savings for long predictable spans (Pagnoni et al., 2024).

5. Model Conversion, Distillation, and Inference-Time Byteization

Byte-level text modeling can be achieved from a token-trained LM via explicit distillation and alignment (Bao et al., 1 Feb 2026). The two-stage curriculum aligns byte-level student embeddings, boundaries, and logits to the teacher’s subword outputs, then fine-tunes for byte-level next-token prediction. On-standard benchmarks, only $2$–$3$ point accuracy drop is incurred with $125$B byte-level training steps, representing an order of magnitude reduction in required data relative to prior work.

Inference-time “byte-sampler” and next-byte probability extraction algorithms (Firestone et al., 5 Nov 2025, Hayase et al., 17 Jun 2025, Phan et al., 2024) enable sampling and ensembling of LMs in unified byte space regardless of tokenizer heterogeneity or prompt boundary alignment. These approaches are statistically exact, solve the Prompt Boundary Problem, and allow plug-and-play composition of LM outputs for downstream tasks, including robust fill-in-the-middle (FIM) and instruction proxy-tuning.

6. Cross-Lingual and Fairness Considerations

Byte-level tokenization exhibits non-uniform compression rates across scripts: abugida and complex scripts (e.g., Tamil, Sinhala, Hindi) suffer from extreme fragmentation under UTF-8 byte-level schemes (Tokenization Parity >2.5 vs. English), imposing a significant computational penalty (Velayuthan et al., 2024). Grapheme-based pre-tokenization (GPE) that aligns tokens to Unicode grapheme clusters yields higher compression ratios and tokenization parity, fostering cross-lingual fairness and more egalitarian tokenizers.

7. Open Challenges and Future Directions

Recent work highlights unresolved issues:

• Compression-Quality Tradeoffs: Sequence length inflation and compute inefficiency remain the weakest link for pure byte-level models. Dynamic deletion/merging, patching, or sophisticated per-language compression strategies show promise but introduce new sources of complexity (Kallini et al., 2024, Pagnoni et al., 2024).
• Ill-formed Sequences: Ensuring model outputs can be converted to well-formed UTF-8 remains nontrivial, with implications for streaming inference engines and downstream tools.
• Integration With Task-Specific Pipelines: Systematic benchmarking and best practice guidelines are needed for deploying byte-level models in diverse domains (NLP, ASR, MT) and for interoperability with legacy token-based models.
• Egalitarian Modeling: Adaptive or grapheme-level approaches compatible with the byte paradigm, and equitable allocation of compression across high- and low-resource languages, represent promising directions.

---

In total, byte-level tokenization constitutes a strict, mathematically minimal, and highly extensible approach to text representation for neural models. Its adoption has catalyzed progress in language-agnostic, robust, and open-vocabulary LLMs but demands continued innovation in efficiency, segmentation granularity, and model reliability (Xue et al., 2021, Moryossef et al., 19 Oct 2025, Jang et al., 2024, Bao et al., 1 Feb 2026, Sreedhar et al., 2022, Slagle, 2024, Moon et al., 9 Jun 2025, Kim et al., 2 Feb 2026).