Byte Language Models: A Tokenization-Free Approach

• Byte Language Models are neural models that process raw UTF-8 bytes without tokenization, offering script-agnostic and open-vocabulary capabilities.
• They employ hierarchical and patch-based architectures with dynamic compression and two-stage distillation to enhance efficiency and performance.
• BLMs overcome issues like out-of-vocabulary tokens and subword brittleness, reducing data and computational needs while excelling in text, code, and multimodal applications.

Byte LLMs (BLMs) are neural LLMs that ingest and generate text at the byte level, entirely bypassing subword or character tokenization. By treating raw UTF-8 bytes as atomic modeling units, BLMs resolve long-standing issues in conventional LLMs such as out-of-vocabulary (OOV) tokens, script and domain adaptation bottlenecks, and cross-lingual transfer barriers. Recent advances in BLM architectures and transfer methodologies have dramatically reduced their data and computational requirements, establishing BLMs as practical, high-performance alternatives to tokenization-based LMs.

1. Foundational Principles and Motivations

Traditional LLMs rely on a fixed tokenization scheme, typically BPE or variants, with vocabularies covering tens of thousands of subwords. This introduces several drawbacks:

• Large, static vocabularies cause inefficiency in multilingual and heterogeneous domains due to OOV tokens and inconsistent fragmentation across scripts or rare terms (Wei et al., 2021, Lee et al., 2022, Neitemeier et al., 17 Jan 2025).
• Subword tokenization is brittle under noise, spelling variations, or language shifts, leading to degraded generalization and robustness (Kallini et al., 2024, Pagnoni et al., 2024).
• Tokenizer dependence impedes interoperability (e.g., model ensembling, instruction post-training) and prevents seamless byte-level modeling for non-textual data including code, audio, and multimodal inputs (Hayase et al., 17 Jun 2025, Wu et al., 2024, Egli et al., 20 Feb 2025).

BLMs address these issues by omitting pre-tokenization and directly processing streams of UTF-8 bytes. Every input byte is in-vocabulary, and the same architecture generalizes across languages and modalities (Lee et al., 2022, Abonizio et al., 2022, Wu et al., 2024, Bhattacharyya et al., 21 May 2025). This endows BLMs with script agnosticism, open-vocabulary capabilities, and architectural uniformity for arbitrary byte streams (including images, binaries, and mixed-format data) (Egli et al., 20 Feb 2025, Yang et al., 2024).

2. Core Architectures and Methodological Advances

2.1 Hierarchical and Patch-Based Design

Direct byte-level modeling results in much longer input sequences, aggravating the quadratic cost of attention and posing challenges for training and inference. Recent BLMs employ architectural compression and dynamic patching to mitigate this:

• Hierarchical Encoder/Decoder Stacks: Models such as BLT (Byte Latent Transformer) and MBLM (Multiscale Byte LLM) introduce multi-stage processing pipelines where local encoders operate on sequences of bytes, group them into variable-length patches or chunks, and global transformers process patch-level states (Pagnoni et al., 2024, Egli et al., 20 Feb 2025).
• Dynamic Patching by Entropy: Entropy-driven patch boundary detection allows the model to allocate larger patches to predictable regions and shorter patches to high-entropy segments, optimizing FLops-per-byte and enhancing efficiency (Pagnoni et al., 2024).
• Boundary Prediction with Lookahead: Non-causal, one-byte lookahead routers (as in Bolmo and H-Net) enable the model to recover near-perfect alignment to teacher token boundaries, facilitating exact distillation and chunk-wise parallelism (Minixhofer et al., 17 Dec 2025, Bao et al., 1 Feb 2026).

2.2 Distillation and Byteification

Pretraining BLMs fully from scratch on trillions of bytes is computationally prohibitive (Bao et al., 1 Feb 2026). Modern BLM recipes exploit distillation from existing high-quality token-based LLMs:

• Two-Stage Distillation: The SOTA method comprises (1) progressive knowledge distillation, aligning encoder representations with teacher embeddings and transferring token distributions via KL-divergence, followed by (2) supervised byte-level fine-tuning for end-to-end byte generation (Bao et al., 1 Feb 2026, Minixhofer et al., 17 Dec 2025).
• Task-Parallel Loss Schedules: Staged optimization (embedding alignment, distribution matching, boundary learning) yields stable and resource-efficient training, with typical data budgets under ~125B bytes, <10% of full-scale BLM pretraining (Bao et al., 1 Feb 2026).
• Exact Likelihood Matching: Distillation objectives in Bolmo match the teacher's patch-wise and token-wise outputs exactly on every batch by aligning boundary and encoder losses, ensuring isoperformance with the origin model after minimal training (Minixhofer et al., 17 Dec 2025).
• Inference-Level Byteification: Alternatives such as ByteSampler derive byte-level output distributions from any token LLM at inference without retraining, enabling prompt-boundary-robust sampling and multi-model ensembling (Hayase et al., 17 Jun 2025).

2.3 Adaptive Sequence Compression

Sequence length and compression are critical in BLMs due to O(N) byte input. Approaches include:

• Dynamic Token Merging (MrT5): Learned delete gates after early transformer layers remove non-essential bytes and merge their content with surviving neighbors, yielding up to 80% sequence reduction and ~40% runtime improvement with sub-point accuracy drop (Kallini et al., 2024).
• Bit-level Compression: Bit-BPE, below the byte boundary, further compresses UTF-8 sequences by exploiting common high-bit prefixes in character-rich scripts, reducing sequence length by 3–6% on CJK and emoji texts losslessly (Moon et al., 9 Jun 2025).
• Hierarchical Embedding: Byte2Word and HAT collapse byte sequences into word-level or token-level states via attention or bidirectional encoders, shrinking memory and compute footprints with minimal performance trade-off (Lee et al., 2022, Neitemeier et al., 17 Jan 2025).

3. Training Regimes and Empirical Performance

Data Regimes

BLMs are trained either from scratch or via distillation on large-scale, diverse byte corpora:

• Monolingual and multilingual BLMs (MonoByte, BanglaByT5) achieve near-BERT parity across language tasks using only raw byte sequences (Abonizio et al., 2022, Bhattacharyya et al., 21 May 2025).
• Multimodal corpora (bGPT, MBLM) demonstrate unified capabilities in processing text, audio, image and binary files in a single model (Wu et al., 2024, Egli et al., 20 Feb 2025).

Performance

• Distilled BLMs: Token-trained LLMs converted via distillation retain >90% of teacher performance, with a typical mean drop of 2–3 points on MMLU-scale benchmarks (e.g., Llama-3.2 3B teacher 56.0 → BLM 51.8) for only ~125B bytes of training (Bao et al., 1 Feb 2026).
• Scaling Laws: BLTs with FLOP-matched settings achieve compute-parity or superiority relative to BPE LLMs at scale (up to 8B parameters, 4T training bytes) while providing up to 50% gain in inference FLOPS in high-compression regimes (Pagnoni et al., 2024, Minixhofer et al., 17 Dec 2025).
• Character and Coding Tasks: Byteified LMs (Bolmo) close or surpass subword LMs on fine-grained code generation and character understanding tasks; e.g., Bolmo-7B achieves 78.6% on CUTE vs. 56.9% for its teacher (Minixhofer et al., 17 Dec 2025).

Robustness

• BLMs are intrinsically robust to noise, casing, and corruption, with performance degradation under input perturbations 30–50% smaller than subword LMs (Lee et al., 2022, Neitemeier et al., 17 Jan 2025, Pagnoni et al., 2024).
• Byteification eliminates OOV and rare-token failure modes, particularly in morphologically rich or code-switched inputs (Bhattacharyya et al., 21 May 2025, Wei et al., 2021).

4. Applications Across Modalities and Domains

BLMs underpin applications beyond classical NLP:

• Omnimodal Modeling: bGPT and MBLM demonstrate state-of-the-art next-byte prediction across text, images (RGB bytes), audio (WAV bytes), symbolic music, and CPU trace simulation without any modality-specific encoder or vocabulary (Wu et al., 2024, Egli et al., 20 Feb 2025).
• Multimodal Visual QA: Byte-level models serialize images and questions as bytes, achieving comparable accuracy to CNN+LSTM architectures in VQA tasks (e.g., CLEVR) with fully autoregressive next-byte heads (Egli et al., 20 Feb 2025).
• Software Fuzzing: FuzzCoder formulates file mutation for program analysis as a sequence-to-sequence byte modeling task, increasing effective mutation rates and discovery of code vulnerabilities across ELF, MP3, JPG, and XML binary domains (Yang et al., 2024).

5. Comparative Analysis with Token-Driven Models

Table: Comparative Summary of Byte-Level vs. Token-Level LMs (metrics from cited works)

Aspect	BLMs	Token-Level LMs
OOV Handling	None (atomic bytes)	Frequent in long tail
Cross-script Generality	Full (single byte vocab)	Script/language-dependent
Robustness to Perturb.	High (noise, casing, etc.)	Low (token boundaries break)
Sequence Compression	Patch/hierarchical methods	Static (BPE subwords)
Domain Adaptation	Seamless (no retokenization)	Needs new or adapted vocab
Inference Efficiency	Comparable/Better at scale	Baseline (fixed token sizes)
Character/Code Tasks	Superior	Variable
Scaling Laws	Parity at 8B parameters	SOTA baseline

The above table summarizes that, with recent advances in patching, distillation, and boundary prediction, BLMs not only eliminate fundamental bottlenecks of token-driven models but match or exceed their performance and efficiency in many practical axes (Minixhofer et al., 17 Dec 2025, Pagnoni et al., 2024, Bao et al., 1 Feb 2026).

6. Limitations, Open Problems, and Future Directions

Despite rapid progress, BLMs face open research questions:

• Extreme Compression Limits: Excessive merging or deletion (>70% atoms) can begin to harm semantic accuracy on complex tasks, highlighting the need for learned, task-conditioned compression (Kallini et al., 2024).
• Scalability to Ultra-Long Contexts: Multiscale and hierarchical stacking (MBLM) now enable million-byte contexts in a single GPU via subquadratic complexity; extending to tens or hundreds of MB remains an open engineering challenge (Egli et al., 20 Feb 2025).
• Modality-Specific Inductive Biases: While BLMs are inherently modality-agnostic, some structured tasks (e.g. spatial visual reasoning) may benefit from explicit cross-patch or spatial attention, which current stacks lack (Egli et al., 20 Feb 2025).
• Bit-Granularity and Tokenization Fusion: Bit-BPE demonstrates that sub-byte tokenization further reduces redundancy, especially on CJK scripts and emoji, but requires embedding initialization and entropy management to avoid convergence slowdown (Moon et al., 9 Jun 2025).
• Ensembling and Proxy-Tuning: Inference-time byteification and logit-space task arithmetic (Bolmo, ByteSampler) enable new forms of ensemble and instruction porting, but practical deployment and hybrid-sampling behaviors require further study (Hayase et al., 17 Jun 2025, Minixhofer et al., 17 Dec 2025).
• Architectural Generalization: Future research includes byteification-friendly design from scratch (non-causal boundaries, subword-residual embeddings), end-to-end merger predictors, and efficient variable patch inference schemes (Minixhofer et al., 17 Dec 2025, Bao et al., 1 Feb 2026).

7. Resource Ecosystem and Implementation Guidance

• Open-source toolchains for BBPE/bit-BPE vocabulary creation and pretrained byte-level NEZHA models are publicly available, providing baseline infrastructure and reproducibility for research (Wei et al., 2021).
• Proof-of-concept multimodal and million-byte context scripts, along with core MBLM libraries (“mblm”) and patching utilities, are maintained by the AI4SD group (Egli et al., 20 Feb 2025).
• Byte2Word and H-Net recipes establish clear integration paths for field practitioners seeking to migrate from subword to byte-level architectures, with off-the-shelf code and detailed ablation studies (Lee et al., 2022, Bao et al., 1 Feb 2026).

---

Byte LLMs represent a convergence of architectural minimalism, universal coverage, and robust performance. By leveraging hierarchical compression, staged distillation, and patch-based modeling, BLMs now rival or exceed the best token-level models in text, code, and multimodal domains, while laying the foundation for tokenization-free, script-agnostic, and omnimodal AI systems (Bao et al., 1 Feb 2026, Minixhofer et al., 17 Dec 2025, Pagnoni et al., 2024, Egli et al., 20 Feb 2025, Neitemeier et al., 17 Jan 2025, Wu et al., 2024).