Byte-level Modeling: Approaches & Challenges

• Byte-level modeling is a tokenizer-free approach that encodes digital data as raw bytes, providing a universal representation for various modalities.
• It removes language-specific segmentation by eliminating tokenization, thereby reducing vocabulary fragmentation and handling noise effectively.
• While offering robust cross-domain applications, byte-level modeling faces computational challenges due to longer sequence lengths and efficiency trade-offs.

Byte-level modeling is a tokenizer-free or nearly tokenizer-free approach in which the basic modeling units are raw bytes rather than words, characters, or subwords. In this formulation, text is encoded as UTF-8 bytes, while non-text modalities can likewise be treated as byte sequences, so a single modeling framework can operate on text, network packets, genomic strings, order-book events, images, audio, or other digital artifacts using a small fixed alphabet of byte values, often augmented with a small number of special tokens (Bhattacharyya et al., 21 May 2025, Luo et al., 24 Mar 2025, Wu et al., 2024).

1. Conceptual basis and motivation

Byte-level modeling replaces language- or modality-specific segmentation with a representation that starts from the physical form in which digital data are actually transmitted or stored. In NLP, this means modeling raw UTF-8 bytes rather than BPE or SentencePiece tokens; in packet compression and traffic classification, it means treating headers and payloads as a single byte stream; in genomics, it means operating on single nucleotide characters rather than k-mers; and in financial event modeling, it means learning directly from packed binary records instead of discretized symbolic fields (Bhattacharyya et al., 21 May 2025, Luo et al., 24 Mar 2025, Malusare et al., 2023, Li et al., 4 Aug 2025).

A central motivation is the removal of tokenizer-induced assumptions. Subword tokenizers learn a large vocabulary of statistically frequent segments and thereby impose fixed boundaries that may not align with morphology, orthography, protocol structure, or local edit operations. The literature repeatedly describes the resulting failure modes as vocabulary explosion and fragmentation in morphologically rich languages, brittle behavior under spelling variation and noisy text, loss of precision under discretization of numerical data, and prompt-boundary artifacts when prompts end inside a token (Bhattacharyya et al., 21 May 2025, Wang et al., 2019, Li et al., 4 Aug 2025, Hayase et al., 17 Jun 2025).

Another motivation is universality. The fixed byte alphabet is language-agnostic and, in the general case, modality-agnostic. This property is emphasized in multilingual machine translation with byte-level BPE, Bangla language modeling, byte-level digital-world simulation, and universal packet compression. In each case, the same representational substrate avoids language-specific tokenizers, modality-specific parsers, or handcrafted symbol inventories (Wang et al., 2019, Bhattacharyya et al., 21 May 2025, Wu et al., 2024, Luo et al., 24 Mar 2025).

The main cost of this universality is sequence length. Byte sequences are typically much longer than subword sequences, so byte-level modeling moves complexity from tokenization into the sequence model itself. Much of the recent literature can therefore be read as a search for architectural or inference-time mechanisms that preserve byte-level fidelity while controlling the resulting computational burden (Kallini et al., 2024, Lee et al., 13 May 2026).

2. Representation and learning objectives

The basic representational object is a sequence of bytes,

$$p_\theta(x_{1:T})=\prod_{t=1}^{T} p_\theta(x_t\mid x_{<t}),$$

with $x_t \in \{0,\dots,255\}$ in the simplest case. Several systems extend this alphabet with structural tokens. ByteTrans, for example, casts raw bytes to int16 and adds an end-of-packet marker 256 and a padding marker 257, yielding a 258-symbol vocabulary for fixed-length packet sequences of length 256 (Luo et al., 24 Mar 2025). bGPT uses the 256 byte values plus an additional special symbol for padding or end-of-patch, yielding a 257-way discrete vocabulary (Wu et al., 2024). ENBED uses a byte-level tokenizer with 384 tokens, comprising the 256 extended ASCII codes together with special tokens such as MASK, PAD, UNKNOWN, and multiple distinct mask variants (Malusare et al., 2023). BanglaByT5 likewise uses a 384-token byte-level tokenizer, including 100 special tokens, while still preserving the character of byte-level modeling (Bhattacharyya et al., 21 May 2025).

The dominant learning objective is autoregressive next-byte prediction. This appears in packet compression, order-book event generation, digital-world simulation, multiscale byte language modeling, and many hierarchical byte models. Under this objective, better next-byte prediction translates directly into lower empirical entropy, shorter code length, or lower bits-per-byte (Luo et al., 24 Mar 2025, Li et al., 4 Aug 2025, Wu et al., 2024, Egli et al., 20 Feb 2025).

Byte-level denoising objectives are also prominent. BanglaByT5 applies the T5 span-corruption objective at the byte level, masking contiguous spans of bytes and reconstructing them with an encoder-decoder Transformer (Bhattacharyya et al., 21 May 2025). ENBED uses masked language modeling with 15% masked tokens and contiguous masked spans, optimized over byte-level genomic sequences (Malusare et al., 2023). The efficiency-gap study further analyzes masked diffusion modeling on bytes, where independent masking destroys local contiguity and leads to markedly worse scaling behavior than autoregressive byte modeling (Lee et al., 13 May 2026).

A distinct line of work studies byte-level distributions as an inference-time transformation of a subword model rather than as a separately trained model. The Valid Covering Tree construction converts any autoregressive LM with a deterministic BPE tokenizer into a character-level or byte-level LM at inference time, exactly marginalizing over valid tokenizations and thereby preserving the model’s text-level generative distribution while resolving the Prompt Boundary Problem (Hayase et al., 17 Jun 2025).

3. Architectural families

The architectural literature is diverse. Byte-level modeling is not restricted to flat next-byte Transformers; it includes encoder-decoder denoisers, hierarchical latent-token models, selective state-space models, dynamic deletion or merging mechanisms, continuous compression schemes, and distillation-based byteification.

Family	Representative systems	Core byte-level idea
Flat autoregressive decoders	ByteTrans, bGPT	Predict next byte directly from past bytes
Encoder-decoder denoisers	BanglaByT5, ENBED	Learn byte representations via span corruption or MLM
Selective SSMs	MambaNetBurst	Process every byte at native temporal resolution without tokenization
Hierarchical or latent-patch models	MBLM, ByteFlow, HoloByte	Compress long byte streams into shorter latent or patch sequences
Distilled or inference-time byteification	Bolmo, Valid Covering Tree	Convert subword LMs into byte-level behavior without training from scratch

Decoder-only Transformers remain common in domains where exact next-byte distributions are the central object. ByteTrans uses a multi-layer decoder-only Transformer over packet bytes, and ByteGen uses an H-Net-style hybrid Mamba-Transformer to autoregress over 32-byte packed order-book events (Luo et al., 24 Mar 2025, Li et al., 4 Aug 2025). By contrast, BanglaByT5 and ENBED employ encoder-decoder Transformers because denoising and sequence-to-sequence tasks benefit from a separate encoder state and cross-attention over the input byte sequence (Bhattacharyya et al., 21 May 2025, Malusare et al., 2023).

Selective state-space models provide another major branch. MambaNetBurst performs direct undiluted byte-to-classification learning on raw packet bursts with a stack of residual pre-normalized Mamba-2 blocks, and its ablations argue that preserving byte-level temporal resolution is critical while early downsampling through striding is consistently harmful (Kulatilleke et al., 11 May 2026). In long-context byte language modeling, MBLM treats each stage as a generic autoregressive block, allowing Transformer, Mamba-2, or hybrid stage models inside a multiscale decoder stack (Egli et al., 20 Feb 2025).

Hierarchical architectures are especially common because they decouple local byte processing from global reasoning. MBLM organizes bytes into nested patches across multiple stages; ByteFlow learns adaptive segmentation by selecting positions with large marginal coding rate under a fixed Top-$K$ budget; HoloByte superimposes fixed-capacity chunks of bytes into continuous hyperspherical vectors and delegates most long-range computation to a macroscopic Transformer over chunk states (Egli et al., 20 Feb 2025, Deng et al., 3 Mar 2026, Khasia, 10 Mar 2026). These systems differ substantially in mechanism, but all move away from the idea that every byte must participate in full global attention at native resolution.

4. Efficiency, scaling, and compression

The fundamental systems problem in byte-level modeling is sequence inflation. MrT5 formulates this explicitly: moving from subwords to bytes increases sequence length dramatically, making training and inference inefficient under standard self-attention (Kallini et al., 2024). The efficiency-gap study sharpens the point by showing that the byte penalty depends strongly on the learning objective. Under autoregressive modeling, the compute overhead of bytes relative to BPE shrinks with scale and is extrapolated to parity around $1.3\times10^{22}$ FLOPs, whereas under masked diffusion the overhead remains severe and parity is extrapolated only around $4.2\times10^{26}$ FLOPs (Lee et al., 13 May 2026). This suggests that byte-level viability is objective-dependent rather than uniform.

Several mechanisms have been proposed to reduce this burden. MrT5 adds a learned deletion gate in the encoder, dynamically shortening byte sequences after a few initial layers and reducing sequence lengths by up to 75% with 25–55% inference speedups while largely preserving downstream performance (Kallini et al., 2024). MBLM uses a model-agnostic hierarchy of patch-level decoders and reports training with context windows of 5M bytes on a single GPU, with hybrid Mamba–Transformer configurations exhibiting near-linear generational efficiency (Egli et al., 20 Feb 2025). HoloByte replaces direct byte-level attention with continuous hyperspherical chunk embeddings, reducing the attention term from $\mathcal{O}(N^2D)$ to $\mathcal{O}\left(\frac{N^2}{W^2}D + ND^2\right)$ while retaining exact byte-level decoding through a localized causal micro-decoder (Khasia, 10 Mar 2026).

Adaptive compression is another recurrent theme. ByteFlow computes a lossy coding rate over contextualized local byte representations, ranks positions by marginal coding-rate gain, and uses Top-$K$ selection to preserve a static computation graph while learning input-dependent segmentation (Deng et al., 3 Mar 2026). ByteTrans uses a different compression interface: it predicts the next-byte distribution, converts each observed byte into its rank in that distribution, and then compresses the resulting sparse rank sequence with zlib; the reported best compression ratio is 46.3%, with improvements over vanilla zlib of 11.8% to 14.6% (Luo et al., 24 Mar 2025).

Generation speed has recently become a separate subproblem. Fast Byte Latent Transformer introduces BLT-D, BLT-S, and BLT-DV, combining block-wise diffusion, self-speculation, and verification so that multiple bytes can be proposed per decoding step. Across these methods, the paper reports estimated memory-bandwidth cost reductions over 50% relative to BLT on generation tasks (Kallini et al., 8 May 2026). Bolmo takes yet another route: instead of training a large byte model from scratch, it byteifies existing subword LMs with an exact distillation objective and less than 1% of a typical pretraining token budget, then trains end-to-end as a byte model (Minixhofer et al., 17 Dec 2025).

5. Applications across domains

Byte-level modeling is no longer confined to text. In NLP, BanglaByT5 demonstrates that a 300M byte-level encoder-decoder model can outperform similarly sized Bangla-specific subword models and remain within 2–5% of much larger multilingual systems on several benchmarks, while also benefiting from robustness to spelling variation and morphology (Bhattacharyya et al., 21 May 2025). In multilingual machine translation, byte-level BPE offers a compromise between pure bytes and character-based BPE: it is OOV-free, achieves comparable translation quality, and uses a vocabulary about 1/8 the size of character-based BPE in the reported settings (Wang et al., 2019).

In networking, byte-level methods are especially natural because all payloads and headers are already bytes on the wire. ByteTrans models heterogeneous packets with a single autoregressive Transformer and rank-based post-processing, achieving compression ratios below 50% and successful deployment on servers, Nvidia Jetson devices, and Raspberry Pi 4B (Luo et al., 24 Mar 2025). MambaNetBurst addresses a different task—burst-level traffic classification—and shows that a compact tokenizer-free byte-level classifier can be competitive with or outperform substantially heavier and often pre-trained baselines across encrypted app identification, VPN/Tor traffic, malware traffic, and IoT attack traffic (Kulatilleke et al., 11 May 2026).

In genomics, ENBED argues that byte-level precision is particularly important because single-base changes, sequencing mismatches, insertions, and deletions have outsized biological consequences. Its encoder-decoder foundation model improves over prior genomic Transformers on regulatory classification tasks and shows a substantial advantage in influenza mutation generation, with Top-1 accuracy 76.9% versus 32.0% for a BPE-tokenized encoder-decoder baseline and mean Levenshtein distance 2.3 versus 30.6 (Malusare et al., 2023).

In finance, ByteGen treats each limit order book event as a fixed 32-byte packed record and trains a tokenizer-free H-Net-style generator on over 34 million CME Bitcoin futures events. The resulting model reproduces heavy-tailed returns, bursty event timing, realistic price distributions, and competitive market-quality metrics without discretizing prices or quantities into handcrafted token vocabularies (Li et al., 4 Aug 2025).

A broader multimodal ambition is visible in bGPT and MBLM. bGPT models raw binary files from text, audio, images, symbolic music, and CPU traces, reporting 0.0011 bits per byte on ABC-to-MIDI conversion and CPU execution accuracy exceeding 99.99% (Wu et al., 2024). MBLM extends hierarchical byte language modeling to million-length contexts and reports that a pure next-token-prediction byte model can match a custom CNN-LSTM architecture with a designated classification head on CLEVR-style visual question answering despite serializing images and using no encoder (Egli et al., 20 Feb 2025).

6. Limitations, misconceptions, and open directions

The main limitation remains computational. Longer byte sequences imply higher memory use, slower attention, and larger decoding depth unless some form of compression, hierarchy, or state-space approximation is introduced. This has led to a persistent misconception that byte-level modeling is synonymous with an impractical flat Transformer over raw bytes. The literature does not support that simplification: it includes dynamic deletion, multiscale patching, adaptive compression, continuous chunk embeddings, latent-token decoders, selective SSMs, and exact inference-time byteification of token LMs (Kallini et al., 2024, Egli et al., 20 Feb 2025, Khasia, 10 Mar 2026, Hayase et al., 17 Jun 2025).

A second misconception is that eliminating tokenization automatically eliminates structural bias. In practice, byte-level systems often reintroduce structure through boundaries, patches, rank transforms, chunkers, or distillation targets. ByteFlow learns boundaries via marginal coding rate; MrT5 learns where to delete; BLT variants draft and verify blocks; Bolmo learns byte-level behavior through exact distillation from a subword teacher (Deng et al., 3 Mar 2026, Kallini et al., 2024, Kallini et al., 8 May 2026, Minixhofer et al., 17 Dec 2025). This suggests that the real design question is not whether structure should exist, but where it should reside: outside the model in a fixed tokenizer, or inside the model as a learned and input-dependent mechanism.

A third open issue is objective selection. The efficiency-gap study argues that byte-level modeling interacts differently with different generative objectives: autoregressive byte models can rediscover subword-like structure efficiently, whereas masked diffusion on bytes suffers from context fragility because masking destroys the local contiguity needed to resolve semantics from raw bytes (Lee et al., 13 May 2026). This implies that conclusions drawn from autoregressive byte models do not automatically transfer to non-autoregressive byte models.

Finally, several domain-specific limits remain visible in the current systems. ByteTrans restricts packets to 256-byte segments and does not model cross-packet dependencies explicitly (Luo et al., 24 Mar 2025). BanglaByT5 uses a 512-token context and acknowledges that longer inputs are especially costly in the byte regime (Bhattacharyya et al., 21 May 2025). MambaNetBurst shows that early striding harms traffic classification, but longer contexts and streaming inference remain open directions (Kulatilleke et al., 11 May 2026). ENBED reaches 16k-token genomic contexts through structured attention, yet megabase-scale genomic reasoning remains out of scope (Malusare et al., 2023).

Taken together, the literature presents byte-level modeling not as a single model class but as a broad research program. Its common premise is that bytes are the most stable and universal discrete substrate for digital data. Its central technical challenge is to recover the efficiency, abstraction, and inductive structure that tokenization used to provide, while retaining byte-level fidelity. The most successful recent work therefore combines raw-byte representations with learned multiscale structure rather than choosing between “bytes” and “tokens” as a strict dichotomy (Egli et al., 20 Feb 2025, Minixhofer et al., 17 Dec 2025).