Byte Latent Transformer: Patches Scale Better Than Tokens (2412.09871v1)

Abstract: We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.

The paper "Byte Latent Transformer: Patches Scale Better Than Tokens" (Pagnoni et al., 13 Dec 2024 ) introduces the Byte Latent Transformer (BLT), a novel tokenizer-free architecture for LLMs. BLT operates directly on raw byte data, aiming to match the performance of traditional tokenization-based LLMs at scale while offering significant improvements in inference efficiency and robustness.

The core idea behind BLT is to move away from fixed-vocabulary tokenization, which the authors argue biases models and leads to shortcomings like sensitivity to noise, lack of orthographic knowledge, and multilingual inequity. Instead, BLT processes bytes grouped into dynamically sized "patches." These patches serve as the primary units for computation by the computationally expensive Latent Transformer component. Compute is dynamically allocated based on data complexity, specifically by segmenting patches based on the entropy of the next byte prediction from a smaller byte-level LLM.

The BLT architecture comprises three main modules:

1. Local Encoder: A lightweight transformer that encodes input bytes into patch representations. It uses local attention and cross-attention to pool byte information into patch representations. It also incorporates hash n-gram embeddings to provide contextual information about preceding bytes.
2. Latent Global Transformer: A large autoregressive transformer that operates on the sequence of patch representations. This module is the main consumer of compute. Its computational cost is directly tied to the number of patches.
3. Local Decoder: Another lightweight transformer that decodes the patch representations back into bytes, predicting the next byte in the sequence. It uses cross-attention (with roles reversed compared to the encoder) and local attention.

Patching functions determine how bytes are grouped. BLT explores dynamic entropy-based patching, where new patches start when the estimated next-byte entropy exceeds a threshold or increases significantly. This contrasts with static strided patching (fixed number of bytes per patch, like in MegaByte) or rule-based space patching (like in SpaceByte). Entropy patching allows the model to allocate more compute (smaller patches, more transformer steps) to complex or unpredictable byte sequences and less compute (larger patches, fewer steps) to predictable sequences. A key distinction from BPE tokenization is that BLT's patching is incremental (does not depend on future bytes) and does not use a fixed vocabulary, preserving access to underlying byte information.

The authors conduct extensive scaling studies, comparing BLT with Llama 3 (a state-of-the-art BPE-based model) in terms of Bits-Per-Byte (BPB) on training data and performance on downstream tasks.

Practical Implications and Findings:

1. Compute-Optimal Scaling: BLT models demonstrate the feasibility of training byte-level models at scale. In compute-optimal scaling studies up to 8B parameters and 4T training bytes, BLT models matched or slightly outperformed BPE Llama 3 models in terms of BPB on the training distribution. This confirms that byte-level models can be competitive with token-based models at current scales.
2. Inference Efficiency: A major advantage of BLT is its inference efficiency. Since the core Latent Transformer operates on patches, increasing the average patch size reduces the number of steps and thus the FLOPs per byte during inference. BLT achieved comparable performance to Llama 3 while using significantly larger average patch sizes (e.g., 6 or 8 bytes vs. Llama 3's 4.4 bytes), leading to potential inference FLOP savings of up to 50%.
3. New Scaling Axis: BLT introduces a new way to scale models under a fixed inference budget. By increasing the average patch size (reducing inference steps), the saved compute can be reallocated to build a larger Latent Transformer, leading to improved performance within the same inference cost constraint. Fixed-inference scaling studies show that BLT models with larger patch sizes exhibit better scaling trends than BPE models, overtaking them beyond the compute-optimal point.
4. Enhanced Robustness and Character Awareness: Operating directly on bytes makes BLT models inherently more robust to input noise (e.g., capitalization changes, character drops/repeats) and grants them a deeper understanding of sub-word and character-level structures. Evaluations on noisy HellaSwag, Phonology-G2P, and the CUTE benchmark show significant performance gains for BLT compared to Llama 3 models, even against a larger Llama 3.1 model trained on significantly more data, suggesting this byte-level capability is hard for BPE models to acquire solely through scale.
5. Improved Long-Tail Generalization: BLT shows better performance on low-resource machine translation tasks (FLORES-101), outperforming Llama 3, particularly in translation to/from lower-resource languages. This is attributed to the byte-level model's ability to generalize better to less common byte sequences encountered in such languages.
6. "Byte-ifying" Pre-trained Models: The paper demonstrates that initializing BLT's Latent Transformer with weights from a pre-trained BPE model (Llama 3.1) and continuing training can significantly improve performance compared to training BLT from scratch with the same compute budget. This suggests a practical pathway to leverage existing large BPE models to train performant tokenizer-free BLT models more efficiently.

Implementation Considerations:

• Patching Implementation: The dynamic entropy-based patching requires training a separate, smaller byte-level LM to estimate next-byte entropies. This entropy model needs to be efficient enough for inference or pre-compute entropies. Patch boundary identification occurs during data loading.
• Architecture: BLT's architecture involves specialized cross-attention layers connecting byte-level and patch-level representations. Efficient implementations, like using FlexAttention for dynamic patch masks, are necessary for competitive wall-clock training times.
• FLOPs vs. Wall-Clock Time: While FLOPs analysis shows theoretical efficiency gains, achieving parity or exceeding BPE models in real-world wall-clock time requires highly optimized implementations of the local modules and cross-attention, which might not be as mature as standard transformer implementations.
• Dynamic Patching Complexity: Dynamic patching leads to variable bytes-to-patches ratios in batches. Efficient training requires packing patches to utilize compute resources fully and potentially padding/truncating byte sequences to manage memory spikes from very large patches.
• Scaling Laws: The optimal balance between model size, data, and patch size for BLT might differ from that for BPE models (like Llama 3). Determining BLT-specific scaling laws is identified as future work.

Overall, BLT presents a compelling case for moving beyond fixed-vocabulary tokenization by demonstrating that byte-level models can scale effectively, match or exceed the performance of token-based models, and offer inherent advantages in efficiency, robustness, and handling diverse data distributions, particularly at scales relevant to current LLM development.