Bolmo: Byteifying the Next Generation of Language Models (2512.15586v1)

Abstract: We introduce Bolmo, the first family of competitive fully open byte-level LMs at the 1B and 7B parameter scales. In contrast to prior research on byte-level LMs, which focuses predominantly on training from scratch, we train Bolmo by byteifying existing subword-level LMs. Byteification enables overcoming the limitations of subword tokenization - such as insufficient character understanding and efficiency constraints due to the fixed subword vocabulary - while performing at the level of leading subword-level LMs. Bolmo is specifically designed for byteification: our architecture resolves a mismatch between the expressivity of prior byte-level architectures and subword-level LMs, which makes it possible to employ an effective exact distillation objective between Bolmo and the source subword model. This allows for converting a subword-level LM to a byte-level LM by investing less than 1\% of a typical pretraining token budget. Bolmo substantially outperforms all prior byte-level LMs of comparable size, and outperforms the source subword-level LMs on character understanding and, in some cases, coding, while coming close to matching the original LMs' performance on other tasks. Furthermore, we show that Bolmo can achieve inference speeds competitive with subword-level LMs by training with higher token compression ratios, and can be cheaply and effectively post-trained by leveraging the existing ecosystem around the source subword-level LM. Our results finally make byte-level LMs a practical choice competitive with subword-level LMs across a wide set of use cases.

• The paper presents a two-stage byteification process that distills subword LMs into efficient byte-level models with less than 1% pretraining overhead.
• It leverages a non-causal boundary predictor and mLSTM-based local encoders/decoders to enhance fine-grained tokenization and overall inference speed.
• Empirical results demonstrate that Bolmo 7B outperforms previous byte-level models in code generation, character understanding, and efficiency, while supporting seamless post-training integration.

Bolmo: Byteifying the Next Generation of LLMs

Motivation and Background

Bolmo addresses persistent limitations of subword tokenization that have long dominated the design of modern LMs, including deficient handling of fine-grained character patterns, tokenization bias, rigid vocabulary bottlenecks, and constraints on computational allocation. Byte-level LMs have been proposed as a compelling alternative, but have historically underperformed owing to data and compute inefficiencies and architectural mismatches when trained from scratch. The Bolmo approach instead byteifies existing subword-level LMs—transferring knowledge into a flexible, byte-level architecture with a near-negligible pretraining token budget increment (less than 1%) over the already established subword model.

Bolmo Architecture Overview

Bolmo adopts and extends the Latent Tokenizer LLM (LTLM) paradigm, but is architected specifically for knowledge-distillation-based byteification. The architecture comprises several key components:

• Tokenization Embedding: Each UTF-8 byte is embedded. Critically, embeddings are augmented via sparse residual addition of subword token embeddings (from the source LM) based on matching suffixes, improving representational capacity and compute efficiency (Figure 1).

  Figure 1: The Bolmo architecture, highlighting the flow from byte tokenization, embedding, and local encoding, through boundary prediction and pooling, to global modeling, then back through depooling and final local decoding and output.

• Local Encoder: A single mLSTM block provides localized contextualization. Section capacity is flexibly increased via residual subword embeddings instead of scaling the local RNN stack size.
• Boundary Predictor: Unlike prior LTLMs enforcing causal (incremental) patch boundaries, Bolmo incorporates a non-causal boundary predictor during prefill, allowing a single-byte lookahead (future context) to match the expressivity of subword tokenization. Boundary bytes are predicted during generation via a dedicated boundary token (Figure 2).

  Figure 2: Comparative diagram of how Bolmo's non-causal boundary prediction (right) restores the expressivity of subword-level LMs versus prior byte-level LTLMs (middle) and classical subword LMs (left).

• Pooling/Depooling: As in prior LTLMs, patches (latent tokens) are formed by pooling last-byte representations at each predicted boundary, and depooling reassigns global-layer outputs to the corresponding byte sequences.
• Local Decoder: Multiple mLSTM layers recontextualize depooled representations prior to byte-wise next-token prediction.
• Language Modeling Head: Projects final representations to the byte vocabulary, supporting a fused extension where boundary prediction is included at the output layer.

Byteification Procedure

The Bolmo training pipeline is defined as a two-stage process:

1. Stage 1: Subword-to-Byte Distillation Only the local encoder, local decoder, boundary predictor, and LM head are trained, with the global model frozen. The objective is to exactly recover the source subword LM's behavior by (i) mimicking subword patch boundaries, (ii) ensuring local representations after pooling match pooled subword embeddings, and (iii) matching local decoder outputs to subword-level LM log-likelihoods.
2. Stage 2: End-to-End Training The entire model is trained with cross-entropy and boundary losses, allowing the global model to adjust to byte-level information and, optionally, higher compression factors for efficiency. This phase leverages the prior initialization to rapidly converge with minimal continued training.

Main Empirical Results

Bolmo 7B and 1B models considerably outperform all prior fully open byte-level LMs. On challenging benchmarks, including code generation, math, multiple-choice QA, and especially fine-grained character understanding tasks (e.g., CUTE, EXECUTE), Bolmo achieves state-of-the-art byte-level results and in many cases surpasses the source subword LM—particularly on character-sensitive tasks.

Strong empirical results:

• Character Understanding: Bolmo 7B achieves massive gains over Olmo 3 on both CUTE (78.6% vs. 56.9%) and EXECUTE (71.6% vs. 55.1%).
• Code Generation: Bolmo 7B is competitive or better than Olmo 3—HumanEval pass@16 increases from 71.1% to 74.7%.
• Compression/Efficiency Frontier: Bolmo dominates the Pareto frontier for efficiency versus task performance at high compression factors, due to being unconstrained by vocabulary softmax overhead (Figure 3).

  Figure 3: Task performance versus efficiency for subword LMs (with increasing vocabulary/bytes per patch via SuperBPE) and Bolmo, indicating the clear efficiency advantage of byte-level LTLMs at higher compression rates.

• Inference Speed: Using a compact mLSTM-based local model, Bolmo's wallclock decoding speed nears or exceeds baseline subword LMs, particularly as compression increases (Figure 4).

  Figure 4: Decoding throughput and prefilling latency for Bolmo 7B and source model across compression settings; right, candidate local architectures demonstrate throughput (bytes/sec) and wallclock latency superiority for optimally chosen local models.

• Zero-Cost Post-Training: Via Task Arithmetic, post-trained subword LMs (e.g., RL-fine-tuned Olmo 3) can be seamlessly fused into Bolmo, immediately conferring gain on instruction-following benchmarks with no additional training cost (Figure 5).

  Figure 5: Zero-cost post-training: infusing instruction-tuned model capabilities via weight difference addition, showing matched instruction-following eval gains.

Ablation Analysis

• Boundary Prediction: Non-causal predictors are essential—causal (incremental) predictors force a trade-off between patch semantic coherence and accurate boundary alignment, while non-causal predictors resolve both simultaneously (Figure 6).

  Figure 6: Ablation of causal vs. non-causal boundary predictors—only non-causal prediction achieves both high patch accuracy and representation fidelity post-stage 1.

• Training Procedure: Two-stage training improves convergence and final accuracy, but is not strictly required; direct end-to-end training is feasible but sub-optimal, especially for smaller models (Figure 7).

  Figure 7: Bits-per-byte training trajectory showing an enduring advantage from using two-stage over end-to-end training alone.

• Local Model Design: Careful selection of local encoder/decoder architecture for inference speed is critical—FLOP counts alone are insufficient; wallclock speed must guide design, with mLSTM (in TFLA form) found optimal under realistic deployment regimes.

Theoretical and Practical Implications

Bolmo rigorously demonstrates that with carefully controlled byteification and architecture design, byte-level LMs are now a practical, competitive alternative to subword models, breaking through the bottlenecks of subword units in both accuracy and efficiency. The techniques detailed for embedding retention, non-causal patch boundary prediction, two-phase training, and ecosystem-compatible post-training protocols collectively enable rapid deployment and extensibility of byte-level LMs, leveraging the full ecosystem and prior investments of high-quality subword models.

The architecture and training approach facilitate further research in:

• End-to-end learning of non-causal boundary policies.
• Scaling local encoder/decoder capacity for larger patch sizes and improved compute/perf trade-offs.
• Multibyte/parallel prediction regimes for further inference speedups.
• Equitable sub-byte atomic units for greater language inclusivity.

Conclusion

Bolmo establishes byteification as a cost-effective, robust path to high-performance byte-level LMs, making them competitive with, and on certain axes superior to, state-of-the-art subword LMs at scale. The approach enables fast architectural iteration, improved efficiency, expanded language universality, and direct leverage of the subword LM's entire post-training ecosystem.

Figures Index

• Figure 1: Bolmo full architecture schematic.
• Figure 2: Comparison of subword, causal byte-level, and Bolmo non-causal boundary prediction.
• Figure 3: Performance/efficiency Pareto frontier with compression.
• Figure 5: Zero-cost post-training via Task Arithmetic.
• Figure 6: Ablation study on boundary prediction schema.
• Figure 7: Two-stage vs. one-stage bits-per-byte trajectories.
• Figure 4: Inference throughput and latency evaluation.

Minixhofer, B. et al., "Bolmo: Byteifying the Next Generation of LLMs" (2512.15586)