Incremental BPE Tokenization

Abstract: We propose a novel algorithm for incremental Byte Pair Encoding (BPE) tokenization. The algorithm processes each input byte in worst-case $\mathcal{O}(\log^2 t)$ time, leading to an overall complexity of $\mathcal{O}(n \log^2 t)$, where $n$ is the input length and $t$ is the maximum token length. The algorithm incrementally maintains BPE tokenization results for every prefix of the input text, implementing the standard BPE merge procedure defined by a fixed set of merge rules. This enables efficient partial tokenization in streaming settings. Functioning as a drop-in replacement for standard BPE, our approach achieves a speedup of up to ${\sim}3\times$ over Hugging Face's tokenizers, and demonstrates significant latency reductions over OpenAI's tiktoken on pathological inputs. We further introduce an eager output algorithm that enables streaming output, emitting tokens as soon as token boundaries are determined during incremental tokenization. Overall, our results demonstrate that BPE tokenization can be performed incrementally with strong worst-case guarantees, while providing practical latency benefits in modern LLM pipelines. Code: https://github.com/ModelTC/mtc-inc-bpe

• The paper introduces an incremental BPE algorithm that offers streaming tokenization with strict O(n log² t) worst-case guarantees and semantic equivalence to canonical BPE.
• It leverages a novel prefix-consistency framework with efficient data structures like an augmented Aho–Corasick automaton and centroid decomposition for rapid token lookup.
• Empirical evaluations demonstrate significant speedups and robust performance on adversarial inputs, paving the way for low-latency, energy-efficient LLM pipelines.

Incremental BPE Tokenization: Algorithmic Foundations and System Implications

Introduction and Background

Byte Pair Encoding (BPE) remains the canonical subword tokenization approach for modern LLMs, due to its efficiency in balancing vocabulary size with expressive power. Yet, BPE tokenizers implemented in prevalent frameworks such as Hugging Face tokenizers and OpenAI's tiktoken fundamentally operate in an offline mode, requiring full input observation prior to producing a canonical tokenization. This constraint introduces serialization bottlenecks, prohibits fine-grained pipelining with model inference, and exposes systems to adverse worst-case complexity ($\mathcal{O}(n^2)$) under degenerate input scenarios.

The paper "Incremental BPE Tokenization" (2605.30813) proposes an algorithmic and structural revision to BPE tokenization, addressing these deficiencies by providing incremental, streaming-compatible BPE with strict worst-case guarantees and semantic equivalence to standard BPE pipelines.

Figure 1: The standard LLM tokenization pipeline, where BPE is applied after a sequence of model-specific preprocessing stages. The proposed method targets the BPE stage to be incremental without affecting correctness or surrounding stages.

Core Theoretical Contributions

Structural Analysis

The authors formalize prefix-consistent BPE tokenization, capitalizing on the observation that tokenizations for all prefixes of a string form a hierarchical prefix tree. They introduce key constructs:

• Last Token ($\theta(s)$): The last token in the BPE tokenization of string $s$, with the recursive property $T(s) = T(s_{pre}) \oplus [\theta(s)]$, where $s_{pre}$ is $s$ with $\theta(s)$ removed.
• Prefix Consistency Lemma: For any prefix $\mu$ of the token sequence, $T(\pi(\mu)) = \mu$, showing each prefix remains a valid tokenization of its substring.
• Successor Forest: A directed, acyclic graph over tokens, with edges pointing to token successors as defined by canonical BPE merge rules.
• Suffix-Successor Tree: For token $t$, the subgraph induced by all canonical suffix tokens of $\theta(s)$0.

  Figure 2: Visualization of the Successor Forest, solid edges are successor relationships, providing the backbone for efficient incremental token search.

The core structural theorem, the Monotonic Path Property, states: for a given input string, the valid suffix tokens eligible as the last token ($\theta(s)$1) form a single monotonic path in the Suffix-Successor Tree rooted at the longest canonical suffix.

Figure 4: Inductive visualization of Prefix Consistency: for any post-merge prefix, there is exactly one valid corresponding pre-merge prefix maintaining string identity, guaranteeing correctness of truncation.

Algorithmic Framework

Building on these foundations, the incremental algorithm is decomposed into:

1. Search Space Identification: Efficiently locating the longest matching suffix token using an augmented Aho–Corasick automaton, providing $\theta(s)$2 lookups due to optimized, tiled transition tables.
2. Path Extension Search: Efficiently identifying the deepest valid node in the Suffix-Successor Tree (the true $\theta(s)$3). This is achieved via a Centroid Decomposition of the tree, guaranteeing $\theta(s)$4 update per byte, where $\theta(s)$5 is max token length.

A critical technical maneuver involves DFS linearization, assigning interval labels to tree fragments such that candidate validation reduces to constant-time interval membership tests.

Eager Output and Streaming

To enable streaming output, the authors introduce an eager output mechanism. As input is incrementally ingested, the stable prefix of tokenizations (shared by all possible future extensions) is emitted immediately once determined. This is tracked by dynamically maintaining the “active frontier” in the prefix tree and advances in amortized $\theta(s)$6 time per operation.

Benchmarks and Empirical Findings

The incremental BPE is implemented in Rust as a drop-in replacement for the core BPE stage in existing tokenization frameworks. Across diverse tasks (English, Chinese, and Code datasets), it achieves measurable end-to-end speedups, especially in regimes where pre-tokenization is coarse (code or Chinese), and demonstrates stable throughput on adversarial inputs where heap-based implementations degrade super-linearly.

Figure 3: End-to-end throughput under pathological inputs (log-log scale): incremental BPE delivers robust performance while tiktoken degrades as input grows, consistent with its unfavorable $\theta(s)$7 BPE algorithm.

Performance profiling reveals that in modern pipelines, the BPE gap is often closed to the point where non-BPE pipeline stages (normalization, regex-based pre-segmentation) dominate.

Figure 5: Flame graph of Qwen-3 tokenization pipeline: the incremental BPE comprises only 13% of overall execution—most cost is from normalization and pre-tokenization.

Figure 6: For tiktoken (O200K model), regex pre-tokenization dominates, with BPE merge itself constituting a small fraction of total runtime.

Latency distributions across different BPE implementations demonstrate that the new incremental mechanism matches or exceeds the best prior incremental algorithms (including Rust Gems’ backtracking and table-based methods) while outperforming the original tiktoken BPE.

Figure 8: Distribution of end-to-end execution times for various BPE implementations, indicating consistent or superior latency for incremental BPE.

Bold and Contradictory Claims

• The authors claim strict semantic equivalence to standard BPE, asserting that their streaming/incremental approach produces identical tokenizations to the non-incremental canonical BPE (where dictionary properization is possible).
• They assert strict $\theta(s)$8 worst-case guarantees, independent of input characteristics, eliminating the quadratic worst-case behavior inherent in other major BPE implementations.
• The work suggests that pre-tokenization and chunking—deployed in practice primarily to curb BPE complexity—become algorithmically redundant in the presence of this approach.

Implications and Future Directions

On the practical side, the result enables low-latency, responsive LLM deployments with native support for streaming input and output, benefiting high-throughput and interactive applications. The elimination of the $\theta(s)$9 worst-case makes production systems robust to adversarial or pathological input, removing a significant attack vector for denial-of-service.

On a systems level, the incremental BPE algorithm, by reducing the BPE computational footprint, shifts the design focus to the efficiency of normalization and pre-tokenization stages. This can encourage re-engineering upstream pipeline components for full streaming compatibility, ultimately reducing overall I/O and inference latency.

Theoretically, the work demonstrates an important intersection of formal properties and data structure design, opening possibilities for similar algorithmic upgrades to other tokenization and pre-processing schemes (e.g., Unigram, SentencePiece variants), or even to subword algorithms in multimodal models.

Conclusion

"Incremental BPE Tokenization" (2605.30813) provides a comprehensive solution to the efficiency, streaming, and worst-case complexity limitations of BPE as deployed in large-scale NLP systems. By solidifying the structural underpinnings of prefix-consistent tokenization and introducing a $s$0 incremental algorithm, the authors show that algorithmic rigor can yield practical, measurable benefits across both throughput and reliability axes. This paves the way for more responsive, robust, and energy-efficient LLM pipelines and invites reconsideration of current approaches to pipeline segmentation and streaming tokenization.