Tokenization and Compression

Updated 4 June 2026

Tokenization and compression are intertwined principles that convert raw data into discrete symbols while minimizing redundancy.
They balance sequence length, vocabulary size, and model capacity through information-theoretic objectives and practical scaling laws.
Modern approaches, including BPE, greedy algorithms, and adaptive tokenization, optimize performance across language, vision, and multimodal systems.

Tokenization and compression are foundational, intertwined principles that govern the efficiency and effectiveness of modern machine learning systems across language, vision, and multimodal domains. Tokenization refers to the transformation of a raw data stream—text, image, audio, video, or mesh—into a discrete sequence of symbols, while compression quantifies how much redundancy, entropy, or information is preserved or removed in this process. The design of tokenization schemes is driven by rigorous information-theoretic objectives, empirical scaling laws, and intricate trade-offs between sequence length, vocabulary size, model capacity, and downstream task performance. This article synthesizes the key theoretical results, methodologies, and empirical findings from recent research on arXiv, emphasizing the centrality of compression in modern tokenization and highlighting modality-specific innovations.

1. Theoretical Foundations: Compression and Tokenization

Shannon’s source coding theorem (Delétang et al., 2023) establishes that any predictive model can be transformed into a lossless compressor and vice versa. For a true data distribution $P$ and a coding scheme $c$ , the expected code length $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ is bounded by the entropy $H(P)\leq L(c)<H(P)+1$ , with $H(P)=\mathbb{E}_{x\sim P}[-\log_2 P(x)]$ . In practical language modeling, if a learned model $Q$ is used to drive an entropy coder, the per-sequence code length converges to $\ell(x_{1:n})\approx-\sum_{i=1}^n\log_2 Q(x_i|x_{<i})$ , corresponding to the cross-entropy $H(P,Q)$ . Thus, minimizing the language modeling loss directly minimizes the expected number of bits required to encode data.

Tokenization acts as "pre-compression"—a deterministic map from raw strings $\Sigma^*$ to discrete sequences over a vocabulary of size $|V|$ —inducing a coding scheme whose effectiveness is governed by both the distributional properties of the data and the structure of the tokenizer (Erdogan et al., 14 Jan 2026). Notably, the choice of tokenization granularity (bytes, subwords, words, phrases, or variable-length units) critically impacts the compression ratio, information redundancy, and model context efficiency.

2. Tokenization Schemes: Algorithms and Formal Objectives

Tokenization strategies are systematically grounded in compression theory, with each major approach embodying specific optimization objectives and algorithmic constraints.

Byte-Pair Encoding (BPE), UnigramLM, and WordPiece define greedy or probabilistic merge schemes to minimize sequence length for a fixed vocabulary. BPE's objective is to iteratively find pairs of symbols to merge, reducing the number of tokens while balancing vocabulary growth (Schmidt et al., 2024).
PathPiece seeks for each document a segmentation into the minimum possible number of tokens under a given vocabulary and implements a shortest-path dynamic program for segmentation, then greedily prunes the vocabulary (Schmidt et al., 2024).
GreedTok formulates tokenization as an NP-hard partition cover problem: for a vocabulary budget $c$ 0, select $c$ 1 non-base tokens maximizing coverage of symbol adjacencies, reducible to weighted maximum coverage, and solvable by a greedy algorithm with (1–1/e)-approximation guarantee (Lim et al., 8 Jan 2025).
Split Trees (ToaST) constructs for each input pretoken a full binary split tree using corpus statistics. The inference procedure emits the first in-vocabulary substring found along each root-to-leaf path. The global vocabulary is chosen via integer programming, directly minimizing the aggregate token count under this procedure (Schmidt et al., 21 May 2026).
LZW-derived schemes such as MultiTok and zip2zip dynamically build multi-word (or multi-subword) tokens based on the recurrence of phrase patterns in the training data, achieving variable-length tokenization and adaptive compression (Elias et al., 2024, Geng et al., 1 Jun 2025).

These algorithms support both fixed- and variable-length outputs, can be extended for adaptive/multimodal settings, and enable direct theoretical connections to universal source coding and channel capacity.

3. Compression, Scaling Laws, and Empirical Trade-Offs

Compression rates—the effective ratio of raw bytes to tokens ( $c$ 2)—are central scaling parameters in training and inference. Scaling law analyses (Limisiewicz et al., 2 May 2026) reveal several key findings:

In compute-optimal regimes, the number of model parameters $c$ 3 scales linearly with total training data in bytes $c$ 4, not in tokens, for a given total compute $c$ 5: $c$ 6.
There exists a compute-optimal compression rate $c$ 7, typically lower than that of conventional BPE, which decreases with increasing compute; for English at $c$ 8 FLOPs, $c$ 9 bytes/token, compared to BPE's $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 0 (Limisiewicz et al., 2 May 2026).
Bytes-per-token and bytes-per-parameter ratios remain near constant ( $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 1 bytes/param for English) in these optimal regimes.
Over-compression (higher $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 2) reduces context length and compute but eventually degrades bits-per-byte and downstream loss due to increased per-token entropy; under-compression inflates sequence lengths and compute with diminishing returns (Delétang et al., 2023, Limisiewicz et al., 2 May 2026).

For vision and multimodal transformers, similar compression trade-offs are observed: reducing visual tokens (e.g., via pooling or "patchification") yields nearly linear improvements in throughput and FLOPs up to a threshold, beyond which performance sharply degrades (Wang et al., 11 Mar 2026, Weng et al., 2024, Zeng et al., 6 Jun 2025).

4. Information Structure, Induced Biases, and Robustness

Tokenization not only compresses but also shapes the statistical dependencies that downstream models must learn (Erdogan et al., 14 Jan 2026):

As tokenizer training scale increases, unigram entropy $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 3 (diversity) rises and higher-order conditional entropies $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 4 (local unpredictability) fall: more marginal diversity, less local redundancy. This manifests in token streams that are more uniformly distributed and easier to model in context.
Compression-aware metrics, such as Shannon capacity utilization $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 5 and Rènyi efficiency, track the "fill" of the vocabulary and the presence of overly frequent tokens, guiding the avoidance of codebook degeneracy (Erdogan et al., 14 Jan 2026).
Domain mismatch (e.g., applying an English-trained tokenizer to code or Turkish) negates these entropy gains and causes both compression ratios and predictability to degrade. Thus, robust tokenization demands domain- or script-aware vocabulary construction (Erdogan et al., 14 Jan 2026).
Overly aggressive compression through variable-length or hierarchical tokenization must be balanced to avoid excessive long-range dependencies, which can render autoregressive modeling intractable (Geng et al., 1 Jun 2025).

Empirically, best practices include monitoring both marginal and higher-order entropies, matching training and target application domains, and moderating vocabulary size to avoid rare or fragmented tokens (Erdogan et al., 14 Jan 2026, Lim et al., 8 Jan 2025).

5. Modalities Beyond Text: Vision, Video, and Mesh Compression

Modern tokenization extends far beyond text, with bespoke compression schemes for each modality:

Vision: UniCompress achieves up to $L(c)=\mathbb{E}_{x\sim P}[\ell_c(x)]$ 6 reduction in visual tokens for unified vision-language transformers by extracting global meta-tokens and aggressively average-pooling local visual features, preserving performance with minimal downstream retraining (Wang et al., 11 Mar 2026). One-D-Piece enables quality-controllable, variable-rate discrete tokenization for images by concentrating information in head tokens via a stochastic tail-drop regularizer (Miwa et al., 17 Jan 2025).
Video: TVC proposes a dual-stream tokenization pipeline—discrete "semantic" tokens (quantized via a codebook) and continuous "fidelity" tokens—masking and entropy coding only a fraction of the former. Masked tokens are then reconstructed via a transformer decoder, fused with the continuous stream, achieving strong LPIPS and PSNR at ultra-low bitrates (Zhou et al., 22 Apr 2025).
Mesh: Blocked and Patchified Tokenization (BPT) compresses mesh sequence length by 75% via block-wise indexing and patch aggregation, allowing training of memory-efficient transformers on large (8k+) meshes without loss of geometric fidelity (Weng et al., 2024).

Adaptive schemes such as InfoTok leverage the evidence lower bound (ELBO) to dynamically allocate token budgets to spatial-temporal regions of highest informational richness, approaching theoretical lower bounds on average code length and outperforming heuristic adaptive baselines (Ye et al., 18 Dec 2025).

6. Compression and Downstream Model Performance

A persistent question is whether greater compression—the minimization of token sequence length—predicts or causes superior downstream task accuracy. Comprehensive experimental campaigns (Goldman et al., 2024, Schmidt et al., 2024) indicate:

There is a significant, reliable negative correlation between a tokenizer’s held-out compression ratio (fewer tokens per document) and model accuracy, especially for generation tasks and smaller models (Goldman et al., 2024).
However, minimizing token count alone does not always yield the best performance. Tokenizers such as PathPiece, which achieve minimum-token segmentations, do not outperform BPE or UnigramLM (statistically indistinguishable for 350M–2.4B parameter LMs), and sometimes linguistically oriented pre-tokenization or BPE-based vocabulary initialization yields higher accuracy for a fixed vocabulary size (Schmidt et al., 2024).
Linguistically aware pre-tokenization (e.g., treating spaces and digits as boundaries) improves performance, even as it increases token count above compression-optimal schemes (Schmidt et al., 2024).
Metrics such as compression-aware bits-per-byte and coverage objectives (partition cover, maximum coverage) are predictive but not sufficient; morphological alignment and consistent segmentation strategies remain critical.

7. Advances in Adaptive, Variable-Length, and End-to-End Tokenization

Contemporary research explores breaking the fixed-token paradigm using adaptive, variable-length, and fully learnable tokenization strategies:

Dynamic tokenization: zip2zip incorporates an LZW-based compressor at inference, creating hypertokens on the fly and adaptively augmenting the vocabulary and embedding/projection heads, reducing token sequence length by 20–60% with ≤1 FIDpoint performance cost and substantial throughput gains (Geng et al., 1 Jun 2025).
RL-based end-to-end learning: Token boundaries can be optimized via a reinforcement learning policy (score-function estimator) tightly coupled to the model’s ultimate next-token loss. Such architectures can surpass straight-through estimators in bits-per-byte and downstream accuracy, aligning boundaries with meaningful semantic units (Dauncey et al., 15 Feb 2026).
Regularized, generation-aware tokenization: Causally Regularized Tokenization (CRT) explicitly encourages the encoder to produce token sequences whose next tokens can be easily predicted, optimizing stage 1 for stage 2 generativity at slight cost to reconstruction but achieving SOTA FID/generation at half the baseline tokens and quarter the compute (Ramanujan et al., 2024).
Multi-granular and split-tree frameworks: ToaST generalizes subword tokenization as recursive binary split trees and optimally selects a vocabulary by linear programming, reducing token count and increasing Rényi efficiency across major LLM tasks (Schmidt et al., 21 May 2026).

In all cases, careful control of the compression–modeling trade-off remains essential. Overly aggressive compression can fragment semantics, while under-compression squanders compute and context efficiency.

In sum, tokenization and compression are not only mathematically dual but, in contemporary practice, inseparable axes along which language, vision, and multimodal systems are engineered for efficiency, capacity, and performance. Modern tokenizers are increasingly information-theoretic, data-driven, and adaptive, each striving to encode the maximal informational content in the minimal symbol stream, without sacrificing the compositional or semantic structures underpinning downstream generalization. Ongoing work at the intersection of optimal compression, linguistically aligned pre-tokenization, and hardware-constrained compute promises continued advances in scalable, robust foundation models across modalities.