Parity-aware Byte-Pair Encoding (PaBPE)

• Parity-aware BPE is a modified byte-pair encoding algorithm that tackles cross-lingual tokenization disparities by prioritizing merges for low-resource languages.
• It computes per-language compression rates and selects token merges based on the worst-compressed language, ensuring a fairer token distribution.
• Empirical results show that PaBPE dramatically lowers the token-cost Gini coefficient with minimal overall compression loss and stable downstream performance.

Parity-aware Byte-Pair Encoding (PaBPE) is a variant of the classical Byte-Pair Encoding (BPE) algorithm designed specifically to address cross-lingual disparities in tokenization. Unlike frequency-based merge strategies that dominate standard BPE and bias token vocabularies toward high-resource languages, PaBPE introduces a targeted criterion at each merge step to prioritize compression gain for the currently worst-compressed language. The result is a significant reduction in inequality of token counts across languages, with minimal loss in overall compression rate and negligible effect on downstream LLM performance (Foroutan et al., 6 Aug 2025).

1. Motivation: Cross-Lingual Disparities in Tokenization

Standard subword tokenizers, most notably BPE, greedily merge the most frequent adjacent symbol pairs across the entirety of the multilingual training corpus. This approach optimizes a global frequency objective, unintentionally favoring high-resource languages whose subword pairs disproportionately occupy the most frequent slots. Consequently, the resulting tokenization scheme provides:

• Shorter, more computationally efficient sequences for high-resource languages, lowering both computational and financial costs.
• Longer, potentially morphologically implausible token sequences for low-resource languages, increasing both token cost and the occurrence of <UNK> placeholders, and degrading downstream performance.
• Amplification of systemic inequities, since metrics such as compute time and financial charges are frequently measured by token count, thereby disadvantaging speakers of under-represented languages.

PaBPE is formulated to rectify this by shifting the focus from overall global compression to a form of min–max parity, where merges are chosen to benefit the worst-off language at any given step.

2. Formal Algorithmic Framework

PaBPE substantially revises the merge selection criterion central to classical BPE. The algorithm operates over a set of $R$ languages $L = \{\ell_1,\,\ldots,\,\ell_R\}$, with training corpora $D_\ell$ and parallel development sets $Dev_\ell$ per language.

For each step $k$ in the merge sequence:

1. Compression Measurement: For every $\ell \in L$, the current compression rate is computed as $$CR_\ell(T_{<k}) = (\sum_{x\in Dev_\ell} |x|_u) / C_\ell(T_{<k})$$, where $|x|_u$ is the number of unit tokens (e.g., bytes) and $C_\ell(T_{<k}) = \sum_{x\in Dev_\ell} |T(x)|$ is the total code length under the partial tokenizer $T_{<k}$.
2. Worst-Compressed Language Identification: Compute $L = \{\ell_1,\,\ldots,\,\ell_R\}$0.
3. Pair Selection: In $L = \{\ell_1,\,\ldots,\,\ell_R\}$1, find the most frequent adjacent token pair $L = \{\ell_1,\,\ldots,\,\ell_R\}$2 maximizing $L = \{\ell_1,\,\ldots,\,\ell_R\}$3.
4. Global Merge Application: Apply the selected merge $L = \{\ell_1,\,\ldots,\,\ell_R\}$4 to all corpora $L = \{\ell_1,\,\ldots,\,\ell_R\}$5 and $L = \{\ell_1,\,\ldots,\,\ell_R\}$6, updating $L = \{\ell_1,\,\ldots,\,\ell_R\}$7.

This min–max ("fair-max") objective can be formalized as:

$L = \{\ell_1,\,\ldots,\,\ell_R\}$8

where $L = \{\ell_1,\,\ldots,\,\ell_R\}$9 is the token savings in $D_\ell$0 from applying the merge.

Variants and Hyperparameters

• Hybrid PaBPE: The initial $D_\ell$1 merges follow classical BPE; the subsequent $D_\ell$2 merges implement the PaBPE criterion, controlling the balance between global compression and fairness.
• Moving-Window Balancing: Maintains a window of the $D_\ell$3 most recent merges; restricts the selection of $D_\ell$4 if it appears more than $D_\ell$5 times, guarding against starvation and premature convergence for any single language.

Pseudocode

$k$0

3. Mechanistic Analysis and Theoretical Motivation

The classical BPE update maximizes total global compression $D_\ell$6, leading to a merge allocation biased toward languages with the greatest data mass. PaBPE, by maximizing $D_\ell$7, operates as a "worst-case" optimization procedure. This approach:

• Ensures that, at each step, compression improvement is directed at the language experiencing the highest tokenization burden.
• Analogous to "watering the driest plant," the strategy systematically reduces disparities in compression rates.
• Over iterative merges, this dramatically lowers both the variance and Gini coefficient of compression rates across languages, leading to fairer, more uniform token distributions.

4. Experimental Protocol and Metrication

Data and Resource Tiers

• Training corpus: Multilingual C4 (mC4) for frequency estimation.
• Development corpus: Dev portion of FLORES+ (used exclusively for cross-lingual compression measurement).
• Language sets: Evaluation performed over 30-language ("BrickRed", reflecting observed imbalance in mC4) and 60-language subsets (including "OliveGreen", with uniform per-language data).
• Vocabulary budgets: Results reported for both 128 k and 256 k merge settings.

Intrinsic and Extrinsic Metrics

Intrinsic metrics (measured on FLORES+ devtest):

• Fertility (tokens per word)
• Compression Rate ($D_\ell$8) (tokens per document)
• Vocabulary Utilization (fraction of $D_\ell$9 used)
• Type-Token Ratio, Average Token Rank
• Rényi entropy $Dev_\ell$0 of the unigram distribution ($Dev_\ell$1)
• MorphScore (alignment of morpheme and token boundaries)
• Tokenizer Fairness Gini Coefficient (per-language token costs)

Extrinsic metrics (downstream evaluation):

• Model: 3 B-parameter decoder-only Transformer (LLaMA style architecture)
• Pretraining: 100 B tokens from FineWeb2, temperature sampling $Dev_\ell$2
• Benchmarks: 13 multilingual tasks (XNLI, PAWS-X, XCSQA, Belebele, mTruthfulQA, MMMLU, etc.), with accuracy and perplexity aggregated per language

5. Empirical Results and Parity Outcomes

Intrinsic Fairness and Compression

Tokenizer	Compression Rate	Gini Coefficient
Classical BPE	0.0303 ± 0.0001	0.064
Parity-aware BPE	0.0300 ± 0.0001	0.011

• PaBPE reduces the Gini coefficient of token-cost from 0.064 to 0.011, signifying a substantial decrease in cross-lingual disparity with a less than 1% decrease in global compression.
• Vocabulary utilization and MorphScore metrics for PaBPE are comparable to those of classical BPE.
• Vocabulary utilization is most improved for low-resource languages.
• The distribution of per-language compression rates is far narrower under PaBPE.

Downstream Performance

Language	Classical BPE	Hybrid PaBPE
English	43.04 ± 1.84	44.15 ± 1.85
Bengali	24.95 ± 3.09	23.54 ± 2.98
...	...	...

• Hybrid PaBPE exhibits a median per-language accuracy change of +0.19 percentage points across 13 benchmarks (14 languages improve, 6 worsen), with no systematic degradation.
• Per-language perplexities under PaBPE become more uniform, eliminating the extended tail of poor tokenization outcomes observed for low-resource languages in classical BPE.

6. Implementation, Data, and Complexity Considerations

• Computational Overhead: Implements an $Dev_\ell$3 pass per merge to recompute $Dev_\ell$4 on $Dev_\ell$5, but retains the same asymptotic complexity as classical BPE.
• Development Data: Requires only a modest parallel dev set for compression rate measurement. Main training data requires merely language label annotation.
• Hyperparameters: Choice of hybrid threshold $Dev_\ell$6 and parity-phase $Dev_\ell$7 tunes the trade-off between overall compression and fairness. Window length $Dev_\ell$8 and multiplier $Dev_\ell$9 effectively prevent overfitting merge decisions to one language.

7. Conclusions and Significance

Parity-aware BPE functions as a drop-in replacement for tokenization procedures in multilingual NLP pipelines. By revising the merge selection criterion to focus on the language with the highest current tokenization burden, it reduces cross-lingual disparities—shrinking the token-cost Gini coefficient from 0.064 to 0.011—with an under 1% reduction in total compression rate and without negative impact on downstream task performance. This directly addresses the hidden “token tax” imposed on low-resource language users by standard tokenization pipelines, advancing computational and financial equity in multilingual NLP settings (Foroutan et al., 6 Aug 2025).