---
title: Parity-Aware BPE for Fair Multilingual Tokenization
url: https://www.emergentmind.com/topics/parity-aware-bpe
type: topic
---

# Parity-Aware BPE for Fair Multilingual Tokenization

Parity-aware Byte-Pair Encoding (PaBPE) is a modification of the classical Byte-Pair Encoding (BPE) algorithm designed to enhance cross-lingual fairness in tokenization workflows, particularly in multilingual natural language processing tasks. By iteratively maximizing the compression gain for the currently worst-compressed language during subword vocabulary construction, PaBPE mitigates the disparities in tokenization efficiency that arise when frequency-based tokenization strategies favor high-resource languages. PaBPE yields a more equitable distribution of tokenization costs, improved fairness metrics, and maintains negligible impact on overall compression rates and downstream model performance [2508.04796].

## 1. Byte-Pair Encoding: Classical Foundations

Byte-Pair Encoding (BPE) is a greedy, frequency-driven algorithm for learning a fixed-size subword vocabulary that optimizes corpus-level compression. Given corpus $D$, initial alphabet $B$ ($|B| = 256$, e.g., bytes), and $K$ target merges, the algorithm starts with $V_0 = B$ and maintains a tokenizer $T_k$ defined by merges $m_1, \ldots, m_{k-1}$. The average compression rate for $D$ under $T_k$ is:

\[
CR(D;T_k) = \frac{1}{|D|} \sum_{x \in D} \frac{|x|_u}{|T_k(x)|}
\]

where $|x|_u$ is the length in bytes, chars, or another unit. Classical BPE greedily chooses at each step the pair $(v,v')$ with the largest bigram count in $D$ and merges it, maximizing immediate corpus-level compression gain:

\[
\mathbf{m}^* = \arg\max_{m_1, \dots, m_K} CR(D;T_K)
\]

The resulting subword vocabulary and merge history enable tokenization that efficiently compresses text but can induce significant inequity across languages in multilingual corpora.

## 2. Motivation and Cross-Lingual Fairness Challenges

Typical frequency-based BPE objectives heavily privilege languages contributing the most tokens, disadvantaging low-resource languages in several ways:

- **Increased tokenization length**: Higher fertility (tokens per word) for low-resource languages.
- **Morphological misalignment**: Fragmented or implausible subwords in minority languages.
- **Placeholder proliferation**: Elevated rates of <UNK> assignment due to poor segmentation.
- **Operational costs**: Users of low-resource languages pay more per API token due to less efficient tokenization.

To address this, PaBPE introduces an explicit fairness objective, aiming for per-language compression rates $CR(D_l; T)$ to be as equal as possible, thereby lowering the computational "tax" on under-resourced languages.

## 3. Mathematical Formulation of Parity-aware Objectives

### 3.1. Global and Per-Language Compression Gain

The global compression gain for a merge $m = (v, v')$ on corpus $D$ is:

\[
G_{\mathrm{global}}(m) = CR(D;T \oplus m) - CR(D;T)
\]

Where $T\oplus m$ is the tokenizer after applying merge $m$. 

### 3.2. Identifying the Worst-Compressed Language

Partitioning the corpus by languages $L = \{ l_1, ..., l_R \}$, the worst-off language at step $k$ is:

\[
l^* = \arg\min_{l \in L} CR(D_l ; T_{<k})
\]

### 3.3. Parity-aware Min–Max Objective

The ideal (but intractable) objective would be:

\[
\max_{m_1, \dots, m_K} \min_{l \in L} CR(D_l; T_K)
\]

PaBPE greedily approximates this by, at each step, focusing solely on the worst-compressed language $l^*$. The selected merge is:

\[
m_k = \arg\max_{(v,v')} |\{ \text{occurrences of } v v' \text{ in } D_{l^*} \}|
\]

Only bigram statistics from $D_{l^*}$ inform the merge, and the same merge is applied corpus-wide, thus incrementally improving fairness.

## 4. Algorithmic Procedures and Pseudocode

The distinction between classical and Parity-aware BPE centers on bigram counting and selection scope:

**Classical BPE**

- Counts bigrams over the entire corpus and chooses the most frequent pair.

**Parity-aware BPE**

- For each merge:
  - Computes per-language compression rates on dev data.
  - Identifies $l^*$, the language with lowest compression rate.
  - Counts bigrams only in $D_{l^*}$.
  - Selects the most frequent bigram in $D_{l^*}$.
  - Applies the chosen merge to all languages.

The pseudocode structure is as follows:

| Step                    | Classical BPE                    | Parity-aware BPE                       |
|-------------------------|----------------------------------|----------------------------------------|
| Bigram Counting         | All of $D$                       | $D_{l^*}$ only                         |
| Merge Selection         | Largest corpus-wide bigram count | Largest bigram count in $D_{l^*}$      |
| Merge Application       | All of $D$                       | All of $D$                             |

PaBPE avoids aggregation across all languages during merge selection, a key difference that drives parity in compression rates.

## 5. Runtime, Compression, and Empirical Trade-Offs

- **Computational Overhead:** At each merge, only an additional $O(|L|)$ pass over dev data is required to compute $CR_l$ values. Overall asymptotic complexity remains unchanged versus classical BPE.
- **Global Compression Loss:** In a 30-language, 128k-vocabulary BrickRed setup, classical BPE achieves $CR = 0.0303$ tokens/char; PaBPE slightly decreases global compression to $CR \approx 0.0300$, corresponding to a 1% relative drop.
- **Downstream Model Performance:** Across 13 multilingual benchmarks, PaBPE and hybrid variants match or exceed classical performance, with a median per-language accuracy change of $+0.19$ percentage points ($\pm 1$ standard error), and no significant performance drops observed. Empirical perplexity measures are as uniform and mean-matched as classical BPE.

## 6. Fairness Metrics and Experimental Results

Intrinsic parity measures applied on the parallel development set include:

- **Per-language Compression Rate ($CR_l$)**
- **Fertility:** Tokens per word
- **Vocabulary Utilization:** Fraction of vocabulary used per language
- **MorphScore:** Morpheme alignment precision/recall
- **Fairness Gini Coefficient:**

\[
\mathrm{Gini} = \frac{1}{R} \left(R + 1 - 2 \frac{\sum_{i=1}^R (R+1-i) c_i }{\sum_i c_i}\right)
\]

where $c_i$ are sorted per-language average tokens/line.

**Key Results (BrickRed, 30 languages, 128k merges):**

| Tokenizer         | Compression Rate | Gini  | Fertility |
|-------------------|-----------------|-------|-----------|
| Classical         | 0.0303          | 0.064 | 4.260     |
| Parity-aware      | 0.0300          | 0.011 | 4.204     |
| Hybrid            | 0.0303          | 0.018 | 4.191     |
| Window+Hybrid     | 0.0305          | 0.022 | 4.203     |

Parity-aware methods substantially reduce the Gini coefficient, indicating decreased disparity in per-language tokenization cost, with only minimal reduction in global compression and fertility. Vocabulary utilization analyses (Fig. 2) show reduced gaps for low/medium-resource languages without harming high-resource ones. Downstream results further confirm parity with classical baselines.

## 7. Broader Implications and Prospects for Extension

PaBPE represents a “drop-in” modification to classical BPE, imparting:

- Significant reduction in cross-lingual compression gaps (Gini from 0.064 to 0.011).
- Less than 1% reduction in global compression rates.
- Negligible computational overhead per merge ($O(|L|)$).
- No negative effect on downstream language model performance.

Potential broader impacts include reducing the hidden “token tax” on under-resourced languages, facilitating equitable billing, and improving deployment fairness. Extensions to related subword tokenization families (WordPiece, UnigramLM) and other modalities (speech, vision) are viable directions. Further research could explore additional fairness objectives, such as semantic parity in representation. 

Code for PaBPE is publicly available at [https://github.com/swiss-ai/parity-aware-bpe] [2508.04796].

Source: https://www.emergentmind.com/topics/parity-aware-bpe