Token Agreement Metric Overview

• Token Agreement Metrics are quantitative measures that assess concordance in token-level outputs between language models or segmentation systems, underpinning applications in multilingual transfer and LLM compression.
• Subword token alignability leverages directional alignments and symmetrization to sensitively evaluate statistical similarities across languages using shared tokenizers.
• Divergent Token Metrics compare original and compressed models by analyzing prediction divergences through statistical log-probability measures, highlighting compression-induced degradation.

A token agreement metric quantitatively assesses the degree to which two models, subword tokenizations, or language representations produce concordant token-level predictions or segmentations. These metrics serve as core diagnostics in multilingual modeling and LLM compression by capturing the statistical similarities, divergences, or alignability at the token or subword level, often providing greater sensitivity and predictive power than classic measures such as token overlap, perplexity, or accuracy. Recent advances have established token agreement-based metrics as central to evaluating and optimizing transferability across disparate scripts, detecting compression-induced degradation, and constructing robust multilingual tokenizers (Hämmerl et al., 10 Feb 2025, Deiseroth et al., 2023).

1. Mathematical Definitions of Token Agreement Metrics

Two foundational approaches to token agreement emerge in current research: subword token alignability for cross-lingual transfer (Hämmerl et al., 10 Feb 2025) and divergent token metrics for model compression (Deiseroth et al., 2023).

Subword Token Alignability

Consider a pair of languages $L$ and $L'$, a shared subword tokenizer, and a parallel corpus $C_\text{test} = \{(u_i, v_i)\}$. Using a statistical word aligner (e.g., eflomal), directional alignments $$a_{L\to L'}(u,v) \subseteq \{1…|u|\} \times \{1…|v|\}$$ are symmetrized. Two key scores are defined:

1. Directional one-to-one alignability:

$$\alpha_{L\to L'} = \frac{1}{N_L} \sum_{(u,v)\in C_\text{test}} |\{ i \in [1..|u|] : \exists! j,\ (i,j) \in a_{L\to L'}(u,v) \}|$$

where $N_L$ is the total number of source tokens in all $u$.

2. Symmetrised eflomal log-probability:

$$E_{L,L'} = - \frac{1}{2} \left( \frac{\mathcal{L}_{L\to L'}}{|C_\text{test}|} + \frac{\mathcal{L}_{L'\to L}}{|C_\text{test}|} \right)$$

where $\mathcal{L}_{L\to L'}$ is the sum of unnormalized log-probabilities under the learned alignment priors.

Lower values of $E_{L,L'}$ indicate higher alignability.

Divergent Token Metrics (DTMs)

Given two autoregressive models, $L'$0 (original) and $L'$1 (compressed),