---
title: Token Agreement Metric Overview
url: https://www.emergentmind.com/topics/token-agreement-metric
type: topic
---

# Token Agreement Metric Overview

A token agreement metric quantitatively assesses the degree to which two models, subword tokenizations, or language representations produce concordant token-level predictions or segmentations. These metrics serve as core diagnostics in multilingual modeling and large language model (LLM) compression by capturing the statistical similarities, divergences, or alignability at the token or subword level, often providing greater sensitivity and predictive power than classic measures such as token overlap, perplexity, or accuracy. Recent advances have established token agreement-based metrics as central to evaluating and optimizing transferability across disparate scripts, detecting compression-induced degradation, and constructing robust multilingual tokenizers [2502.06468][2311.01544].

## 1. Mathematical Definitions of Token Agreement Metrics

Two foundational approaches to token agreement emerge in current research: subword token alignability for cross-lingual transfer [2502.06468] and divergent token metrics for model compression [2311.01544].

### Subword Token Alignability

Consider a pair of languages $L$ and $L'$, a shared subword tokenizer, and a parallel corpus $C_\text{test} = \{(u_i, v_i)\}$. Using a statistical word aligner (e.g., eflomal), directional alignments $a_{L\to L'}(u,v) \subseteq \{1…|u|\} \times \{1…|v|\}$ are symmetrized. Two key scores are defined:

1. **Directional one-to-one alignability**:
   $$
   \alpha_{L\to L'} = \frac{1}{N_L} \sum_{(u,v)\in C_\text{test}} |\{ i \in [1..|u|] : \exists! j,\ (i,j) \in a_{L\to L'}(u,v) \}|
   $$
   where $N_L$ is the total number of source tokens in all $u$.

2. **Symmetrised eflomal log-probability**:
   $$
   E_{L,L'} = - \frac{1}{2} \left( \frac{\mathcal{L}_{L\to L'}}{|C_\text{test}|} + \frac{\mathcal{L}_{L'\to L}}{|C_\text{test}|} \right)
   $$
   where $\mathcal{L}_{L\to L'}$ is the sum of unnormalized log-probabilities under the learned alignment priors.

Lower values of $E_{L,L'}$ indicate higher alignability.

### Divergent Token Metrics (DTMs)

Given two autoregressive models, $P$ (original) and $Q$ (compressed),

Source: https://www.emergentmind.com/topics/token-agreement-metric