Conditional Unigram Tokenization

• Conditional Unigram Tokenization is a probabilistic model that conditions token probabilities on external variables like language or source sentence while using a fixed shared vocabulary.
• It enables language-specific segmentation and improved morphological alignment by reweighting token distributions without altering the underlying vocabulary.
• Empirical studies show enhanced intrinsic metrics and language identification performance, though downstream gains can vary with application.

Searching arXiv for the cited tokenization papers to ground the article in the current literature. Conditional unigram tokenization is a probabilistic tokenization framework in which the segmentation score of a string is still factorized as a unigram model over tokens, but the unigram parameters are conditioned on an external variable rather than being globally shared. In the clearest current instantiations, the conditioning variable is language: a fixed shared vocabulary is retained, while each language receives its own unigram distribution over that vocabulary, and inference selects the segmentation preferred by the appropriate conditional distribution (Meister et al., 22 Jun 2026). More generally, the same idea extends to bilingual settings, where target-side token probabilities are conditioned on source-side tokens from parallel data (Vico et al., 10 Jul 2025). This distinguishes conditional unigram tokenization from standard UnigramLM, which uses one global token distribution, and from separate language-specific tokenizers, which abandon a shared token inventory (Meister et al., 19 Feb 2026).

1. Conceptual definition and scope

Standard UnigramLM assumes that an input string admits multiple latent segmentations and assigns each segmentation a probability given by a product of token probabilities from a single unigram distribution. Conditional unigram tokenization preserves that latent-segmentation view, but replaces the single parameter vector with condition-specific vectors, so that the segmentation preference depends on a variable such as language while the vocabulary remains fixed (Meister et al., 22 Jun 2026).

In the language-conditioned case, the central design choice is not vocabulary redesign but conditional reweighting. This addresses a multilingual pathology identified explicitly in recent work: even when a multilingual vocabulary already contains suitable subwords for a target language, a globally estimated unigram distribution can still force poor segmentations because the pooled corpus is dominated by other languages or scripts (Meister et al., 22 Jun 2026). The key insight is therefore that multilingual tokenization problems are often not only about vocabulary coverage. They are also about the distribution used to score candidate segmentations.

This usage is narrower than a fully context-sensitive tokenizer. In the papers that most directly instantiate the paradigm, the condition is sequence-level or sentence-level—most notably a language hypothesis or a source sentence—rather than an arbitrary local context feature at each token boundary (Meister et al., 19 Feb 2026). The resulting models are still unigram in the sense that, conditional on the side information, token sequence probability factorizes across token identities rather than token histories.

The term also has a broader interpretive use in the literature. Work on tokenization theory argues that a tokenizer can absorb short-range regularities into token identities, thereby transforming a character process with strong conditional dependence into a token process that is much closer to unigram-like from the downstream model’s perspective (Rajaraman et al., 2024). This does not define a conditional tokenizer in the formal probabilistic sense, but it clarifies why conditionalizing the tokenization mechanism is a natural extension of the UnigramLM family.

2. Formal basis in UnigramLM

Let $\mathbf S=\langle s_1,\dots,s_N\rangle$ be an input string and let $\mathcal V$ be a token vocabulary. In standard UnigramLM, a latent tokenization $\mathbf V=\langle v_1,\dots,v_m\rangle$ is assumed, with each token drawn independently from a unigram distribution $\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$. The token-sequence model is

$$P(\mathbf v) \propto \prod_{t=1}^{|\mathbf v|}\phi[v_t].$$

Given a fixed vocabulary, tokenization is MAP decoding over all segmentations compatible with the string: $$h_{\boldsymbol\phi}(\mathbf s) = \argmax_{\mathbf v\in \mathcal T_{\mathcal V}(\mathbf s)} \prod_{t=1}^{|\mathbf v|}\phi[v_t].$$ Training on a corpus uses EM because segmentation is latent, with expected token counts in the E-step and normalized expected counts in the M-step (Meister et al., 22 Jun 2026).

Conditional unigram tokenization changes exactly one structural component: it replaces $\boldsymbol\phi$ with a family of conditional parameter vectors. In the language-conditioned formulation of LangMAP, one learns

$\{\boldsymbol\phi^\ell\}_{\ell\in\Omega},$

all over the same shared vocabulary $\mathcal V$, and for each language-specific corpus $\mathcal C^\ell$ applies the standard UnigramLM EM procedure independently, except that vocabulary learning and pruning are skipped because the vocabulary is fixed (Meister et al., 22 Jun 2026). This yields the natural conditional analogue

$\mathcal V$0

An analogous formalization appears in UniLID. There, a string $\mathcal V$1 is generated from a latent token sequence $\mathcal V$2 with

$\mathcal V$3

and the standard UnigramLM tokenization is

$\mathcal V$4

Conditionalization introduces a language-specific unigram distribution $\mathcal V$5 over the same vocabulary $\mathcal V$6, while leaving the segmentation lattice and inference machinery unchanged (Meister et al., 19 Feb 2026).

This formal basis is important historically because standard UnigramLM is already a latent-variable segmentation model, unlike BPE’s greedy merge history. That probabilistic structure makes conditionalization comparatively direct: the model can change token probabilities without changing the vocabulary or abandoning tractable dynamic-programming inference (Bostrom et al., 2020).

3. Language-conditioned models over a shared vocabulary

The most explicit realizations of conditional unigram tokenization to date are language-conditioned models with a shared vocabulary. LangMAP is best understood as a language-conditional extension of UnigramLM for multilingual tokenization. It learns one unigram distribution per language over a fixed shared vocabulary, then performs per-language MAP segmentation and selects the highest-likelihood segmentation overall (Meister et al., 22 Jun 2026). UniLID uses the same basic idea for language identification: learn language-conditional unigram distributions over a shared tokenizer vocabulary and recompute the best segmentation under each language’s token probabilities (Meister et al., 19 Feb 2026).

A defining property of this family is that segmentation itself becomes language-specific. The same surface string can admit different preferred tokenizations under different language-conditioned token distributions. In UniLID, the test-time score for a language $\mathcal V$7 is the probability of its own best segmentation,

$\mathcal V$8

and language prediction is

$\mathcal V$9

LangMAP uses an analogous two-stage max-max decoder: compute the best segmentation under each $\mathbf V=\langle v_1,\dots,v_m\rangle$0, then choose the segmentation with highest likelihood across languages (Meister et al., 19 Feb 2026, Meister et al., 22 Jun 2026).

This architecture differs from several nearby alternatives. It is not a jointly trained mixture model with an explicit prior over languages, not a posterior-regularization framework, and not a marginal decoder over all languages and segmentations jointly (Meister et al., 22 Jun 2026). It also differs from maintaining separate tokenizers per language, because the shared vocabulary preserves interoperability with an existing multilingual model and keeps the embedding inventory unchanged.

That shared-vocabulary constraint is central to the adaptation story. LangMAP explicitly accepts any base vocabulary, including one learned by BPE or the vocabulary of a pretrained model, skips vocabulary pruning, and re-estimates only the unigram token probabilities. A pretrained model can therefore be retokenized for a target language without altering the embedding matrix or architecture (Meister et al., 22 Jun 2026). UniLID makes the same modular point from the LID perspective: any tokenizer with a fixed vocabulary can be used, including a pretrained LLM tokenizer (Meister et al., 19 Feb 2026).

The approach does, however, require labeled conditions at training time. LangMAP states directly that language labels are required during training, although empirically only a few thousand examples per language are needed for estimates to stabilize, and held-out fertility nearly converges after $\mathbf V=\langle v_1,\dots,v_m\rangle$1k examples for German, Finnish, and Turkish (Meister et al., 22 Jun 2026). UniLID similarly requires language-labeled strings but no token boundaries or morphology annotations, and reports strong low-resource behavior, including 69.46% accuracy with 5 samples per language and 80.01% with 10 on WiLI subsampling (Meister et al., 19 Feb 2026).

4. Beyond language ID: source-conditioned tokenization from parallel data

A second major branch conditions target-side tokenization on source-side tokens from parallel data. “Conditional Unigram Tokenization with Parallel Data” introduces this explicitly as conditional unigram tokenization in the bilingual setting (Vico et al., 10 Jul 2025). Here the source sentence $\mathbf V=\langle v_1,\dots,v_m\rangle$2 is already tokenized by a fixed source tokenizer, while the target string $\mathbf V=\langle v_1,\dots,v_m\rangle$3 is segmented using target token probabilities conditioned on the source tokens.

The paper defines the target tokenization objective as

$\mathbf V=\langle v_1,\dots,v_m\rangle$4

with the sentence-level conditional token probability approximated through source–target co-occurrence counts: $\mathbf V=\langle v_1,\dots,v_m\rangle$5 It also defines token-to-token alignment probabilities

$\mathbf V=\langle v_1,\dots,v_m\rangle$6

and an unconditional fallback obtained by marginalizing over source tokens (Vico et al., 10 Jul 2025).

The intended effect is cross-lingual semantic alignment: a target tokenizer should not only fit target-side statistics but also favor segmentations that correspond more cleanly to source-side semantic units. Vocabulary pruning is therefore based on mutual information with the source vocabulary,

$\mathbf V=\langle v_1,\dots,v_m\rangle$7

rather than only target-side likelihood (Vico et al., 10 Jul 2025).

Empirically, the results are mixed in a way that has become central to the topic. On machine translation, the baseline SentencePiece unigram tokenizer generally outperforms the conditional model, sometimes substantially. On language modeling, however, the conditional tokenizer yields consistent perplexity-per-byte improvements across all evaluated language pairs and vocabulary sizes (Vico et al., 10 Jul 2025). The paper’s main explanation is a quadratic scaling bottleneck: standard unigram estimates $\mathbf V=\langle v_1,\dots,v_m\rangle$8 parameters, whereas the conditional count table scales as $\mathbf V=\langle v_1,\dots,v_m\rangle$9, creating a data-efficiency problem.

This bilingual line of work demonstrates that conditional unigram tokenization is not restricted to language identity. It can be formulated with richer side information, but the count-table parameterization used so far appears to trade conceptual clarity for poor sample efficiency (Vico et al., 10 Jul 2025).

5. Relation to boundary conditioning, multilingual allocation, and tokenization theory

Conditional unigram tokenization is related to, but not identical with, several other proposals about how tokenizers should encode context. One influential example is the alternative treatment of spaces in Unigram and BPE. “Improving Tokenisation by Alternative Treatment of Spaces” does not condition token probabilities on context, but removes a boundary-conditioned artifact of standard subword tokenization by always treating spaces as individual tokens (Gow-Smith et al., 2022). In standard SentencePiece-style tokenization, equivalent strings are treated differently depending on whether they occur at the beginning of a word. The modified Unigram$\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$0 keeps lexical tokens boundary-invariant and represents the boundary separately. This is relevant because it shows that some apparent context sensitivity in tokenization is an artifact of vocabulary design rather than an explicit conditional model.

From a broader theoretical perspective, tokenizers can be analyzed as transforms that redistribute entropy. “An Information-Theoretic Perspective on LLM Tokenizers” reports that as tokenizer training scale grows, the token stream exhibits higher unigram entropy but lower higher-order conditional entropies, indicating that tokenization absorbs substantial short-range regularity (Erdogan et al., 14 Jan 2026). This supports a central intuition behind conditional unigram approaches: tokenization quality cannot be understood solely through vocabulary coverage or marginal token frequencies; it must also be understood through the conditional structure induced in the emitted token sequence.

A related theoretical argument appears in “Toward a Theory of Tokenization in LLMs.” On simple Markov sources, transformers trained without tokenization can collapse to a character-level unigram predictor, whereas with appropriate tokenization, even a unigram model over tokens can achieve near-optimal end-to-end cross-entropy (Rajaraman et al., 2024). The tokenizer is not formally conditional there, but it packages conditional dependencies into token identities. This suggests why a conditional extension of UnigramLM is natural: the tokenizer can be designed to internalize dependencies that would otherwise have to be modeled downstream.

The multilingual literature also provides an indirect motivation. Work on Indian languages shows that UnigramLM is generally preferable to BPE for multilingual Indic tokenization at small-to-medium vocabulary sizes and for morphology alignment, while joint multilingual vocabularies can under-serve some languages and cluster-based vocabulary construction often helps underrepresented ones (Karthika et al., 21 Jun 2025). LangMAP makes this point sharper: even with a fixed vocabulary, globally pooled unigram parameters can favor dominant languages and scripts (Meister et al., 22 Jun 2026). A plausible implication is that conditional unigram tokenization can be understood as a way of restoring language-specific token preferences without abandoning the efficiencies of vocabulary sharing.

6. Empirical performance, misconceptions, and open problems

The strongest empirical support for conditional unigram tokenization is intrinsic rather than uniformly downstream. LangMAP improves morphological boundary alignment across 11 distinct tokenizers and 9 natural languages, with pooled MorphScore changes of $\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$1 recall, $\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$2 precision, $\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$3 micro-F1, and $\boldsymbol\phi\in\Delta^{|\mathcal V|-1}$4 macro-F1, and it improves AST-leaf-boundary alignment across all 9 programming languages tested (Meister et al., 22 Jun 2026). UniLID shows that language-specific segmentation can be operationally useful: on DSL-ML 2024 it improves macro F1 from 0.532 for fastText to 0.723, and on Tatoeba it more than doubles fastText’s macro F1, 0.414 versus 0.160 (Meister et al., 19 Feb 2026).

Downstream task evidence is more mixed and has become a recurring point of caution. LangMAP improves MultiBLiMP grammatical acceptability in 5 of 6 language-size settings after fine-tuning Qwen2.5, but its benefits are less consistent on Global-PIQA and Belebele (Meister et al., 22 Jun 2026). The bilingual conditional tokenizer shows consistent perplexity-per-byte gains in language modeling but no machine-translation improvement (Vico et al., 10 Jul 2025). This pattern reinforces a common misconception: better token alignment or more linguistically plausible segmentation does not automatically translate into better downstream performance.

A second misconception is that tokenizer quality can be reduced to a single unigram-distribution metric. “Two Counterexamples to Tokenization and the Noiseless Channel” shows that Rényi efficiency of the unigram distribution can be increased while downstream performance worsens, by decomposing useful subwords or duplicating high-frequency tokens into redundant IDs (Cognetta et al., 2024). This matters directly for conditional unigram tokenization because conditional models are also built on unigram statistics; the paper demonstrates that marginal unigram balance alone is insufficient to evaluate a tokenizer.

A third misconception is that morphology by itself explains Unigram’s success. In Telugu, Unigram-based tokenizers consistently outperform BPE across most settings, and better morphological alignment shows a moderate positive correlation with downstream performance, especially for structure prediction, but the tokenizer algorithm remains the strongest factor (Vemula et al., 11 Aug 2025). This suggests that conditional extensions based solely on morphology-aware conditioning should not be assumed to subsume the broader algorithmic advantages of UnigramLM.

Several practical constraints remain unresolved. Language-conditioned models require labeled conditions at training time (Meister et al., 22 Jun 2026). Bilingual conditional models suffer from quadratic parameter growth in the source–target count table (Vico et al., 10 Jul 2025). Multilingual tokenization work on Indic languages indicates that shared vocabularies remain sensitive to corpus composition, script normalization, and vocabulary allocation strategy (Karthika et al., 21 Jun 2025). Implementation studies of standard Unigram further show that while the core model is elegant, practical training pipelines involve nontrivial pruning and initialization choices, even before any conditioning is introduced (Land et al., 14 Dec 2025).

Taken together, the current literature supports a concise characterization. Conditional unigram tokenization is best viewed as a family of probabilistic tokenizers that retain UnigramLM’s latent-segmentation and dynamic-programming structure while making token probabilities depend on an external variable such as language or source sentence. Its principal advantage is that it can alter segmentation behavior without changing the vocabulary. Its strongest evidence lies in improved intrinsic structure alignment, language identification, and some language-modeling settings. Its chief unresolved problem is how to introduce richer conditioning without sacrificing the data efficiency and tractability that make the unigram framework attractive in the first place (Meister et al., 22 Jun 2026, Vico et al., 10 Jul 2025).