LangMAP: A Language-Adaptive Approach to Tokenization

Abstract: Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new model must be trained from scratch, or the vocabulary of an existing pretrained model must be adapted. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, a tokenization scheme that extends the UnigramLM algorithm to the multilingual setting, producing language-specific tokenization from a single shared vocabulary. Notably, LangMAP can be used when training a multilingual LLM from scratch or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. While language labels are required at training time, a key feature of the algorithm is that it then performs language-specific tokenization at inference without knowledge of the input's language. Across 14 open-source tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and, for all coding languages tested, alignment with abstract syntax tree (AST) leaf boundaries. In fine-tuning experiments, results are mixed: LangMAP improves target-language grammatical acceptability (MultiBLiMP) on the languages tested; its benefits are less consistent on knowledge-related tasks (Global-PIQA, Belebele).

• The paper introduces a language-adaptive tokenization method that reweights a shared vocabulary using language-specific unigram distributions to improve segmentation quality.
• The approach leverages MAP inference to select the optimal segmentation, yielding significant gains in morphological alignment and syntactic structure on natural and programming languages.
• Evaluation shows enhanced downstream performance in morphosyntactic tasks, with benefits dependent on adequate vocabulary coverage for non-Latin scripts.

LangMAP: A Language-Adaptive Approach to Tokenization

Motivation and Problem Statement

Tokenization constitutes a fundamental preprocessing step in NLP pipelines, mapping character sequences to token sequences over a fixed vocabulary, typically optimized for English. Existing multilingual tokenizers are frequently suboptimal for morphologically rich or non-Latin-script languages, leading to over-segmentation, inflated token counts, semantic opacity, and compromised downstream performance [petrov-etal-2023-tokenizers, lesci-etal-2025-causal]. Language-specific tokenizers improve segmentation quality, but their adoption incurs substantial computational and compatibility costs, including retraining models from scratch or adapting embedding matrices for new vocabularies [rust-etal-2021-how, feher-etal-2025-retrofitting]. Critically, many tokenizers already contain the requisite vocabulary pieces, but global parameterizations, biased by English statistics, induce suboptimal decompositions.

LangMAP addresses the need for language-adaptive segmentation while maintaining compatibility with a fixed, shared vocabulary. This paradigm ensures both segmentation quality and operational efficiency, without necessitating retraining or architectural modifications.

LangMAP Algorithm: Architecture and Operation

LangMAP extends the UnigramLM framework [kudo-2018-subword], leveraging a probabilistic generative model that posits the latent segmentation of an input string as a MAP inference problem. Instead of a single global distribution, LangMAP learns a collection of language-specific unigram distributions over the shared vocabulary. These distributions are estimated independently via EM, using language-labeled corpora. The shared vocabulary is never altered; only token probabilities are adapted per-language.

During inference, for any input string, LangMAP computes the MAP segmentation under each language-specific unigram distribution. The segmentation with the highest likelihood across all languages is selected, implicitly performing language identification (akin to UniLID [meister2026languagethisasktokenizer]). Owing to the fixed segmentation lattice, inference overhead is linear in the number of languages, supporting scalable multilingual deployment.

Intrinsic Evaluation

LangMAP was validated across 14 tokenizers, 9 natural languages, and 9 programming languages. The empirical strategy leverages TokEval metrics [meister_tokenizer_analysis_2025], including morphological boundary alignment (MorphScore micro-F1, recall, precision), surface efficiency (fertility, compression), and information-theoretic metrics (R$\mathrm{e}$nyi entropy).

Morphological alignment gains are consistently positive for Latin-script languages, with pooled recall improvement ($+0.086$) and micro-F1 ($+0.032$). For programming languages, LangMAP improved AST-leaf-boundary alignment and identifier fragmentation in 86% and 85% of the (model, language) combinations, respectively.

Figure 1: Code AST and identifier metrics against the base tokenizer for 9 model families and 9 programming languages, showing substantial increases in alignment and reduced fragmentation under LangMAP.

LangMAP's improvements are contingent on vocabulary coverage; languages with insufficient script representation (e.g., Hindi with limited Devanagari pieces) show negligible gains. Where adequate subwords exist, LangMAP re-aligns segmentation boundaries to morpheme or syntactic units, yielding higher intrinsic metrics.

Coverage Headroom and Script-Specific Performance

LangMAP's MorphScore gains are largest for tokenizers with low initial recall, reaffirming that adaptation is limited by vocabulary coverage. Analysis reveals dramatic disparities: tokenizers contain tens of thousands of Latin subwords, but only a handful for non-Latin scripts. This underscores that language-adaptive scoring cannot compensate for the absence of relevant pieces.

Figure 2: LangMAP MorphScore recall as a function of original tokenizer recall; improvements are most pronounced where base recall is lowest, bounded by the presence of vocabulary pieces supporting the target script.

Extrinsic Evaluation: Downstream Impact

LangMAP's downstream effectiveness was assessed via fine-tuning experiments on Finnish, Hungarian, and Turkish (agglutinative languages). After adaptation of pretrained models using either the base tokenizer or LangMAP, grammatical acceptability on MultiBLiMP benchmarks increased in 5 of 6 model-size/language trials, e.g., Finnish 0.740 → 0.902 (+0.162) for 0.5B models.

Performance on knowledge-intensive tasks (Global-PIQA, Belebele) was unchanged or marginally degraded, supporting the claim that tokenization improvements primarily affect tasks dependent on linguistic structure, rather than factual recall.

Figure 3: Performance comparison on downstream tasks after fine-tuning with LangMAP versus base tokenization; gains manifest primarily in morphosyntactic acceptability.

Practical and Theoretical Implications

LangMAP demonstrates that language-adaptive parameterization over a shared vocabulary can mitigate segmentation bias in multilingual pipelines, achieving segmentation optimized for each language without imposing embedding or architectural adaptation costs. The method is fully compatible with pretrained models and only requires language labels during estimation (not inference), offering scalability and operational simplicity.

Theoretical implications include a reframing of tokenization quality in terms of segmentation plausibility, not vocabulary augmentation. However, representational limitations remain: absent vocabulary pieces restrict the upper bound achievable by parameter reweighting alone. Pairing LangMAP with vocabulary extension methods would remove this bottleneck.

Operationally, LangMAP is computationally efficient and supports incremental extension to new languages with minimal sample complexity. Language identification during inference is robust and label-free, enabling integration with large-scale pipelines.

Limitations and Future Directions

LangMAP's reliance on fixed vocabulary constitutes its main limitation; languages with inadequate script coverage cannot benefit from reweighting. Extension with dynamic vocabulary augmentation and unsupervised estimation strategies is a logical progression.

Extrinsic improvements are task-dependent, largely confined to linguistic acceptability. Broader downstream evaluation across more models, language types, and tasks is required to fully characterize the method's generalizability.

Conclusion

LangMAP provides an effective, language-adaptive tokenization framework that achieves improved morphological and syntactic alignment for languages with sufficient vocabulary coverage, while maintaining compatibility with existing pretrained models. Empirical results validate substantial gains in intrinsic metrics and select downstream tasks. Future research should focus on vocabulary extension, unsupervised adaptation, and broader task evaluation to overcome current limitations and further enhance multilingual NLP model fairness and efficacy (2606.23566).