How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Abstract: In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual LLMs versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data, with monolingually and multilingually trained tokenizers. We find that while the pretraining data size is an important factor, a designated monolingual tokenizer plays an equally important role in the downstream performance. Our results show that languages that are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts. We further find that replacing the original multilingual tokenizer with the specialized monolingual tokenizer improves the downstream performance of the multilingual model for almost every task and language.

• The paper's main contribution is demonstrating that tokenizer design significantly affects the performance gap between monolingual and multilingual models.
• Controlled experiments across nine languages and five tasks reveal that using language-specific tokenizers notably enhances performance.
• Findings indicate that tokenizer efficiency is as crucial as training data size, guiding future improvements in NLP model optimization.

An Empirical Evaluation of Monolingual vs. Multilingual LLMs

The paper "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual LLMs" presents a comprehensive empirical comparison between pretrained multilingual LLMs and their monolingual counterparts. The research focuses on the monolingual task performance across nine typologically diverse languages and five diverse downstream tasks. This analysis is crucial for understanding the efficacy of multilingual models versus specialized monolingual models, particularly in terms of representation and tokenizer efficiency.

Methodology and Results

The study investigates whether a performance gap exists between multilingual and monolingual models and explores potential reasons for such differences. This is achieved by engaging in controlled experiments that isolate variables such as tokenizer types and data sizes. Monolingual models are trained on the same datasets with both monolingual and multilingual tokenizers to differentiate the effects of the tokenizer from the pretraining data size.

Key findings indicate that an appropriately designed tokenizer is as critical as the quantity of pretraining data. The paper reports that languages well-represented in the multilingual model's vocabulary tend to exhibit minimal performance decreases compared to their monolingual versions. Furthermore, substituting the generic multilingual tokenizer with a language-specific monolingual tokenizer notably enhances performance for almost all evaluated tasks and languages.

Technical Insights

Significant insights from the study reveal that the pretraining data size is a substantial factor in the algorithm's proficiency. However, the designated tokenizer's proficiency plays a pivotal role in downstream task performance. This discovery underscores the necessity of designing tokenizers that are sensitive to language-specific properties, which existing multilingual tokenizers may overlook due to their broad applicability and shared vocabulary constraints.

Implications and Future Directions

The research implies that NLP systems can achieve considerable efficiency gains through language-specific tokenizer design, and further, that the existing gaps in performance between monolingual and multilingual models could be significantly reduced. Practical applications of these findings could include optimized LLM deployment for specific tasks where tokenizer precision is vital, like sentiment analysis or entity recognition.

Theoretical implications suggest an intriguing area for further research on tokenization strategies that eschew traditional shared vocabularies for more finely tuned or dynamically adaptable solutions. Future developments in AI could see advancements in tokenization processes as a pivotal factor for improving multilingual model efficiency without necessarily expanding model size.

The paper's findings emphasize that a monolingual performance optimized through strategic tokenizer adaptation offers a substantial approach to enhancing multilingual NLP applications. This work lays foundational knowledge for the continuous development of efficient, multilingual NLP systems that maintain high performance levels across diverse language tasks.