Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning

Abstract: Multilingual pre-trained LLMs (PLMs) have demonstrated impressive performance on several downstream tasks for both high-resourced and low-resourced languages. However, there is still a large performance drop for languages unseen during pre-training, especially African languages. One of the most effective approaches to adapt to a new language is \textit{language adaptive fine-tuning} (LAFT) -- fine-tuning a multilingual PLM on monolingual texts of a language using the pre-training objective. However, adapting to a target language individually takes a large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language. In this paper, we perform \textit{multilingual adaptive fine-tuning} on 17 most-resourced African languages and three other high-resource languages widely spoken on the African continent to encourage cross-lingual transfer learning. To further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non-African writing scripts before MAFT, thus reducing the model size by around 50%. Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment classification) shows that our approach is competitive to applying LAFT on individual languages while requiring significantly less disk space. Additionally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods.

• The paper proposes MAFT, a novel technique that adapts multilingual PLMs using simultaneous fine-tuning on multiple African languages.
• It achieves a 50% reduction in model size through vocabulary pruning, ensuring efficiency without sacrificing performance.
• Evaluation on NER, news classification, and sentiment tasks demonstrates enhanced cross-lingual transfer and practical benefits for low-resource scenarios.

Introduction

The paper "Adapting Pre-trained LLMs to African Languages via Multilingual Adaptive Fine-Tuning" (2204.06487) addresses a core challenge in the deployment of multilingual pre-trained LLMs (PLMs) for African languages. Multilingual PLMs such as XLM-R and AfriBERTa have shown significant promise in various NLP tasks. However, a significant performance divide still exists between languages seen during pre-training and African languages, which are frequently underrepresented. This paper tackles this issue by proposing a method of adapting PLMs through Multilingual Adaptive Fine-Tuning (MAFT) to better capture the characteristics of African languages.

Methodology

The primary contribution of the paper is the introduction of MAFT, a technique that fine-tunes a multilingual PLM on monolingual texts from multiple languages simultaneously. This differs from the traditional Language Adaptive Fine-Tuning (LAFT) approach which targets individual languages, thus improving cross-lingual transfer capabilities while simultaneously reducing the model's disk space requirements. By performing MAFT on 17 African languages, along with English, French, and Arabic, the study demonstrates enhanced representation for languages that significantly lack resources.

A key component involves the pruning of non-essential vocabulary tokens from the embedding layers, effectively reducing the model size by approximately 50% without significant performance trade-offs. This is essential for resource-constrained environments commonly found in African countries, making the adjusted PLM not only efficient but also pragmatic for real-world applications.

Evaluation

Extensive evaluations were conducted on two multilingual PLMs (AfriBERTa and XLM-R) over three NLP tasks: named entity recognition (NER), news topic classification, and sentiment classification. The results indicate that the MAFT approach is competitive relative to standalone LAFT models, achieving similar levels of performance while being more resource-efficient. Specifically, the paper exhibits that its adapted models provide better zero-shot cross-lingual transfer ability in conjunction with parameter-efficient fine-tuning methods such as Adapters and sparse fine-tuning techniques.

Furthermore, the study expands on existing resources by creating an African News Topic Classification (ANTC) corpus, broadening the evaluation dataset pool in five additional African languages. This new corpus facilitates a deeper and broader evaluation of model capabilities across diverse linguistic landscapes.

Results

The results indicate success in maintaining competitive performance levels across multiple tasks and languages. With MAFT, the adapted models require significantly less storage compared to models necessitating fine-tuning for each language individually. The evaluation shows gains in efficiency with minimal impact on performance, underscoring the potential for broader application in settings with limited computational resources.

By generating a 50% model size reduction post-vocabulary pruning, the technique not only simplifies deployment but also ensures that the model remains feasible for users with limited hardware capabilities—a significant consideration for African languages and computing environments.

Implications and Future Work

The implications of this study are profound, suggesting that the future of multilingual PLMs will likely rely heavily on adaptive techniques that maximize efficiency and performance across varied linguistic landscapes. Leveraging MAFT highlights an avenue whereby significant reductions in computational footprint do not inherently detract from model efficacy, promoting sustainable and inclusive NLP advancements.

Future developments within AI, particularly in the automated processing and understanding of underrepresented languages, can greatly benefit from this approach. Outsized improvements for low-resource contexts are evident, opening pathways for the development of even more sophisticated language tools and resources. Targeted expansion of language coverage and increased access to high-quality data remain crucial for further research, with the ultimate aim of democratizing AI benefits across all global language speakers.

Conclusion

The paper successfully presents MAFT as a robust and scalable solution to the challenges faced by African languages in the NLP landscape. By offering superior adaptability through vocabulary efficiency and cross-lingual transfer capabilities, it sets a precedent for the future direction of inclusive, multilingual NLP research. Ultimately, this research underscores the importance of catering to linguistic diversity—ensuring that AI developments cater to the needs of all linguistic communities.