INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (2411.19799v1)

Abstract: The performance differential of LLMs (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

• The paper introduces a novel benchmark of 197,243 questions in 44 languages to evaluate multilingual LLMs.
• The methodology involves native speaker verification and categorization of questions by regional and academic dimensions.
• The findings reveal significant performance gaps in regional specificity, stressing the need for targeted multilingual training.

Evaluating Multilingual Language Understanding with Regional Knowledge

The paper addresses a significant challenge in the development of multilingual LLMs: the imbalance in performance across different languages due to the lack of comprehensive evaluation resources. This gap restricts the deployment and utility of LLMs in many regions, thus limiting the societal and economic potential of AI tools globally. The authors tackle this problem by constructing and releasing a multilingual benchmark designed to evaluate LLMs in diverse regional and cultural contexts.

Contribution and Methodology

The core contribution of the paper is the introduction of a novel benchmark consisting of 197,243 multiple-choice questions in 44 languages, specifically curated to evaluate multilingual LLMs. This dataset is sourced from regional educational, professional, and occupational exams, ensuring that the evaluation is grounded in authentic and contextually relevant material. The benchmark stands out not only due to its scale but also due to its focus on capturing regional nuances that are often overlooked in translated datasets.

The methodology involved collecting and verifying questions by native speakers, ensuring linguistic and cultural accuracy. The authors categorized these questions into regional and non-regional types and further divided them among academic fields such as Humanities, STEM, and Domain-specific studies. This classification allows for nuanced analysis of LLM performance across different dimensions of regional knowledge.

Findings and Analysis

The paper provides a thorough analysis of LLM performance using the benchmark, revealing considerable variance in model capabilities across languages and disciplines. Notably, LLMs excel in languages they were explicitly trained on but show substantial performance degradation in "unseen" languages or those with different scripts. This emphasizes the contribution of cross-lingual transfer facilitated by script similarity and highlights the models’ struggles with regional specificity—particularly in professional and license examinations that require localized knowledge.

The analysis includes evaluations of high-performing models like GPT-4o in multiple settings, including five-shot and zero-shot paradigms. The findings indicate that while models can transfer some global knowledge to languages related by script or linguistic family, they often falter on unique regional content. Moreover, the paper underscores that performance discrepancies are frequently tied to a model's inability to process nuanced regional contexts.

Implications and Future Directions

The implications of this work are significant for both practical and theoretical domains in AI. Practically, the benchmark offers a valuable tool for developers aiming to improve the regional understanding of LLMs, fostering more equitable AI deployment. It also highlights the necessity for more focused training on diverse language data to bridge performance gaps.

Theoretically, the findings spur further questions about language representation in LLMs and the depths of cross-lingual transfer capabilities. This can drive future research exploring how language similarities and script sharing might be leveraged or improved within models’ architectures to enhance multilingual understanding.

Moving forward, the authors' approach of periodically releasing segments of the benchmark is strategic in mitigating saturation effects and preserving its utility over time. This benchmark sets a new precedent for creating and evaluating multilingual AI, balancing comprehensive inclusivity with rigorous contextual relevance. It also opens avenues for constructing similar benchmarks in other domains where cultural and regional specificity is paramount.

In summary, this paper makes a substantial contribution to the field by providing a detailed and contextually relevant resource for evaluating and improving the multilingual capabilities of LLMs. By addressing the core issue of regional and linguistic diversity, it paves the way toward more inclusive and equitable AI systems.