Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2412.03304v2)

Abstract: Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from differences in language but also from the cultural knowledge required to interpret questions, reducing the practical utility of translated datasets like MMLU. Furthermore, translation often introduces artefacts that can distort the meaning or clarity of questions in the target language. A common practice in multilingual evaluation is to rely on machine-translated evaluation sets, but simply translating a dataset is insufficient to address these challenges. In this work, we trace the impact of both of these issues on multilingual evaluations and ensuing model performances. Our large-scale evaluation of state-of-the-art open and proprietary models illustrates that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. Moreover, for questions requiring geographic knowledge, an astounding 84.9% focus on either North American or European regions. Rankings of model evaluations change depending on whether they are evaluated on the full portion or the subset of questions annotated as culturally sensitive, showing the distortion to model rankings when blindly relying on translated MMLU. We release Global MMLU, an improved MMLU with evaluation coverage across 42 languages -- with improved overall quality by engaging with compensated professional and community annotators to verify translation quality while also rigorously evaluating cultural biases present in the original dataset. This comprehensive Global MMLU set also includes designated subsets labeled as culturally sensitive and culturally agnostic to allow for more holistic, complete evaluation.

• The paper analyzes the widely used MMLU dataset, finding significant Western-centric cultural and linguistic biases, with 28% of questions requiring cultural knowledge and 84.9% centered on North American/European geography.
• It introduces Global-MMLU, a culturally sensitive benchmark covering 42 languages with separate culturally sensitive and agnostic subsets, developed using compensated professionals and community annotations.
• Findings suggest cultural sensitivity affects model ranking, with larger proprietary models performing better on sensitive tasks, highlighting the need for nuanced datasets and separate evaluations for equitable model assessment.

An Evaluation of Global-MMLU: Addressing Cultural Bias in Multilingual Models

The paper "Global MMLU -1pt: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation" provides a comprehensive analysis of cultural and linguistic biases inherent in the widely used Massive Multitask Language Understanding (MMLU) dataset. The paper meticulously documents the deficiencies of MMLU when utilized as a benchmark for multilingual models, particularly the entrenched Western-centric biases that distort evaluation outcomes. This analysis is pivotal for researchers striving to develop more culturally inclusive and equitable LLMs.

Methodology and Findings

The paper undertakes a large-scale evaluation of both open-weight and proprietary state-of-the-art models, revealing a significant reliance on Western-centric concepts within the MMLU dataset. Key findings of the paper highlight that 28% of the questions in the dataset require culturally sensitive knowledge, with an alarming 84.9% of questions necessitating geographic knowledge centered around North American or European regions. Such skewness in data impacts the validity of multilingual model assessments, as it overemphasizes certain cultural paradigms to the exclusion of others.

The authors introduce Global-MMLU, an enriched and culturally sensitive variant of MMLU, which incorporates evaluation across 42 languages. The dataset’s development involved rigorous quality checks through compensated professionals and community annotations. The authors released two distinct subsets within Global-MMLU: culturally sensitive and culturally agnostic questions, ensuring a more robust and comprehensive evaluation of LLMs on a global scale.

Analysis of Evaluation Subsets

The release of Global-MMLU substantially enhances the evaluation toolkit for LLMs by differentiating their performance on culturally sensitive versus culturally agnostic questions. The paper finds that cultural sensitivity significantly affects model rankings: proprietary and larger models, like GPT-4o, perform better on culturally sensitive tasks compared to their smaller open-weight counterparts. This variability underlines the necessity of employing culturally nuanced datasets for assessment to avoid implicit biases in model evaluations.

Implications and Recommendations

The implications of these findings are twofold. Practically, they advocate for multilingual datasets that reflect diverse cultural contexts, enhancing the universality and fairness of LLMs. Theoretically, the paper underscores the importance of cultural awareness in artificial intelligence evaluation, pressing for methodologies that adequately accommodate varying cultural paradigms and linguistic structures.

The authors recommend prioritizing Global-MMLU over translated MMLU datasets for evaluating multilingual models. This recommendation underscores the potential pitfalls of relying solely on translations, which may not capture the nuanced cultural distinctions present in different languages. Additionally, separate evaluation of culturally sensitive and agnostic questions is advised to glean more granular insights into model capabilities.

Future Work

Future developments may build upon this work by expanding the language scope and incorporating more dialectal variations within evaluations, thereby further refining model cultural competence. The methodological framework established by the authors could also serve as a template for assessing other multilingual datasets, prompting broader inquiries into regional biases inherent in AI systems.

In conclusion, this paper provides a vital contribution to the ongoing dialogue on the cultural inclusivity of LLMs. By introducing Global-MMLU and highlighting the critical role of cultural sensitivity in model evaluation, the authors advance the standards of AI evaluation towards a more inclusive and equitable future.