What is "Typological Diversity" in NLP? (2402.04222v4)

Abstract: The NLP research community has devoted increased attention to languages beyond English, resulting in considerable improvements for multilingual NLP. However, these improvements only apply to a small subset of the world's languages. Aiming to extend this, an increasing number of papers aspires to enhance generalizable multilingual performance across languages. To this end, linguistic typology is commonly used to motivate language selection, on the basis that a broad typological sample ought to imply generalization across a broad range of languages. These selections are often described as being 'typologically diverse'. In this work, we systematically investigate NLP research that includes claims regarding 'typological diversity'. We find there are no set definitions or criteria for such claims. We introduce metrics to approximate the diversity of language selection along several axes and find that the results vary considerably across papers. Crucially, we show that skewed language selection can lead to overestimated multilingual performance. We recommend future work to include an operationalization of 'typological diversity' that empirically justifies the diversity of language samples.

• The paper critically evaluates typological diversity claims in NLP literature and introduces metrics like mean pairwise syntactic distance (MPSD) to assess language selection.
• It systematically reviews multilingual studies to reveal discrepancies between language sampling practices and actual linguistic diversity.
• The authors recommend standardizing language selection reporting to prevent skewed generalizations and improve the generalizability of multilingual models.

Introduction

The NLP research community predominantly focuses on English, while multilingual NLP has been a secondary concern. Recently, there has been a shift to encompass more languages, with an emphasis on evaluating multilingual model performance across a suitably diverse sample of the world's languages. Considering linguistic diversity is thought to imply robust generalizability, yet the lack of a clear definition of 'typological diversity' in NLP research has been problematic. This paper provides a vital critique of how 'typological diversity' claims are substantiated in NLP literature and offers metrics for assessing the diversity of language selection along several axes.

Survey Methodology

The paper presents a comprehensive analysis of the use of 'typological diversity' in NLP research. To conduct their investigation, the authors formulated a set of metrics to evaluate and define 'typological diversity' across a series of studies. They surveyed NLP literature for claims of typological diversity and scrutinized the language sample justification provided in these studies, if any. Inter-annotator agreement was employed to ensure consistency in scoping the claims about linguistic diversity and dataset introduction. The systematic review covered well-known conferences and journals, and multiple justifications for typological diversity claims were annotated and discussed.

Analysis of Language Diversity

In assessing language diversity, the authors revealed significant variation in 'typological diversity' claims across papers. They proposed using mean pairwise syntactic distance (MPSD) and typological feature inclusion as approximate metrics. The data suggested a discrepancy between multilingual model evaluation and real-world linguistic diversity due to skewed language selections, which often lead to overestimated multilingual performance. The analysis demonstrated that simply adding more languages to a paper does not necessarily increase its typological diversity; rather, researchers should consider language selection more carefully to improve the generalizability of their findings.

Recommendations and Implications

The authors advocate for future research to incorporate a defined operational method for typological diversity to avoid skewed generalizations. They recommend documenting language selection and employing measures such as MPSD or typological feature inclusion, enhancing our understanding of linguistic challenges in multilingual NLP modelling. Additionally, it is noted that the developed metrics and tools are approximations, given the incomplete linguistic resources and understanding. The authors proceed with an ethical perspective, emphasizing that expanding NLP applications to numerous languages is not an inherently positive aim without considering the sociocultural impact.

In conclusion, the paper highlights the issue of unsubstantiated 'typological diversity' claims in multilingual NLP literature and the potential pitfalls in assuming generalizability when such diversity is assumed but not empirically justified. It emphasizes the importance of principled reporting on linguistic diversity and the need to refine methodologies for claiming typological diversity. Not only does this have implications for the accuracy of multilingual model performance, but it also affects our understanding of the actual diversity present in the data that such models are tested against.