Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization (2406.13718v2)

Abstract: While LLMs exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.

• The paper introduces a Bayesian framework to simulate phonological, morphological, and lexical noise, showing its effect on LLM performance.
• The paper finds that machine translation tasks degrade predictably under phonological noise, while classification tasks show additional complexity.
• The paper demonstrates that performance trends in synthetic languages reliably predict real-world cross-lingual generalization, urging diverse training data.

Evaluating LLMs along Dimensions of Language Variation: A Systematic Investigation of Cross-lingual Generalization

This paper presents a comprehensive paper of cross-lingual generalization in LLMs, specifically focusing on performance degradation (PD) in closely related languages (CRLs) and dialects that are not part of their high-resource language neighbors (HRLNs) in training data. The paper aims to understand how linguistic distances affect PD and proposes a framework for generating artificial languages that simulate controlled linguistic variance from the HRLN. The authors propose Bayesian noise models to simulate phonological, morphological, and lexical distance as factors contributing to PD.

Methodology Overview

The authors approach the challenge by parametrizing noise processes along three linguistic axes: phonological, morphological, and lexical. These processes are modeled as probabilistic noisers applied to input data, thus synthesizing artificial languages with controlled variance. By testing LLMs on these synthesized languages, the paper assesses the robustness and generalization capabilities of models across various tasks, including machine translation, story comprehension, and natural language inference.

Key Findings

1. Intrinsic Task Sensitivity: The paper finds that task sensitivity to linguistic noise greatly varies, with machine translation showing a more predictable degradation pattern in PD due to its reliance on local understanding, whereas classification tasks like XNLI and XStoryCloze exhibit additional complexity due to their reliance on specific content words.
2. Noise Type Impact: Different noise types lead to varying impacts on model performance. Phonological noise, for instance, introduces a sharp drop in performance due to its extensive input alteration at a character level. On the other hand, morphological noise is less impactful, suggesting models retain considerable reliability in understanding stem-level information.
3. Linguistic Distance and Model Performance: The authors validate their hypothesis by demonstrating that PD trends, as derived from the artificial languages, effectively predict LLM performance for real CRL-HRLN pairs. This validation suggests that the proposed noising approach successfully captures linguistic variance as it affects PD.
4. Linguistic Typology Influence: Typologically rich languages suffer more from noise, underpinning the necessity for handling linguistic diversity in training and evaluation of LLMs to improve cross-lingual performance.

Implications and Future Directions

The framework provides a principled basis for understanding and predicting LLM generalization capabilities in the context of linguistic variation. It opens a pathway for targeted interventions to mitigate PD in LRLs by informing systematic improvements or augmentations during training. For instance, incorporating more diverse and detailed linguistic phenomena into training processes, even syntactic ones not explored in this paper, could enhance robustness.

Furthermore, given the framework's ability to generate plausible pseudo-CRLs using linguistic posteriors, this approach might extend beyond research to practical applications in natural language processing for low-resource languages. Coupled with developing more linguistically informed noising processes, this could ground more equitable NLP technology across diverse linguistic spaces.

In conclusion, this paper represents a meaningful stride towards understanding and enhancing cross-lingual generalization in LLMs. It positions itself critically within ongoing dialogues about language inclusion in AI, underscoring the value of linguistic typology awareness and Bayesian simulation methods in advancing LLM capabilities.