QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments

Abstract: Large and Transformer-based LLMs perform outstandingly in various downstream tasks. However, there is limited understanding regarding how these models internalize linguistic knowledge, so various linguistic benchmarks have recently been proposed to facilitate syntactic evaluation of LLMs across languages. This paper introduces QFrCoLA (Quebec-French Corpus of Linguistic Acceptability Judgments), a normative binary acceptability judgments dataset comprising 25,153 in-domain and 2,675 out-of-domain sentences. Our study leverages the QFrCoLA dataset and seven other linguistic binary acceptability judgment corpora to benchmark seven LLMs. The results demonstrate that, on average, fine-tuned Transformer-based LM are strong baselines for most languages and that zero-shot binary classification LLMs perform poorly on the task. However, for the QFrCoLA benchmark, on average, a fine-tuned Transformer-based LM outperformed other methods tested. It also shows that pre-trained cross-lingual LLMs selected for our experimentation do not seem to have acquired linguistic judgment capabilities during their pre-training for Quebec French. Finally, our experiment results on QFrCoLA show that our dataset, built from examples that illustrate linguistic norms rather than speakers' feelings, is similar to linguistic acceptability judgment; it is a challenging dataset that can benchmark LM on their linguistic judgment capabilities.

• The paper presents QFrCoLA as a novel benchmark dataset for evaluating linguistic acceptability in Quebec French using binary judgments.
• It systematically compares fine-tuned Transformer models with cross-lingual LLMs, highlighting notable drops in out-of-domain performance.
• The study offers actionable insights and future directions for improving language models in handling syntactic and morphological nuances in non-English contexts.

QFrCoLA: A Quebec-French Corpus of Linguistic Acceptability Judgments

Introduction

The paper "QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments" introduces a novel dataset designed for evaluating linguistic acceptability in Quebec French. This dataset, known as QFrCoLA, consists of normative binary acceptability judgments and is composed of both in-domain and out-of-domain sentences. It serves as a benchmark to assess the performance of LMs in distinguishing between acceptable and unacceptable linguistic forms, with specific application to Quebec French.

Dataset Composition

QFrCoLA comprises 25,153 in-domain and 2,675 out-of-domain sentences, extracted from two primary sources: the Banque de dépannage linguistique (BDL) and the Académie française's online journal. The BDL, provided by the Office québécois de la langue française, represents the in-domain source, containing normative examples of Quebec French usage across various linguistic phenomena. The out-of-domain data are derived from the Académie française, presenting examples that reflect normative grammar used in France.

Figure 1: Snipped of the BDL article for the French adverb ``#1''{alentour.

Figure 2: Snipped of an Académie française article for the ``#1''{Omission de la conjonction « que » (Omission of the conjunction "that")}.

Benchmarking and Model Evaluation

The study evaluates eight LLMs on QFrCoLA and seven other linguistic binary acceptability judgment corpora. Models range from monolingual Transformer-based models to cross-lingual LLMs under a zero-shot evaluation setup. Notably, fine-tuned Transformer-based LMs were identified as strong baselines, demonstrating robust performance in these tasks, exceeding state-of-the-art (SOTA) methods like LA-TDA in certain metrics.

However, the results revealed that pre-trained cross-lingual LLMs, including those optimized for French such as Lucie, did not inherently acquire the linguistic judgment capabilities for Quebec French during pre-training. Fine-tuned models displayed a significant drop in performance on the OOD split, indicating challenges in generalizing beyond the in-domain context.

Practical Implications and Future Developments

The introduction of QFrCoLA marks a significant development in the study of linguistic judgments in non-English languages, particularly for Quebec French, where previous resources were limited. This dataset can be pivotal in training and evaluating LMs to enhance their understanding of linguistic acceptability, offering insights into their robustness against syntactic and morphological errors.

Future research directions suggested include extending the granularity of linguistic phenomena within the dataset and developing minimal pair benchmarks for more nuanced evaluation. Additionally, exploring the qualitative errors made by LLMs when handling Quebec French presents a valuable avenue for refining LLMs and improving their cross-linguistic transfer capabilities.

Conclusion

QFrCoLA offers a substantial contribution to the field of NLP by providing a robust dataset for examining linguistic acceptability in Quebec French. It challenges existing models to refine their linguistic competencies and presents opportunities for future explorations in multilingual NLP applications. As this dataset expands, it promises to refine our understanding of LLMs' capabilities and limitations in processing non-English linguistic variations.