---
title: 'QFrCoLA: Quebec-French Acceptability Benchmark'
url: https://www.emergentmind.com/topics/qfrcola
type: topic
---

# QFrCoLA: Quebec-French Acceptability Benchmark

Searching arXiv for the QFrCoLA paper and closely related acceptability-benchmark work.
QFrCoLA, short for **Quebec-French Corpus of Linguistic Acceptability Judgments**, is a binary acceptability-judgment benchmark for **Quebec French** introduced to support the syntactic and grammatical evaluation of language models in a setting analogous to CoLA-style benchmarking across other languages [2508.16867]. It consists of **25,153 in-domain** sentences and **2,675 out-of-domain** sentences labeled according to whether they are **grammatical/acceptable** or **ungrammatical/unacceptable**, with labels defined as **0 = ungrammatical / not acceptable** and **1 = grammatical / acceptable** [2508.16867]. QFrCoLA was created to address the absence of a French acceptability corpus in the multilingual benchmarking landscape and is positioned as the **first large-scale French acceptability corpus** and the **second largest such corpus in the literature overall** after NoCoLA [2508.16867].

## 1. Position within multilingual acceptability benchmarking

QFrCoLA belongs to the family of **binary sentence classification** benchmarks in which a model must distinguish acceptable from unacceptable sentences. In this respect it follows the broad paradigm of **CoLA** and related resources for other languages, enabling direct comparison of language-model performance on linguistic acceptability across a multilingual set of corpora [2508.16867].

The paper explicitly situates QFrCoLA alongside **CoLA** for English, **DaLAJ** for Swedish, **ITACoLA** for Italian, **RuCoLA** for Russian, **CoLAC** for Chinese, **NoCoLA** for Norwegian, and **JCoLA** for Japanese [2508.16867]. Before QFrCoLA, this set lacked a French benchmark. The dataset therefore fills a structural gap in multilingual evaluation rather than merely extending an existing French resource.

The authors also distinguish acceptability classification from the **minimal-pairs** probing paradigm associated with resources such as BLiMP and CLiMP. QFrCoLA is not a minimal-pair benchmark; it is a binary acceptability dataset. Their future work aims to convert QFrCoLA into a French minimal-pair benchmark as well [2508.16867]. This suggests a broader research agenda in which French linguistic evaluation would be supported by both coarse acceptability classification and contrastive probing.

## 2. Quebec French, normativity, and the definition of judgment

A defining property of QFrCoLA is its focus on **Quebec French**. The in-domain portion is grounded in a Quebec linguistic resource, whereas the out-of-domain split is derived from a France-based normative source [2508.16867]. This design matters because Quebec French and France French share much of their grammar but differ in some accepted usages, including title feminization, so the corpus encodes not only source shift but also variety shift [2508.16867].

The dataset is explicitly based on **normative** judgments rather than direct elicitation of speakers’ acceptability intuitions. Its examples illustrate what official sources consider correct or incorrect, not speakers’ “feelings” about sentence well-formedness [2508.16867]. The paper nonetheless concludes that the dataset is “similar to linguistic acceptability judgment” in practice and can benchmark language models on linguistic judgment capabilities [2508.16867].

This distinction is methodologically important. QFrCoLA does not claim to be a corpus of psycholinguistic acceptability ratings. Instead, it operationalizes grammatical acceptability through curated normative examples. A plausible implication is that the benchmark probes a model’s ability to recover institutionally codified grammatical distinctions in Quebec French, while still functioning similarly to other acceptability resources used in NLP and generative-linguistic evaluation.

## 3. Sources, extraction, and annotation procedure

The **in-domain** corpus was built from the **Banque de dépannage linguistique (BDL)**, an official online resource maintained by the **Office québécois de la langue française (OQLF)**. The BDL contains **2,667 articles** across categories including spelling and syntax, and these articles provide examples of correct and incorrect usage [2508.16867]. The **out-of-domain** split comes from the **Académie française**, specifically its online journal *La langue française: Dire, Ne pas dire*, which contains **1,013 articles** with normative examples of proper and improper French usage [2508.16867].

The data was **manually extracted** from both sources. For the in-domain portion, the authors read all **2,667 BDL articles**, manually extracted **25,153** sentences, and assigned binary labels according to the source’s **green/red color scheme** [2508.16867]. For the OOD split, they manually extracted **2,675** sentences from **1,013 articles** and assigned binary labels following the source’s left/right contrast between correct and incorrect forms [2508.16867].

The annotation protocol relied on inherited source formatting and manual extraction rather than multi-annotator judgment elicitation. The paper does **not** report an inter-annotator agreement study, does **not** describe a multi-annotator adjudication protocol, and does **not** provide standard inter-annotator metrics [2508.16867]. Category mapping for the in-domain portion was guided by French linguistic literature, but quality control beyond this process is not reported [2508.16867]. This is one of the dataset’s explicit methodological limitations.

## 4. Corpus size, splits, and linguistic coverage

QFrCoLA comprises **25,153 in-domain sentences** and **2,675 out-of-domain sentences** [2508.16867]. The in-domain split is partitioned into training, development, and test subsets with nearly identical acceptable-sentence rates.

| Split | Sentences | Acceptable rate |
|---|---:|---:|
| Train | 15,846 | 69.49% |
| Dev | 1,761 | 69.51% |
| Test | 7,546 | 69.49% |
| OOD holdout | 2,675 | 53.91% |

The vocabulary statistics reported in the paper are **18,350** for train, **5,369** for dev, **12,690** for test, **22,131** for total in-domain, and **1,651** for the OOD holdout [2508.16867]. Among compared corpora, QFrCoLA is described as **second largest by sentence count after NoCoLA**, **much larger than most CoLA-style datasets**, and **third largest in vocabulary size among those compared** [2508.16867].

The in-domain data is organized into four broad linguistic classes:

| Category | Sentences | Acceptable rate |
|---|---:|---:|
| Syntax | 5,152 | 77.24% |
| Morphology | 10,642 | 68.26% |
| Semantic | 5,442 | 72.97% |
| Anglicism | 3,917 | 57.18% |

The **syntax** category includes agreement violations, word order corruption, misconstruction of clauses or phrases, apposition misuse, verb transitivity or argument structure violations, ellipsis, and missing constituents or words [2508.16867]. **Morphology** includes incorrect derivation, word building, and non-existent words; **semantic** covers incorrect negation and violations of semantic argument structure; **anglicism** targets words or syntactic structures borrowed from English grammar [2508.16867].

The distribution is skewed, with morphology accounting for about **43%** of the corpus, and the authors explicitly identify this skew as a limitation [2508.16867].

## 5. Experimental framework and evaluation protocol

All experiments use **binary acceptability classification** and are evaluated with **Accuracy** and **Matthews correlation coefficient (MCC)** [2508.16867]. The paper states that dev-set accuracy is used as the target metric for hyperparameter tuning and early stopping, and that results for non-LLM fine-tuned models are averaged over **ten restarts** with random seeds given as `$[42, 43, \cdots, 50, 51]$` [2508.16867].

The trivial baseline always predicts class **1**, i.e., always predicts that the sentence is acceptable [2508.16867]. The principal monolingual Transformer baseline for French is **CamemBERT-base**, corresponding to the paper’s “BERT” condition for QFrCoLA [2508.16867]. Additional systems include **LA-TDA**, which fine-tunes a transformer, extracts attention maps, uses topological data analysis features, and trains a logistic regression classifier on those features; a fine-tuned cross-lingual **XLM-RoBERTa-base**; and zero-shot binary-classification LLMs including **BLOOM-7B**, **BLOOMZ-7B**, **Mistral-7B-v0.3**, **Llama-3.1-8B**, **Lucie**, and instruct variants where available [2508.16867].

The fine-tuning configuration is fixed across experiments: **4 epochs**, **AdamW**, learning rate `$3e{-5}$`, weight decay `$1e{-2}$`, batch size **32**, default HuggingFace tokenization, maximum sequence length **64 tokens**, and **no lowercasing** during tokenization [2508.16867]. Because the datasets are imbalanced, the experiments use a **weighted balanced loss based on the train split percentage of acceptable sentences** [2508.16867].

This setup places QFrCoLA within a controlled multilingual comparison. The paper’s broader finding across the eight languages is that fine-tuned Transformer models are strong baselines, LA-TDA is often best, zero-shot LLMs generally perform poorly, and cross-lingual XLM-R fine-tuning does not reliably transfer acceptability knowledge across languages [2508.16867].

## 6. Empirical findings on QFrCoLA

On QFrCoLA itself, the best-performing tested method is the monolingual fine-tuned **CamemBERT-base**. On the in-domain test set, it reaches **82.92 ± 0.61** accuracy and **0.578 ± 0.015** MCC, slightly outperforming **LA-TDA**, which reaches **82.79 ± 0.45** accuracy and **0.574 ± 0.012** MCC [2508.16867]. The trivial baseline attains **69.49** accuracy and **0.000** MCC, while the fine-tuned cross-lingual **XLM-R** baseline is markedly weaker and unstable, with **69.91 ± 14.61** accuracy and **0.222 ± 0.240** MCC on test [2508.16867].

The zero-shot LLMs perform poorly. Many are well below the naive baseline in accuracy, and their MCC values are near zero or negative, indicating almost no useful correlation with the gold labels [2508.16867]. The paper emphasizes that this includes **Lucie**, a French-optimized model, which still performs poorly in zero-shot mode on QFrCoLA [2508.16867].

Category-wise evaluation shows that **CamemBERT** is strongest in all four linguistic categories, with the lowest results observed for **Anglicism**, both for Transformer baselines and for zero-shot LLMs [2508.16867]. For CamemBERT, the reported category-wise MCC values are **0.654 ± 0.018** for syntax, **0.563 ± 0.017** for morphology, **0.620 ± 0.011** for semantic, and **0.506 ± 0.028** for anglicism [2508.16867]. The paper interprets the difficulty of anglicism in terms of the frequency of English borrowings in web and educational text, the possibility that multilingual models rate English-like syntax or borrowed words as probable, and the smaller size of the anglicism subset [2508.16867].

The OOD split is substantially harder. On the OOD holdout, CamemBERT reaches **62.69 ± 1.13** accuracy and **0.286 ± 0.020** MCC, compared with **61.36 ± 0.90** accuracy and **0.090 ± 0.019** MCC for LA-TDA and **55.99 ± 4.36** accuracy and **0.107 ± 0.088** MCC for XLM-R [2508.16867]. The paper highlights that all three fine-tuned models show a large OOD degradation, with roughly **22% drop in accuracy** and roughly **50% drop in MCC**, which suggests overfitting to the in-domain BDL source and also reflects the Quebec-versus-France variety shift [2508.16867].

## 7. Interpretation, limitations, and significance

The authors argue that QFrCoLA is a **challenging** benchmark for several reasons: its examples are linguistically curated and theoretically meaningful, its judgments are subtle and norm-based, its OOD split involves both source and language-variety shift, strong models still show clear limits, and cross-lingual LLMs fail badly [2508.16867]. On this basis, they conclude that QFrCoLA functions as a meaningful benchmark of linguistic judgment ability rather than an artifact-driven easy classification task [2508.16867].

A central conclusion of the paper is that the tested pretrained cross-lingual LLMs do **not** appear to have acquired **Quebec-French linguistic judgment capabilities** during pretraining [2508.16867]. The supporting evidence is that zero-shot LLM accuracy is usually below the trivial baseline, MCC is near zero or negative, the multilingual fine-tuned XLM-R model remains far behind the monolingual French baseline, and OOD performance is difficult even for fine-tuned models [2508.16867].

The paper also states several limitations. QFrCoLA is based on **official normative grammar**, not broad usage variation; the OQLF source may reflect a Quebec governmental language-policy perspective; the corpus is category-skewed, especially toward morphology; and the OOD split is especially hard because some France-French norms differ from Quebec-French ones [2508.16867]. In addition, there is no reported inter-annotator agreement study [2508.16867].

Within multilingual acceptability research, QFrCoLA’s significance lies in extending CoLA-style evaluation to French while foregrounding **Quebec French** specifically. It enables direct comparison with existing resources such as CoLA, DaLAJ, ITACoLA, RuCoLA, CoLAC, NoCoLA, and JCoLA, and it provides evidence that fine-tuned monolingual Transformers remain substantially stronger than zero-shot LLMs for this type of grammatical judgment in French [2508.16867]. The dataset therefore occupies a dual role: as a language-specific benchmark for Quebec French and as a comparative instrument within the broader study of multilingual linguistic acceptability.

Source: https://www.emergentmind.com/topics/qfrcola