QFrCoLA: Quebec-French Acceptability Benchmark
- QFrCoLA is a binary acceptability benchmark for Quebec French with 25,153 in-domain and 2,675 out-of-domain sentences labeled as grammatical or ungrammatical.
- The dataset is built from authoritative sources (OQLF and Académie française), categorizing examples into syntax, morphology, semantics, and anglicism.
- Empirical results reveal that monolingual fine-tuned models like CamemBERT outperform cross-lingual and zero-shot LLMs, underscoring the challenges of multilingual benchmarking.
Searching arXiv for the QFrCoLA paper and closely related acceptability-benchmark work. QFrCoLA, short for Quebec-French Corpus of Linguistic Acceptability Judgments, is a binary acceptability-judgment benchmark for Quebec French introduced to support the syntactic and grammatical evaluation of LLMs in a setting analogous to CoLA-style benchmarking across other languages (Beauchemin et al., 23 Aug 2025). It consists of 25,153 in-domain sentences and 2,675 out-of-domain sentences labeled according to whether they are grammatical/acceptable or ungrammatical/unacceptable, with labels defined as 0 = ungrammatical / not acceptable and 1 = grammatical / acceptable (Beauchemin et al., 23 Aug 2025). QFrCoLA was created to address the absence of a French acceptability corpus in the multilingual benchmarking landscape and is positioned as the first large-scale French acceptability corpus and the second largest such corpus in the literature overall after NoCoLA (Beauchemin et al., 23 Aug 2025).
1. Position within multilingual acceptability benchmarking
QFrCoLA belongs to the family of binary sentence classification benchmarks in which a model must distinguish acceptable from unacceptable sentences. In this respect it follows the broad paradigm of CoLA and related resources for other languages, enabling direct comparison of language-model performance on linguistic acceptability across a multilingual set of corpora (Beauchemin et al., 23 Aug 2025).
The paper explicitly situates QFrCoLA alongside CoLA for English, DaLAJ for Swedish, ITACoLA for Italian, RuCoLA for Russian, CoLAC for Chinese, NoCoLA for Norwegian, and JCoLA for Japanese (Beauchemin et al., 23 Aug 2025). Before QFrCoLA, this set lacked a French benchmark. The dataset therefore fills a structural gap in multilingual evaluation rather than merely extending an existing French resource.
The authors also distinguish acceptability classification from the minimal-pairs probing paradigm associated with resources such as BLiMP and CLiMP. QFrCoLA is not a minimal-pair benchmark; it is a binary acceptability dataset. Their future work aims to convert QFrCoLA into a French minimal-pair benchmark as well (Beauchemin et al., 23 Aug 2025). This suggests a broader research agenda in which French linguistic evaluation would be supported by both coarse acceptability classification and contrastive probing.
2. Quebec French, normativity, and the definition of judgment
A defining property of QFrCoLA is its focus on Quebec French. The in-domain portion is grounded in a Quebec linguistic resource, whereas the out-of-domain split is derived from a France-based normative source (Beauchemin et al., 23 Aug 2025). This design matters because Quebec French and France French share much of their grammar but differ in some accepted usages, including title feminization, so the corpus encodes not only source shift but also variety shift (Beauchemin et al., 23 Aug 2025).
The dataset is explicitly based on normative judgments rather than direct elicitation of speakers’ acceptability intuitions. Its examples illustrate what official sources consider correct or incorrect, not speakers’ “feelings” about sentence well-formedness (Beauchemin et al., 23 Aug 2025). The paper nonetheless concludes that the dataset is “similar to linguistic acceptability judgment” in practice and can benchmark LLMs on linguistic judgment capabilities (Beauchemin et al., 23 Aug 2025).
This distinction is methodologically important. QFrCoLA does not claim to be a corpus of psycholinguistic acceptability ratings. Instead, it operationalizes grammatical acceptability through curated normative examples. A plausible implication is that the benchmark probes a model’s ability to recover institutionally codified grammatical distinctions in Quebec French, while still functioning similarly to other acceptability resources used in NLP and generative-linguistic evaluation.
3. Sources, extraction, and annotation procedure
The in-domain corpus was built from the Banque de dépannage linguistique (BDL), an official online resource maintained by the Office québécois de la langue française (OQLF). The BDL contains 2,667 articles across categories including spelling and syntax, and these articles provide examples of correct and incorrect usage (Beauchemin et al., 23 Aug 2025). The out-of-domain split comes from the Académie française, specifically its online journal La langue française: Dire, Ne pas dire, which contains 1,013 articles with normative examples of proper and improper French usage (Beauchemin et al., 23 Aug 2025).
The data was manually extracted from both sources. For the in-domain portion, the authors read all 2,667 BDL articles, manually extracted 25,153 sentences, and assigned binary labels according to the source’s green/red color scheme (Beauchemin et al., 23 Aug 2025). For the OOD split, they manually extracted 2,675 sentences from 1,013 articles and assigned binary labels following the source’s left/right contrast between correct and incorrect forms (Beauchemin et al., 23 Aug 2025).
The annotation protocol relied on inherited source formatting and manual extraction rather than multi-annotator judgment elicitation. The paper does not report an inter-annotator agreement study, does not describe a multi-annotator adjudication protocol, and does not provide standard inter-annotator metrics (Beauchemin et al., 23 Aug 2025). Category mapping for the in-domain portion was guided by French linguistic literature, but quality control beyond this process is not reported (Beauchemin et al., 23 Aug 2025). This is one of the dataset’s explicit methodological limitations.
4. Corpus size, splits, and linguistic coverage
QFrCoLA comprises 25,153 in-domain sentences and 2,675 out-of-domain sentences (Beauchemin et al., 23 Aug 2025). The in-domain split is partitioned into training, development, and test subsets with nearly identical acceptable-sentence rates.
| Split | Sentences | Acceptable rate |
|---|---|---|
| Train | 15,846 | 69.49% |
| Dev | 1,761 | 69.51% |
| Test | 7,546 | 69.49% |
| OOD holdout | 2,675 | 53.91% |
The vocabulary statistics reported in the paper are 18,350 for train, 5,369 for dev, 12,690 for test, 22,131 for total in-domain, and 1,651 for the OOD holdout (Beauchemin et al., 23 Aug 2025). Among compared corpora, QFrCoLA is described as second largest by sentence count after NoCoLA, much larger than most CoLA-style datasets, and third largest in vocabulary size among those compared (Beauchemin et al., 23 Aug 2025).
The in-domain data is organized into four broad linguistic classes:
| Category | Sentences | Acceptable rate |
|---|---|---|
| Syntax | 5,152 | 77.24% |
| Morphology | 10,642 | 68.26% |
| Semantic | 5,442 | 72.97% |
| Anglicism | 3,917 | 57.18% |
The syntax category includes agreement violations, word order corruption, misconstruction of clauses or phrases, apposition misuse, verb transitivity or argument structure violations, ellipsis, and missing constituents or words (Beauchemin et al., 23 Aug 2025). Morphology includes incorrect derivation, word building, and non-existent words; semantic covers incorrect negation and violations of semantic argument structure; anglicism targets words or syntactic structures borrowed from English grammar (Beauchemin et al., 23 Aug 2025).
The distribution is skewed, with morphology accounting for about 43% of the corpus, and the authors explicitly identify this skew as a limitation (Beauchemin et al., 23 Aug 2025).
5. Experimental framework and evaluation protocol
All experiments use binary acceptability classification and are evaluated with Accuracy and Matthews correlation coefficient (MCC) (Beauchemin et al., 23 Aug 2025). The paper states that dev-set accuracy is used as the target metric for hyperparameter tuning and early stopping, and that results for non-LLM fine-tuned models are averaged over ten restarts with random seeds given as (Beauchemin et al., 23 Aug 2025).
The trivial baseline always predicts class 1, i.e., always predicts that the sentence is acceptable (Beauchemin et al., 23 Aug 2025). The principal monolingual Transformer baseline for French is CamemBERT-base, corresponding to the paper’s “BERT” condition for QFrCoLA (Beauchemin et al., 23 Aug 2025). Additional systems include LA-TDA, which fine-tunes a transformer, extracts attention maps, uses topological data analysis features, and trains a logistic regression classifier on those features; a fine-tuned cross-lingual XLM-RoBERTa-base; and zero-shot binary-classification LLMs including BLOOM-7B, BLOOMZ-7B, Mistral-7B-v0.3, Llama-3.1-8B, Lucie, and instruct variants where available (Beauchemin et al., 23 Aug 2025).
The fine-tuning configuration is fixed across experiments: 4 epochs, AdamW, learning rate , weight decay , batch size 32, default HuggingFace tokenization, maximum sequence length 64 tokens, and no lowercasing during tokenization (Beauchemin et al., 23 Aug 2025). Because the datasets are imbalanced, the experiments use a weighted balanced loss based on the train split percentage of acceptable sentences (Beauchemin et al., 23 Aug 2025).
This setup places QFrCoLA within a controlled multilingual comparison. The paper’s broader finding across the eight languages is that fine-tuned Transformer models are strong baselines, LA-TDA is often best, zero-shot LLMs generally perform poorly, and cross-lingual XLM-R fine-tuning does not reliably transfer acceptability knowledge across languages (Beauchemin et al., 23 Aug 2025).
6. Empirical findings on QFrCoLA
On QFrCoLA itself, the best-performing tested method is the monolingual fine-tuned CamemBERT-base. On the in-domain test set, it reaches 82.92 ± 0.61 accuracy and 0.578 ± 0.015 MCC, slightly outperforming LA-TDA, which reaches 82.79 ± 0.45 accuracy and 0.574 ± 0.012 MCC (Beauchemin et al., 23 Aug 2025). The trivial baseline attains 69.49 accuracy and 0.000 MCC, while the fine-tuned cross-lingual XLM-R baseline is markedly weaker and unstable, with 69.91 ± 14.61 accuracy and 0.222 ± 0.240 MCC on test (Beauchemin et al., 23 Aug 2025).
The zero-shot LLMs perform poorly. Many are well below the naive baseline in accuracy, and their MCC values are near zero or negative, indicating almost no useful correlation with the gold labels (Beauchemin et al., 23 Aug 2025). The paper emphasizes that this includes Lucie, a French-optimized model, which still performs poorly in zero-shot mode on QFrCoLA (Beauchemin et al., 23 Aug 2025).
Category-wise evaluation shows that CamemBERT is strongest in all four linguistic categories, with the lowest results observed for Anglicism, both for Transformer baselines and for zero-shot LLMs (Beauchemin et al., 23 Aug 2025). For CamemBERT, the reported category-wise MCC values are 0.654 ± 0.018 for syntax, 0.563 ± 0.017 for morphology, 0.620 ± 0.011 for semantic, and 0.506 ± 0.028 for anglicism (Beauchemin et al., 23 Aug 2025). The paper interprets the difficulty of anglicism in terms of the frequency of English borrowings in web and educational text, the possibility that multilingual models rate English-like syntax or borrowed words as probable, and the smaller size of the anglicism subset (Beauchemin et al., 23 Aug 2025).
The OOD split is substantially harder. On the OOD holdout, CamemBERT reaches 62.69 ± 1.13 accuracy and 0.286 ± 0.020 MCC, compared with 61.36 ± 0.90 accuracy and 0.090 ± 0.019 MCC for LA-TDA and 55.99 ± 4.36 accuracy and 0.107 ± 0.088 MCC for XLM-R (Beauchemin et al., 23 Aug 2025). The paper highlights that all three fine-tuned models show a large OOD degradation, with roughly 22% drop in accuracy and roughly 50% drop in MCC, which suggests overfitting to the in-domain BDL source and also reflects the Quebec-versus-France variety shift (Beauchemin et al., 23 Aug 2025).
7. Interpretation, limitations, and significance
The authors argue that QFrCoLA is a challenging benchmark for several reasons: its examples are linguistically curated and theoretically meaningful, its judgments are subtle and norm-based, its OOD split involves both source and language-variety shift, strong models still show clear limits, and cross-lingual LLMs fail badly (Beauchemin et al., 23 Aug 2025). On this basis, they conclude that QFrCoLA functions as a meaningful benchmark of linguistic judgment ability rather than an artifact-driven easy classification task (Beauchemin et al., 23 Aug 2025).
A central conclusion of the paper is that the tested pretrained cross-lingual LLMs do not appear to have acquired Quebec-French linguistic judgment capabilities during pretraining (Beauchemin et al., 23 Aug 2025). The supporting evidence is that zero-shot LLM accuracy is usually below the trivial baseline, MCC is near zero or negative, the multilingual fine-tuned XLM-R model remains far behind the monolingual French baseline, and OOD performance is difficult even for fine-tuned models (Beauchemin et al., 23 Aug 2025).
The paper also states several limitations. QFrCoLA is based on official normative grammar, not broad usage variation; the OQLF source may reflect a Quebec governmental language-policy perspective; the corpus is category-skewed, especially toward morphology; and the OOD split is especially hard because some France-French norms differ from Quebec-French ones (Beauchemin et al., 23 Aug 2025). In addition, there is no reported inter-annotator agreement study (Beauchemin et al., 23 Aug 2025).
Within multilingual acceptability research, QFrCoLA’s significance lies in extending CoLA-style evaluation to French while foregrounding Quebec French specifically. It enables direct comparison with existing resources such as CoLA, DaLAJ, ITACoLA, RuCoLA, CoLAC, NoCoLA, and JCoLA, and it provides evidence that fine-tuned monolingual Transformers remain substantially stronger than zero-shot LLMs for this type of grammatical judgment in French (Beauchemin et al., 23 Aug 2025). The dataset therefore occupies a dual role: as a language-specific benchmark for Quebec French and as a comparative instrument within the broader study of multilingual linguistic acceptability.