QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs

Abstract: In this paper, we introduce the Quebec-French Benchmark of Linguistic Minimal Pairs (QFrBLiMP), a corpus designed to evaluate the linguistic knowledge of LLMs on prominent grammatical phenomena in Quebec-French. QFrBLiMP consists of 1,761 minimal pairs annotated with 20 linguistic phenomena. Specifically, these minimal pairs have been created by manually modifying sentences extracted from an official online resource maintained by a Qu\'ebec government institution. Each pair is annotated by twelve Quebec-French native speakers, who select the sentence they feel is grammatical amongst the two. These annotations are used to compare the competency of LLMs with that of humans. We evaluate different LLMs on QFrBLiMP and MultiBLiMP-Fr by observing the rate of higher probabilities assigned to the sentences of each minimal pair for each category. We find that while grammatical competence scales with model size, a clear hierarchy of difficulty emerges. All benchmarked models consistently fail on phenomena requiring deep semantic understanding, revealing a critical limitation and a significant gap compared to human performance on these specific tasks.

• The paper introduces a benchmark of 1,761 annotated Quebec-French minimal pairs to fill gaps in prescriptive grammatical evaluations.
• The methodology involves human annotations from native speakers and evaluates 77 LLMs across 20 distinct grammatical phenomena.
• The results reveal a strong log-linear correlation between model size and accuracy, with models struggling on deep semantic and complex syntactic tasks.

QFrBLiMP: A Quebec-French Benchmark of Linguistic Minimal Pairs

Introduction and Motivation

The QFrBLiMP benchmark addresses a critical gap in the evaluation of LLMs for Quebec-French, a variety of French with distinct grammatical, lexical, and prescriptive norms. Existing minimal pair benchmarks such as BLiMP (English), JBLiMP (Japanese), and MultiBLiMP (multilingual) do not capture the unique linguistic phenomena of Quebec-French, particularly those codified by local institutions. QFrBLiMP provides a corpus of 1,761 minimal pairs, each annotated for one of 20 grammatical phenomena, and includes human acceptability judgments from twelve native Quebec-French speakers. The benchmark is constructed from sentences extracted and minimally edited from the Banque de dépannage linguistique (BDL), an official resource maintained by the Office québécois de la langue française (OQLF).

Figure 1: Snipped of the translated BDL article for present participles including'' andexcluding'', illustrating the prescriptive source for minimal pairs.

Corpus Construction and Annotation Protocol

QFrBLiMP's minimal pairs are derived from 2,667 BDL articles, yielding 25,153 candidate sentences. Each pair consists of a grammatical and an ungrammatical variant, with errors targeting a specific phenomenon (e.g., past participle agreement, clitic placement, syntactic islands, lexical selection). The annotation process involved twelve undergraduate native speakers, who selected the grammatical sentence in each pair using a randomized presentation order to mitigate bias.

Figure 2: The Prodigy annotation interface (in French) used by the annotators to evaluate the minimal pairs.

Inter-annotator agreement was measured using the Worker Agreement With Aggregate (WAWA) coefficient, with most phenomena exceeding 90% agreement, though some (e.g., parasitic dependence, orphan prepositions) exhibited lower consensus, reflecting inherent linguistic ambiguity or annotation guideline limitations.

Experimental Design and Evaluation Methodology

The benchmark evaluates 77 open-source LLMs spanning a wide range of parameter counts (from sub-billion to 72B), architectural families, and specialization (general, French-focused, reasoning-tuned, instruction-tuned). Models are assessed by their ability to assign higher probability (lower perplexity) to the grammatical sentence in each minimal pair, following the established methodology for minimal pair evaluation. Human annotator majority vote serves as the gold standard, with random selection as a lower baseline.

Key Results: Scaling Laws and Phenomenon-Specific Performance

Model Scale and Aggregate Accuracy

A strong log-linear correlation is observed between model size and overall grammatical accuracy on QFrBLiMP, with the largest models approaching but not surpassing human performance (88–90% accuracy). Smaller models exhibit greater variance, suggesting that architectural and data quality factors are more influential at lower scales.

Hierarchy of Linguistic Phenomena

• Mastered Phenomena: Regular, frequent rules (e.g., -é/-er distinction, clitic placement, verb inflection) are robustly learned, with top models achieving near-perfect scores and sometimes exceeding human consistency.
• Challenging Phenomena: Complex syntactic rules (e.g., past participle agreement, syntactic islands) are learned imperfectly, with models struggling on edge cases and long-distance dependencies.
• Unsolved Phenomena: Deep semantic tasks (lexical semantics, orphaned prepositions) remain unsolved, with all models performing only marginally above random and lagging human judgment by 20–30%. Scaling does not close this gap, indicating a fundamental limitation in current architectures.

Specialization and Instruction-Tuning

French-specialized and reasoning-tuned models do not consistently outperform general models of similar scale. Instruction-tuning can degrade formal grammatical competence, likely due to catastrophic forgetting or misalignment between tuning data and evaluation language.

Benchmark Comparison

Models consistently score higher on QFrBLiMP than on MultiBLiMP-Fr, likely due to QFrBLiMP's focused, prescriptively curated minimal pairs and broader coverage of phenomena. This suggests that QFrBLiMP provides a more precise measure of grammatical competence for Quebec-French.

Practical and Theoretical Implications

QFrBLiMP establishes a high-quality, linguistically attested resource for evaluating LLMs on Quebec-French, enabling fine-grained analysis of grammatical knowledge and its scaling properties. The persistent failure of models on deep semantic phenomena highlights the need for architectures and training regimes that go beyond surface-level statistical regularities. The observed degradation from instruction-tuning underscores the importance of alignment strategies that preserve foundational linguistic competence.

The benchmark's reliance on prescriptive, didactic examples introduces a potential domain shift relative to naturally occurring text, and the public availability of BDL raises concerns about test data contamination in LLM pretraining. These limitations must be considered when interpreting results and designing future benchmarks.

Future Directions

• Corpus Expansion: Develop training corpora specifically optimized for Quebec-French to test the impact of language specialization.
• Evaluation Methods: Explore alternative metrics (e.g., targeted probing, semantic similarity) for phenomena where perplexity-based evaluation is insufficient.
• Alignment Research: Investigate mechanisms to prevent catastrophic forgetting of grammatical knowledge during instruction-tuning.
• Semantic Competence: Pursue model architectures and training objectives that incentivize deeper semantic and pragmatic understanding.

Conclusion

QFrBLiMP provides a rigorous, human-annotated benchmark for Quebec-French grammatical competence, revealing both the strengths and limitations of current LLMs. While scaling yields substantial improvements on regular syntactic and morphological phenomena, deep semantic understanding remains elusive. The resource will facilitate future research on language-specific model development, evaluation methodology, and the theoretical boundaries of neural language modeling.