- The paper introduces benchmark datasets QFrCoRE and QFrCoRT to evaluate LLMs on Quebec-French idioms with manually compiled definitions and LLM-generated distractors.
- The experimental results reveal a significant performance gap between proprietary and open-source models, with minimal impact from model size or LLM specialization.
- The study emphasizes the need for diverse, culturally enriched training data to overcome dialect comprehension gaps in language models.
A Set of Quebec-French Corpus of Regional Expressions and Terms
Introduction to the Corpus
The paper introduces a unique pair of benchmark datasets aimed at assessing LLMs on understanding idiomatic expressions specific to the Quebec-French dialect. These benchmarks, named QFrCoRE and QFrCoRT, are designed to probe the quality of a model's dialect comprehension by leveraging idiomatic understanding. QFrCoRE includes 4,633 idiomatic phrases, while QFrCoRT focuses on 171 regional idioms. This research is pivotal in addressing the dialect gap, which arises due to the overrepresentation of certain dialects in training data, often resulting in suboptimal performance for underrepresented linguistic variants.
Methodology
The construction of QFrCoRE and QFrCoRT involved manually compiling Quebecois idioms and their definitions from various dictionaries and online resources. The idioms are coupled with distractors, which are plausible but incorrect definitions generated by an LLM to challenge the tested models' ability to discern the correct meanings.
The datasets were then used in zero-shot evaluation settings across 94 LLMs to determine the proficiency of these models in Quebecois idioms. The performance of the models on these tasks was measured using accuracy in identifying the correct definitions from a list of options.
Experimental Setup
The authors conducted an extensive evaluation involving proprietary models accessible via APIs, such as GPT-4 and Claude series from Anthropic, alongside open-source models like Qwen and LLaMa series. These models span a range of sizes and capabilities, from under one billion parameters to more than 100 billion, with some fine-tuned specifically in French to ascertain the impact of linguistic specialization.
Standardized prompts were used to interface with the models, where the system was tasked with selecting the correct definition from multiple-choice options. Performance was then plotted to visually represent the models' proficiencies.
Results and Interpretation
The results demonstrated a stark performance gap between proprietary and open-source models, with proprietary models generally outperforming open-source ones, possibly due to broader training datasets that likely included dialectal content. Notably, models specialized in French often performed poorly, likely because their training data favored the prestige dialect from France without sufficient exposure to Quebecois expressions.
Additionally, no significant correlation was observed between model size and performance, nor between specialization in reasoning tasks and dialect understanding. This highlights that dialect comprehension, particularly Quebecois idioms, requires exposure to culturally specific data rather than increased model complexity or size.
Implications and Future Work
The findings underscore the necessity for diversified training data that encompasses various dialects. Furthermore, the corpora provide a benchmark to measure and mitigate the dialect gap in LLMs. Future work will extend these benchmarks to encompass additional dialects and conduct human evaluation studies to establish performance baselines across linguistic varieties.
In conclusion, the paper contributes to dialect-specific NLU (Natural Language Understanding) benchmarks, offering insights into the limitations of current models and the importance of linguistic and cultural diversity in model training.