---
title: 'CommissionsQC: Québec ASR Benchmark'
url: https://www.emergentmind.com/topics/commissionsqc
type: topic
---

# CommissionsQC: Québec ASR Benchmark

Searching arXiv for recent papers on CommissionsQC and Québec French ASR benchmarks.
CommissionsQC is a speech corpus of spontaneous conversations recorded during public inquiries recently held in Quebec, Canada, and serves as the basis of a benchmark and evaluation pipeline for automatic speech recognition (ASR) on Québec French (QF) under realistic conditions [2508.21193]. In the reported benchmark, CommissionsQC comprises 782 training hours and 27 hours of development and test data, and is used to evaluate large pretrained multilingual models, French-tuned variants, cloud APIs, multimodal models, and models trained entirely on CommissionsQC [2508.21193]. The benchmark is explicitly motivated by the observation that published results on FLEURS or CommonVoice are not good predictors of performance on CommissionsQC, either in absolute error rates or in model ordering [2508.21193].

## 1. Corpus definition and linguistic setting

CommissionsQC is described as a corpus of spontaneous, conversational Québec French recorded during Quebec’s recent public inquiries [2508.21193]. The corpus is situated in a regional-language and realistic-deployment setting rather than a clean read-speech setting. The benchmark paper contrasts it with FLEURS and CommonVoice, which are characterized as European French reading-style speech [2508.21193].

The corpus is reported to contain 782 training hours and 27 hours of development and test data [2508.21193]. It is split into Bast and Charb, and these partitions are described as gender-balanced and composed of distinct speakers [2508.21193]. Training, validation, and test partitions are constructed to avoid speaker overlap and ensure robust evaluation [2508.21193].

The recording environment is an essential part of the dataset definition. CommissionsQC contains realistic courtroom conditions, including reverberation, background noise, and natural speaker variability [2508.21193]. A plausible implication is that the corpus is intended not merely as a linguistic resource for Québec French, but as a stress test for ASR systems under deployment-like acoustic conditions.

## 2. Benchmarking pipeline and evaluation design

The benchmark built around CommissionsQC is described as a unified benchmarking and evaluation pipeline [2508.21193]. The pipeline supports numerous ASR models through common pre-processing, post-processing, transcription normalization, and metric computation, thereby enabling direct comparison across model families [2508.21193].

Two normalization schemes are reported. The first, called *whisper*, replicates Whisper benchmarks. The second, called *basic*, adds various French-specific text processing, including number normalization and compound word splitting [2508.21193]. The paper states that the pipeline automatically computes core metrics for every model on the same data splits, which is intended to support robust comparison [2508.21193].

The evaluation covers 24 recent, publicly available models [2508.21193]. The tested categories include foundation models such as Whisper-large-v3 and the mms-1b series; French-tuned variants such as whisper-large-v3-french and fastconformer_fr; cloud APIs including AWS, Azure, Google Chirp, and GPT-4o; multimodal models including Phi-4-multimodal-instruct, Gemini-2.0, Qwen2-Audio-7B, and SeamlessM4T; and from-scratch models trained entirely on CommissionsQC [2508.21193]. Model sizes range from Whisper Tiny, at approximately 74M parameters, to multimodal LLMs of 8B parameters and above [2508.21193].

This design makes CommissionsQC a benchmark substrate rather than only a passive corpus. It is not just a collection of recordings, but the anchor for a standardized comparison framework spanning open models, cloud systems, and multimodal architectures [2508.21193].

## 3. Evaluation metrics and normalization conventions

The benchmark evaluates models using Word Error Rate (WER), Character Error Rate (CER), Bert F1 as a semantic-accuracy measure, and Real-Time Factor (RTF) as a speed measure [2508.21193]. WER is defined through the Levenshtein edit distance,

\[
\mathrm{WER} = \frac{S+D+I}{N}
\]

with \(S\) the number of substitutions, \(D\) deletions, \(I\) insertions, and \(N\) the number of words in the reference [2508.21193]. CER is computed similarly at the character level [2508.21193].

Bert F1 is reported as a semantic similarity score between model transcript and reference, computed using multilingual BERT embeddings, with values ranging from 0 to 100 and higher values indicating closer semantic fidelity [2508.21193]. RTF is defined as

\[
\mathrm{RTF} = 100 \times \frac{\text{processing time}}{\text{audio duration}}
\]

so that, for example, 25% RTF corresponds to processing at four times real time [2508.21193].

The benchmark also discusses deletion and insertion behavior. Deletion/Insertion Ratio (DIR) is used to track this effect, and the best models are reported as being near 2, while insertion-prone systems exhibit lower values [2508.21193]. The paper further notes that smaller Whisper models have more insertions and that some models are prone to abnormal deletion, especially GPT-4o, which skips segments [2508.21193].

Normalization materially affects the reported error rates. Using the more French-tuned *basic* normalization reduced WER by approximately 1% absolute compared with *whisper* normalization [2508.21193]. This suggests that benchmark comparisons for regional French can depend not only on model architecture and training data, but also on evaluation protocol details specific to French orthographic and lexical normalization.

## 4. Empirical performance on CommissionsQC

On CommissionsQC, the best reported model performance from a from-scratch baseline is achieved by *espnet_transformer*, with 8.2% WER, 3.8% CER, 95.8 Bert F1, and 36% RTF [2508.21193]. Among off-the-shelf and foundation models, *Whisper-large-v3-turbo* is reported at 8.2% WER, 95.4 Bert F1, and 6% RTF, while *Whisper-large-v3* reaches 8.4% WER, 95.8 Bert F1, and 14% RTF [2508.21193]. The French-tuned *w3-large-v3-fr-d16* is reported at 9.2% WER and 95.8 Bert F1, and the cloud API *Aws-fr-ca* at 10.3% WER [2508.21193].

The paper states that many models cluster around 14% WER, 0.2 RTF, and 90–94 Bert F1 [2508.21193]. Multimodal models show heterogeneous behavior: *Phi-4-multimodal-instruct* and *Gemini-2.0* are reported at approximately 13–14% WER, whereas *Qwen2-Audio-7B* and *Seamless-M4T* perform more poorly, with WER above 30% [2508.21193]. Prompt wording is reported to affect WER by approximately 10% for multimodal models, indicating strong prompt sensitivity in this setting [2508.21193].

Cloud systems are also benchmarked. *gpt-4o-transcribe* is reported at 14.2% WER, with 15.2% for male and 13.1% for female speakers, but the paper notes high deletion rates due to skipped segments and estimates that if those are excluded the WER would be approximately 3% [2508.21193]. *aws-fr-CA* is described as the slowest ASR API at 84% RTF, while *google-chirp* reaches 15.4% WER [2508.21193].

The reported speed-accuracy frontier is therefore nontrivial. Some of the strongest-performing models are also relatively fast, notably *Whisper-large-v3-turbo* at 6% RTF and 8.2% WER [2508.21193]. A plausible implication is that CommissionsQC supports not only accuracy-oriented benchmarking but also deployment-oriented trade-off analysis.

## 5. Relation to standard French benchmarks

A central conclusion of the benchmark is that standard French benchmarks do not transfer reliably to CommissionsQC [2508.21193]. The paper states that word error rates on CommissionsQC are much higher than on FLEURS or CommonVoice for the same models, reflecting spontaneity, environmental noise, and accent [2508.21193].

More importantly, model rankings diverge across datasets. *Whisper-large-v3* is reported as best on CommissionsQC but worst on CommonVoice, while GPT-4o is best on FLEURS but only fifth-best on CommissionsQC [2508.21193]. The paper therefore concludes that benchmarks on European French do not predict performance on Québec French, neither in absolute scores nor in model ordering [2508.21193].

This result addresses a recurrent misconception in multilingual ASR evaluation: that high performance on broadly used French benchmarks can be taken as evidence of robustness to regional French varieties. The CommissionsQC results do not support that assumption [2508.21193]. Instead, they indicate that in-domain evaluation on regional, spontaneous, and noisy data is necessary when building systems for Québec French.

## 6. Bias, limitations, and practical significance

The benchmark reports a slight gender difference, with somewhat better WER for female speakers: the best models achieve approximately 8.0% WER for female speakers versus 8.4% for male speakers, corresponding to a 5% relative difference [2508.21193]. The paper adds that there is no clear model-type trend regarding gender [2508.21193].

The scope of CommissionsQC is explicitly limited. It is realistic, but it covers only public inquiry speech and does not include informal, youth, rural, or phone speech [2508.21193]. The paper further notes that spontaneity, accent, and noise co-occur in the corpus, so error sources cannot be uniquely separated [2508.21193]. Prompt selection for multimodal systems introduces some subjectivity, and possible data leakage is said to be difficult to entirely rule out, though unlikely [2508.21193].

Within those limits, the benchmark draws several practical conclusions. It recommends that practitioners building ASR for Québec French or similar regional French varieties should not rely on published FLEURS or CommonVoice scores and should expect WER to be two to five times higher, with model selection based on in-domain evaluation when possible [2508.21193]. It also reports that the best zero-shot performance comes from the largest multilingual models, especially Whisper-Large-V3, while in-domain training or fine-tuning can still outperform them by a small margin [2508.21193].

Taken together, these findings position CommissionsQC as a specialized evaluation resource for speech technology in Québec French. Its significance lies less in scale alone than in the combination of spontaneous speech, public-inquiry acoustics, regional linguistic variation, and standardized benchmarking, all of which expose failure modes that are obscured by cleaner European French benchmarks [2508.21193].

Source: https://www.emergentmind.com/topics/commissionsqc