CommissionsQC: Québec ASR Benchmark

• CommissionsQC is a corpus of spontaneous Québec French recordings from public inquiries, offering a realistic dataset for ASR evaluation.
• The benchmark comprises 782 training hours and 27 hours of validation/test data with strict speaker partitioning to ensure reliable model comparisons.
• Evaluation metrics like WER, CER, Bert F1, and RTF reveal that traditional French benchmarks may not predict performance on regional, spontaneous speech.

Searching arXiv for papers on CommissionsQC and Québec French ASR benchmarks. CommissionsQC is a speech corpus of spontaneous conversations recorded during public inquiries recently held in Quebec, Canada, and serves as the basis of a benchmark and evaluation pipeline for automatic speech recognition (ASR) on Québec French (QF) under realistic conditions (Serrand et al., 28 Aug 2025). In the reported benchmark, CommissionsQC comprises 782 training hours and 27 hours of development and test data, and is used to evaluate large pretrained multilingual models, French-tuned variants, cloud APIs, multimodal models, and models trained entirely on CommissionsQC (Serrand et al., 28 Aug 2025). The benchmark is explicitly motivated by the observation that published results on FLEURS or CommonVoice are not good predictors of performance on CommissionsQC, either in absolute error rates or in model ordering (Serrand et al., 28 Aug 2025).

1. Corpus definition and linguistic setting

CommissionsQC is described as a corpus of spontaneous, conversational Québec French recorded during Quebec’s recent public inquiries (Serrand et al., 28 Aug 2025). The corpus is situated in a regional-language and realistic-deployment setting rather than a clean read-speech setting. The benchmark paper contrasts it with FLEURS and CommonVoice, which are characterized as European French reading-style speech (Serrand et al., 28 Aug 2025).

The corpus is reported to contain 782 training hours and 27 hours of development and test data (Serrand et al., 28 Aug 2025). It is split into Bast and Charb, and these partitions are described as gender-balanced and composed of distinct speakers (Serrand et al., 28 Aug 2025). Training, validation, and test partitions are constructed to avoid speaker overlap and ensure robust evaluation (Serrand et al., 28 Aug 2025).

The recording environment is an essential part of the dataset definition. CommissionsQC contains realistic courtroom conditions, including reverberation, background noise, and natural speaker variability (Serrand et al., 28 Aug 2025). A plausible implication is that the corpus is intended not merely as a linguistic resource for Québec French, but as a stress test for ASR systems under deployment-like acoustic conditions.

2. Benchmarking pipeline and evaluation design

The benchmark built around CommissionsQC is described as a unified benchmarking and evaluation pipeline (Serrand et al., 28 Aug 2025). The pipeline supports numerous ASR models through common pre-processing, post-processing, transcription normalization, and metric computation, thereby enabling direct comparison across model families (Serrand et al., 28 Aug 2025).

Two normalization schemes are reported. The first, called whisper, replicates Whisper benchmarks. The second, called basic, adds various French-specific text processing, including number normalization and compound word splitting (Serrand et al., 28 Aug 2025). The paper states that the pipeline automatically computes core metrics for every model on the same data splits, which is intended to support robust comparison (Serrand et al., 28 Aug 2025).

The evaluation covers 24 recent, publicly available models (Serrand et al., 28 Aug 2025). The tested categories include foundation models such as Whisper-large-v3 and the mms-1b series; French-tuned variants such as whisper-large-v3-french and fastconformer_fr; cloud APIs including AWS, Azure, Google Chirp, and GPT-4o; multimodal models including Phi-4-multimodal-instruct, Gemini-2.0, Qwen2-Audio-7B, and SeamlessM4T; and from-scratch models trained entirely on CommissionsQC (Serrand et al., 28 Aug 2025). Model sizes range from Whisper Tiny, at approximately 74M parameters, to multimodal LLMs of 8B parameters and above (Serrand et al., 28 Aug 2025).

This design makes CommissionsQC a benchmark substrate rather than only a passive corpus. It is not just a collection of recordings, but the anchor for a standardized comparison framework spanning open models, cloud systems, and multimodal architectures (Serrand et al., 28 Aug 2025).

3. Evaluation metrics and normalization conventions

The benchmark evaluates models using Word Error Rate (WER), Character Error Rate (CER), Bert F1 as a semantic-accuracy measure, and Real-Time Factor (RTF) as a speed measure (Serrand et al., 28 Aug 2025). WER is defined through the Levenshtein edit distance,

$\mathrm{WER} = \frac{S+D+I}{N}$

with $S$ the number of substitutions, $D$ deletions, $I$ insertions, and $N$ the number of words in the reference (Serrand et al., 28 Aug 2025). CER is computed similarly at the character level (Serrand et al., 28 Aug 2025).

Bert F1 is reported as a semantic similarity score between model transcript and reference, computed using multilingual BERT embeddings, with values ranging from 0 to 100 and higher values indicating closer semantic fidelity (Serrand et al., 28 Aug 2025). RTF is defined as

$$\mathrm{RTF} = 100 \times \frac{\text{processing time}}{\text{audio duration}}$$

so that, for example, 25% RTF corresponds to processing at four times real time (Serrand et al., 28 Aug 2025).

The benchmark also discusses deletion and insertion behavior. Deletion/Insertion Ratio (DIR) is used to track this effect, and the best models are reported as being near 2, while insertion-prone systems exhibit lower values (Serrand et al., 28 Aug 2025). The paper further notes that smaller Whisper models have more insertions and that some models are prone to abnormal deletion, especially GPT-4o, which skips segments (Serrand et al., 28 Aug 2025).

Normalization materially affects the reported error rates. Using the more French-tuned basic normalization reduced WER by approximately 1% absolute compared with whisper normalization (Serrand et al., 28 Aug 2025). This suggests that benchmark comparisons for regional French can depend not only on model architecture and training data, but also on evaluation protocol details specific to French orthographic and lexical normalization.

4. Empirical performance on CommissionsQC

On CommissionsQC, the best reported model performance from a from-scratch baseline is achieved by espnet_transformer, with 8.2% WER, 3.8% CER, 95.8 Bert F1, and 36% RTF (Serrand et al., 28 Aug 2025). Among off-the-shelf and foundation models, Whisper-large-v3-turbo is reported at 8.2% WER, 95.4 Bert F1, and 6% RTF, while Whisper-large-v3 reaches 8.4% WER, 95.8 Bert F1, and 14% RTF (Serrand et al., 28 Aug 2025). The French-tuned w3-large-v3-fr-d16 is reported at 9.2% WER and 95.8 Bert F1, and the cloud API Aws-fr-ca at 10.3% WER (Serrand et al., 28 Aug 2025).

The paper states that many models cluster around 14% WER, 0.2 RTF, and 90–94 Bert F1 (Serrand et al., 28 Aug 2025). Multimodal models show heterogeneous behavior: Phi-4-multimodal-instruct and Gemini-2.0 are reported at approximately 13–14% WER, whereas Qwen2-Audio-7B and Seamless-M4T perform more poorly, with WER above 30% (Serrand et al., 28 Aug 2025). Prompt wording is reported to affect WER by approximately 10% for multimodal models, indicating strong prompt sensitivity in this setting (Serrand et al., 28 Aug 2025).

Cloud systems are also benchmarked. gpt-4o-transcribe is reported at 14.2% WER, with 15.2% for male and 13.1% for female speakers, but the paper notes high deletion rates due to skipped segments and estimates that if those are excluded the WER would be approximately 3% (Serrand et al., 28 Aug 2025). aws-fr-CA is described as the slowest ASR API at 84% RTF, while google-chirp reaches 15.4% WER (Serrand et al., 28 Aug 2025).

The reported speed-accuracy frontier is therefore nontrivial. Some of the strongest-performing models are also relatively fast, notably Whisper-large-v3-turbo at 6% RTF and 8.2% WER (Serrand et al., 28 Aug 2025). A plausible implication is that CommissionsQC supports not only accuracy-oriented benchmarking but also deployment-oriented trade-off analysis.

5. Relation to standard French benchmarks

A central conclusion of the benchmark is that standard French benchmarks do not transfer reliably to CommissionsQC (Serrand et al., 28 Aug 2025). The paper states that word error rates on CommissionsQC are much higher than on FLEURS or CommonVoice for the same models, reflecting spontaneity, environmental noise, and accent (Serrand et al., 28 Aug 2025).

More importantly, model rankings diverge across datasets. Whisper-large-v3 is reported as best on CommissionsQC but worst on CommonVoice, while GPT-4o is best on FLEURS but only fifth-best on CommissionsQC (Serrand et al., 28 Aug 2025). The paper therefore concludes that benchmarks on European French do not predict performance on Québec French, neither in absolute scores nor in model ordering (Serrand et al., 28 Aug 2025).

This result addresses a recurrent misconception in multilingual ASR evaluation: that high performance on broadly used French benchmarks can be taken as evidence of robustness to regional French varieties. The CommissionsQC results do not support that assumption (Serrand et al., 28 Aug 2025). Instead, they indicate that in-domain evaluation on regional, spontaneous, and noisy data is necessary when building systems for Québec French.

6. Bias, limitations, and practical significance

The benchmark reports a slight gender difference, with somewhat better WER for female speakers: the best models achieve approximately 8.0% WER for female speakers versus 8.4% for male speakers, corresponding to a 5% relative difference (Serrand et al., 28 Aug 2025). The paper adds that there is no clear model-type trend regarding gender (Serrand et al., 28 Aug 2025).

The scope of CommissionsQC is explicitly limited. It is realistic, but it covers only public inquiry speech and does not include informal, youth, rural, or phone speech (Serrand et al., 28 Aug 2025). The paper further notes that spontaneity, accent, and noise co-occur in the corpus, so error sources cannot be uniquely separated (Serrand et al., 28 Aug 2025). Prompt selection for multimodal systems introduces some subjectivity, and possible data leakage is said to be difficult to entirely rule out, though unlikely (Serrand et al., 28 Aug 2025).

Within those limits, the benchmark draws several practical conclusions. It recommends that practitioners building ASR for Québec French or similar regional French varieties should not rely on published FLEURS or CommonVoice scores and should expect WER to be two to five times higher, with model selection based on in-domain evaluation when possible (Serrand et al., 28 Aug 2025). It also reports that the best zero-shot performance comes from the largest multilingual models, especially Whisper-Large-V3, while in-domain training or fine-tuning can still outperform them by a small margin (Serrand et al., 28 Aug 2025).

Taken together, these findings position CommissionsQC as a specialized evaluation resource for speech technology in Québec French. Its significance lies less in scale alone than in the combination of spontaneous speech, public-inquiry acoustics, regional linguistic variation, and standardized benchmarking, all of which expose failure modes that are obscured by cleaner European French benchmarks (Serrand et al., 28 Aug 2025).