Scoreq_ref: Self-Supervised ASR Metric

• Scoreq_ref is a self-supervised, reference-less metric that evaluates ASR output quality using a multilingual MiniLMv2-based Transformer architecture and contrastive learning.
• The metric employs a pairwise ranking loss on ASR hypothesis pairs generated across varied compression settings, directly relating its scalar output to word error rate.
• Scoreq_ref enables efficient model selection and ensemble reranking, achieving notable WER reductions and rapid inference times in diverse ASR applications.

Scoreq_ref

Scoreq_ref is a reference-less, self-supervised metric designed to evaluate the quality of automatic speech recognition (ASR) system outputs without access to ground-truth transcriptions. By leveraging a multilingual, Transformer-based LLM fine-tuned with a contrastive learning objective, Scoreq_ref produces a scalar score for a given ASR hypothesis that correlates strongly with word error rate (WER), allowing comparison, ranking, and model selection in both research and applied pipelines (Yuksel et al., 2023).

1. Model Architecture and Feature Extraction

Scoreq_ref employs the MiniLMv2 backbone—a 117M parameter Transformer distilled from XLM-RoBERTa-Large—capable of processing 100+ languages. The core architecture includes:

• Pre-trained Transformer Encoder: 6-layer, 12-head, 384-dim MiniLMv2.
• Projection Head: A two-layer feedforward “dense encoder” mapping the [CLS]-vector (dimension 384) to a 128-dim hidden layer (with dropout=0.1 and GeLU/ReLU), then to a scalar, yielding a final logit.
• Score Output: At inference, the scalar output is passed through a sigmoid: $$Scoreq\_ref(h) = \sigma(g(\textrm{LM}(h))) \in (0,1)$$ for ASR hypothesis $h$.

This architecture is both lightweight (fast batch inference) and flexible, enabling large-scale evaluations or near-real-time reranking of ASR candidates.

2. Self-Supervised Contrastive Learning Objective

Training is conducted without ground-truth transcriptions, using a contrastive, pairwise ranking loss derived from ASR system outputs themselves.

• Surrogate Quality Ordering: Multiple outputs are obtained for each utterance from OpenAI’s Whisper ASR at six compression settings, $V_1(u),...,V_6(u)$, with $V_1$ the highest quality.
• Pairwise Construction: All pairs $(V_c,V_{c'})$ for $c < c'$ are labeled as “$V_c$ is higher quality than $V_{c'}$.”
• Loss Function: For each ordered pair,

$$L_{c,c'} = w_{c,c'} \cdot \log(1 + e^{-(f_\theta(V_c) - f_\theta(V_{c'}))})$$

where $f_\theta(\cdot)$ is the end-to-end model score, and $h$0 is a positive weight proportional to the WER difference between the paired hypotheses. The overall loss sums $h$1 over all valid pairs and utterances. This InfoNCE-style binary ranking loss directly optimizes the model to distinguish better and worse hypotheses by relative quality (Yuksel et al., 2023).

3. Training Corpus and Procedure

Scoreq_ref was trained on 134 hours of conversational speech in five languages (English, French, Spanish, Portuguese, German) from CMU-MOSEI and MOSEAS:

• Training Data: Each utterance yields 15 unique hypothesis pairs (from six outputs), generating $h$20.8M pairs.
• Split: 80% training / 20% validation, stratified by speaker and language.
• Optimization: Adafactor with $h$3 learning rate, no curriculum, early stopping on validation pairwise ranking accuracy ($h$4 typically achieved).

The model is fully text-based, requiring no reference alignments or audio at inference, facilitating easy deployment in cloud or edge ASR settings.

4. Metric Definition and Application

For any ASR hypothesis $h$5: $h$6 Higher $h$7 implies higher estimated transcription quality.

Model selection or ensemble routing: For a set of $h$8 hypotheses $h$9 on the same utterance, the best is predicted as $V_1(u),...,V_6(u)$0.

Evaluation protocols:

• Hypothesis Ranking Correlation: Pairwise correlation (Pearson/Spearman/Kendall) with true WER for $V_1(u),...,V_6(u)$1 on each utterance.
• Absolute-WER Correlation: Correlation between $V_1(u),...,V_6(u)$2 and reference-based WER across hypotheses and utterances.

Typical results on Common Voice English are: Pearson 0.56 (ranking), 0.42 (absolute), always outperforming XLM-R perplexity (by 2–3x) (Yuksel et al., 2023).

5. Empirical Performance and Limitations

Empirical findings and deployment notes include:

• Ranking and Ensembling Gains: Scoreq_ref-driven selection yields substantial WER reduction vs. best individual commercial engines (e.g., 12% English, 7% French/Spanish).
• Throughput: Single forward pass per hypothesis; 5–20 ms/100 tokens on a modern CPU/GPU.
• Training Cost: $V_1(u),...,V_6(u)$34 hours for 0.8M pairs on a V100 GPU.
• Text Modality Only: Does not address acoustic or phonetic confounds not reflected in text, such as audio quality, accents, or speaker characteristics.
• Language and Domain Scope: Trained on 5 languages, performance on others may degrade; Whisper error spectra may differ from other ASR systems.

6. Extensions and Best Practices

For robust deployment and optimal results:

• Domain Adaptation: Further fine-tuning or calibration on in-domain hypothesis pairs can address domain shift.
• Hybridization: Combine Scoreq_ref with simple audio-based confidence measures to mitigate text-only weaknesses.
• Complementary Use: Validate any absolute-quality claims (not just ranking) with human or reference-based evaluation, especially for out-of-distribution data.

7. Relationship to Other Reference-Free Metrics

Scoreq_ref is a specialized, self-supervised reference-less metric for ASR transcript quality, distinct from reference-less metrics in other modalities (NLG, QA, sequence quality compression, etc.). Unlike methods that exploit sequence redundancy for lossy compression (e.g., BWT/LCP-based quality smoothing in genomics (Janin et al., 2013)), Scoreq_ref is optimized specifically for ASR transcript ranking, leveraging a fully multilingual LLM and a pairwise ranking loss tailored to the available, imperfect ASR system outputs (Yuksel et al., 2023).