---
title: 'Scoreq_ref: Self-Supervised ASR Metric'
url: https://www.emergentmind.com/topics/scoreq_ref
type: topic
---

# Scoreq_ref: Self-Supervised ASR Metric

Scoreq_ref

Scoreq_ref is a reference-less, self-supervised metric designed to evaluate the quality of automatic speech recognition (ASR) system outputs without access to ground-truth transcriptions. By leveraging a multilingual, Transformer-based language model fine-tuned with a contrastive learning objective, Scoreq_ref produces a scalar score for a given ASR hypothesis that correlates strongly with word error rate (WER), allowing comparison, ranking, and model selection in both research and applied pipelines [2306.13114].

## 1. Model Architecture and Feature Extraction

Scoreq_ref employs the MiniLMv2 backbone—a 117M parameter Transformer distilled from XLM-RoBERTa-Large—capable of processing 100+ languages. The core architecture includes:
- **Pre-trained Transformer Encoder:** 6-layer, 12-head, 384-dim MiniLMv2.
- **Projection Head:** A two-layer feedforward “dense encoder” mapping the [CLS]-vector (dimension 384) to a 128-dim hidden layer (with dropout=0.1 and GeLU/ReLU), then to a scalar, yielding a final logit.
- **Score Output:** At inference, the scalar output is passed through a sigmoid: $Scoreq\_ref(h) = \sigma(g(\textrm{LM}(h))) \in (0,1)$ for ASR hypothesis $h$.

This architecture is both lightweight (fast batch inference) and flexible, enabling large-scale evaluations or near-real-time reranking of ASR candidates.

## 2. Self-Supervised Contrastive Learning Objective

Training is conducted without ground-truth transcriptions, using a contrastive, pairwise ranking loss derived from ASR system outputs themselves.
- **Surrogate Quality Ordering:** Multiple outputs are obtained for each utterance from OpenAI’s Whisper ASR at six compression settings, $V_1(u),...,V_6(u)$, with $V_1$ the highest quality.
- **Pairwise Construction:** All pairs $(V_c,V_{c'})$ for $c < c'$ are labeled as “$V_c$ is higher quality than $V_{c'}$.”
- **Loss Function:** For each ordered pair,
  \[
  L_{c,c'} = w_{c,c'} \cdot \log(1 + e^{-(f_\theta(V_c) - f_\theta(V_{c'}))})
  \]
where $f_\theta(\cdot)$ is the end-to-end model score, and $w_{c,c'}$ is a positive weight proportional to the WER difference between the paired hypotheses. The overall loss sums $L_{c,c'}$ over all valid pairs and utterances. This InfoNCE-style binary ranking loss directly optimizes the model to distinguish better and worse hypotheses by relative quality [2306.13114].

## 3. Training Corpus and Procedure

Scoreq_ref was trained on 134 hours of conversational speech in five languages (English, French, Spanish, Portuguese, German) from CMU-MOSEI and MOSEAS:
- **Training Data:** Each utterance yields 15 unique hypothesis pairs (from six outputs), generating $\sim$0.8M pairs.
- **Split:** 80% training / 20% validation, stratified by speaker and language.
- **Optimization:** Adafactor with $1\times10^{-5}$ learning rate, no curriculum, early stopping on validation pairwise ranking accuracy ($\sim77\%$ typically achieved).

The model is fully text-based, requiring no reference alignments or audio at inference, facilitating easy deployment in cloud or edge ASR settings.

## 4. Metric Definition and Application

For any ASR hypothesis $h$:
\[
Scoreq\_ref(h) = \sigma(s(h)) = \sigma ( g ( LM(h) ) )
\]
Higher $Scoreq\_ref$ implies higher estimated transcription quality.

**Model selection or ensemble routing:** For a set of $N$ hypotheses $\{h_1, ..., h_N\}$ on the same utterance, the best is predicted as $h_{i^*} = \arg\max_i Scoreq\_ref(h_i)$.

**Evaluation protocols:**
- **Hypothesis Ranking Correlation:** Pairwise correlation (Pearson/Spearman/Kendall) with true WER for $\{h_{i}\}$ on each utterance.
- **Absolute-WER Correlation:** Correlation between $Scoreq\_ref(h)$ and reference-based WER across hypotheses and utterances.

Typical results on Common Voice English are: Pearson 0.56 (ranking), 0.42 (absolute), always outperforming XLM-R perplexity (by 2–3x) [2306.13114].

## 5. Empirical Performance and Limitations

Empirical findings and deployment notes include:
- **Ranking and Ensembling Gains:** Scoreq_ref-driven selection yields substantial WER reduction vs. best individual commercial engines (e.g., 12% English, 7% French/Spanish).
- **Throughput:** Single forward pass per hypothesis; 5–20 ms/100 tokens on a modern CPU/GPU.
- **Training Cost:** $\sim$4 hours for 0.8M pairs on a V100 GPU.
- **Text Modality Only:** Does not address acoustic or phonetic confounds not reflected in text, such as audio quality, accents, or speaker characteristics.
- **Language and Domain Scope:** Trained on 5 languages, performance on others may degrade; Whisper error spectra may differ from other ASR systems.

## 6. Extensions and Best Practices

For robust deployment and optimal results:
- **Domain Adaptation:** Further fine-tuning or calibration on in-domain hypothesis pairs can address domain shift.
- **Hybridization:** Combine Scoreq_ref with simple audio-based confidence measures to mitigate text-only weaknesses.
- **Complementary Use:** Validate any absolute-quality claims (not just ranking) with human or reference-based evaluation, especially for out-of-distribution data.

## 7. Relationship to Other Reference-Free Metrics

Scoreq_ref is a specialized, self-supervised reference-less metric for ASR transcript quality, distinct from reference-less metrics in other modalities (NLG, QA, sequence quality compression, etc.). Unlike methods that exploit sequence redundancy for lossy compression (e.g., BWT/LCP-based quality smoothing in genomics [1305.0159]), Scoreq_ref is optimized specifically for ASR transcript ranking, leveraging a fully multilingual language model and a pairwise ranking loss tailored to the available, imperfect ASR system outputs [2306.13114].

Source: https://www.emergentmind.com/topics/scoreq_ref