---
title: 'PrefSQA: MOS-Free Pairwise Speech Quality'
url: https://www.emergentmind.com/topics/prefsqa
type: topic
---

# PrefSQA: MOS-Free Pairwise Speech Quality

PrefSQA most directly denotes a framework for **MOS-free pairwise preference prediction in speech quality assessment**. Rather than regressing a scalar Mean Opinion Score (MOS) for a single utterance, it predicts which signal in a pair is preferred, using a dual-encoder architecture with **uncertainty-aware logits**, an **impairment attention head**, and a **non-matching-reference (NMR) comparison module** [2606.19597]. The framework is motivated by the claim that scalar MOS labels are sensitive to rater variability, listening-test protocol differences, label quantization, and corpus effects, whereas comparative judgments can be cleaner and more consistent. In the literature provided here, the same string also appears as an informal label for preference-aware personalized question answering, so the term has a limited but nontrivial terminological overlap across domains [2508.10028].

## 1. From MOS regression to pairwise preference prediction

Perceptual speech quality assessment has classically relied on **Mean Opinion Scores**, in which human raters assign scalar ratings and the scores are averaged. Modern MOS-based systems such as MOSNet, NISQA, and VoiceMOS-style models regress from acoustic features to these scalar labels. The central critique underlying PrefSQA is that MOS supervision is intrinsically noisy: different listeners interpret scales differently, listening tests vary in scale type and instructions, many protocols quantize judgments onto coarse discrete scales, and cross-corpus or cross-domain effects distort the mapping between perception and numeric labels [2606.19597].

PrefSQA reframes the task as **pairwise preference prediction**. The model receives a pair of utterances \((x,y)\) and predicts which one is preferred in perceived quality. This binary formulation is motivated by the observation that listeners are often more consistent when comparing stimuli than when assigning absolute scores. Comparative labels depend less on the exact score anchors used in a listening test, and they can preserve relative information even when small MOS differences would be dominated by annotation noise.

The paper is explicit that PrefSQA is **MOS-free** in its learning objective. Even when a dataset originally contains MOS values, those values are used only to derive pairwise labels during dataset construction; the model is never trained to predict MOS, and MOS is not used as a supervision signal at training or evaluation time. This keeps the method in preference space rather than treating pairwise prediction as a secondary view of scalar regression.

## 2. Core architecture and scoring mechanism

PrefSQA is built around a **dual-encoder semantic–acoustic backbone**. One branch is **wav2vec 2.0**, used as a semantic encoder, and the other is **WavLM**, used as an acoustic encoder. The wav2vec 2.0 branch uses the last hidden state as a sequence representation. The WavLM branch produces multiple layer outputs, which PrefSQA combines through a **learnable layer-weighted sum** with a temperature-controlled softmax; during training, a gate randomly drops some layers before softmax and renormalization so that the model does not rely too narrowly on a small subset of WavLM layers [2606.19597].

Each sequence then passes through a **residual feature processor** composed of two linear layers with GELU, residual addition, and LayerNorm. The processed semantic and acoustic sequences are concatenated and sent into a **single-layer BLSTM** with 256 hidden units per direction, followed by time-average pooling to obtain an utterance-level embedding \(\mathbf{f}\). From this embedding, linear heads produce a latent score \(s_0\) and a log variance \(\log \sigma^2\).

For a pair \((x,y)\), PrefSQA computes an uncertainty-aware Bradley–Terry-style logit

\[
z_{x,y} = \frac{s_x - s_y}{\sqrt{\sigma_x^{2} + \sigma_y^{2} + \epsilon}},
\]

where the denominator acts as an uncertainty-dependent temperature. Large variances reduce the magnitude of the logit and move the preference probability toward \(0.5\), while small variances yield sharper decisions. The temperature term is clamped empirically to the interval \(a = 0.6, b = 2.0\) [2606.19597].

A second architectural component is the **impairment attention head**, which is designed to emphasize localized degradations that may be diluted by global pooling. It applies a 1-D convolution over time, a sigmoid-gated \(1 \times 1\) convolution to produce a time-wise attention mask, an attention-weighted average of the features, and an MLP that outputs a residual score \(s_r\). The final utterance quality score is

\[
s = s_0 + \alpha * s_r,
\]

with \(\alpha = 0.1\), so the attention head acts as a correction rather than a replacement for the global score.

The third distinctive module is the **non-matching-reference comparison head**. After pooling, all utterance embeddings in a batch are collected, and additional in-batch comparisons are formed regardless of whether the utterances have matching content. For each sampled pair \((i,j)\), the input to a small MLP is

\[
\mathbf{u}_{i,j} = [\mathbf{f}_i, \mathbf{f}_j, \mathbf{f}_i - \mathbf{f}_j, |\mathbf{f}_i - \mathbf{f}_j|].
\]

This head predicts feature-level ranking relations using pseudo-targets derived from the model’s own scores, thereby encouraging a more globally consistent embedding geometry for non-matching comparisons.

## 3. Objectives, datasets, and training protocol

The main supervised target is binary: \(c = 1\) if \(x\) is preferred over \(y\), and \(c = 0\) otherwise. **Ties are not modeled**; equal-MOS pairs or otherwise tied cases are removed during dataset preparation. The principal objective is a Bradley–Terry logistic loss implemented as BCE with logits on the uncertainty-aware logit \(z_{x,y}\). The NMR module adds an auxiliary BCE loss on in-batch comparison logits, with soft pseudo-targets derived from latent score differences. The total objective is

\[
\mathcal{L} = \mathcal{L}_{\text{BT}} + \lambda \mathcal{L}_{\text{NMR}},
\]

with \(\lambda = 0.9\) in the reported experiments [2606.19597].

The empirical study uses five main preference datasets plus an unseen test-only set. These span MOS-derived preferences, low-noise simulated preferences, and human-collected preferences.

| Dataset | Label source | Content relation |
|---|---|---|
| NISQA | MOS-derived | Non-matching |
| SOMOS M / SOMOS NM | MOS-derived | Matching / non-matching |
| CHiLi M / CHiLi NM | Simulated SNR-based preference | Matching / non-matching |
| SpeechEval / SpeechJudge | Human preference | Matching |
| IUB-COSINE | MUSHRA-derived test-only set | Derived audio pairs |

**NISQA** is constructed by generating pairs within each original condition and split; the preferred member is the utterance with higher MOS, and each utterance is used at most three times. **SOMOS** is built from SOMOS-clean and appears in both matching-content and non-matching-content variants. **CHiLi** is synthetically generated by mixing LibriSpeech speech with CHiME-3 noise. For each pair, a base SNR \(n_a\) is sampled uniformly from \([-20,30]\) dB, an SNR difference \(\Delta \sim U(0.5,10)\) dB is sampled, and the preferred item is defined as the signal with higher SNR. The paper treats CHiLi as a low-noise preference source because the label is determined by construction rather than by subjective scalar ratings [2606.19597].

The human preference datasets are **SpeechEval** and **SpeechJudge**, both using direct human preference labels and only matching-content pairs. Generalization is assessed on **IUB-COSINE**, derived from COSINE/MUSHRA tests and used only at test time.

Training uses signals resampled at **16 kHz**, truncated or zero-padded to **6 seconds**. The implementation uses AdamW, effective batch size 32, learning rate 0.001 for task heads and wav2vec2, learning rate 0.00003 for WavLM with layer-wise decay 0.95, weight decay 0.01, and gradient clipping at norm 1.0. The feature processors use 64-dimensional bottlenecks; the NMR head uses \(k=3\) partners per anchor, \(\tau_{\text{nmr}} = 1.3\), label smoothing 0.03, and an MLP of size 256-128-1 [2606.19597].

## 4. Empirical performance across dataset regimes

The main evaluation metric is **pairwise preference prediction accuracy**. The strongest baseline families are **SQAPP** and **UPPSQA**, with an additional **PrefSQA-Frozen** variant in which wav2vec 2.0 and WavLM remain frozen during training [2606.19597].

On **MOS-derived datasets**, the gains of PrefSQA are present but small. On **NISQA**, test accuracy is 64.83 for SQAPP, 83.46 for UPPSQA, 82.80 for PrefSQA-Frozen, and **83.84** for PrefSQA. On **SOMOS M**, PrefSQA reaches 73.18 while PrefSQA-Frozen achieves **73.27**. On **SOMOS NM**, PrefSQA achieves **74.72**, slightly above 73.10 for UPPSQA and 73.48 for PrefSQA-Frozen. These margins are described as modest [2606.19597].

On the **low-noise CHiLi datasets**, the improvements are much clearer. On **CHiLi M**, SQAPP scores 94.78, UPPSQA 85.88, PrefSQA-Frozen 91.52, and **PrefSQA 96.29**. On **CHiLi NM**, SQAPP scores 86.90, UPPSQA 81.05, PrefSQA-Frozen 87.50, and **PrefSQA 90.37**. The non-matching CHiLi condition is especially informative because it stresses robustness to content mismatch while retaining deterministic preference labels [2606.19597].

On **human preference datasets**, the picture is more dataset-specific. On **SpeechEval**, UPPSQA reaches 86.31, PrefSQA-Frozen **86.85**, and PrefSQA 84.32. On **SpeechJudge**, SQAPP performs best at **70.40**, while PrefSQA is second-best at 68.20. These results indicate that the relative ranking among architectures depends on how closely the dataset matches each model’s inductive biases rather than on a uniform advantage for any one system [2606.19597].

Generalization to **unseen IUB-COSINE** is strong. When training on CHiLi NM and evaluating on IUB-COSINE, PrefSQA reaches **83.50**, above 78.06 for UPPSQA. When training on SpeechEval and evaluating on IUB-COSINE, PrefSQA-Frozen is best at **91.61** and PrefSQA is second-best at 89.28. These results support the claim that preference-based learning can transfer across corpora, degradation types, and recording conditions, although the choice between full fine-tuning and frozen encoders remains dataset-dependent [2606.19597].

Ablation on CHiLi further isolates the architectural contributions. On **CHiLi M**, removing both attention and NMR yields 95.52, removing only attention yields 96.22, removing only NMR yields 96.25, and the full model reaches **96.29**. On **CHiLi NM**, the corresponding values are 89.35, 88.86, 90.12, and **90.37**. The gains are modest but consistent, and they are larger in the harder non-matching condition than in the easier matching condition [2606.19597].

## 5. The critical role of high-quality preference datasets

A central claim of the PrefSQA study is that **dataset quality determines whether architectural gains are visible at all**. The paper uses **Concordance Correlation Coefficient (CCC)** between lists of misclassified pairs ordered by absolute MOS or SNR difference to analyze whether different models fail on the same examples. On **MOS-derived datasets** such as NISQA and SOMOS, CCC values between models are very high, around **0.9–0.99**, indicating that the different systems tend to make nearly the same errors in nearly the same margin regions. Combined with the small accuracy gaps, this is interpreted as evidence that MOS-derived preference labels inherit substantial noise from the underlying scalar ratings [2606.19597].

On the **simulated CHiLi datasets**, CCC is much lower: **0.30–0.56** on CHiLi M and **0.61–0.80** on CHiLi NM, depending on the model pair. This means the models are making different mistakes rather than being jointly constrained by the same noisy labels. The result is that genuine architectural differences become observable once label noise is reduced.

The paper also studies error distributions over absolute MOS or SNR margins. Across datasets, the median error occurs at **small margins**: roughly MOS differences around **0.3–0.4** or SNR differences around **1–2 dB**. For MOS-derived preferences, this is precisely the region in which annotation noise and scale uncertainty are most likely to dominate. For CHiLi, the same region remains perceptually subtle, but the labels are deterministic rather than rater-dependent, making it possible to expose real model improvements [2606.19597].

This leads to a broad methodological conclusion: progress in speech quality assessment cannot be judged solely from benchmarks whose preference labels are inherited from noisy MOS scores. High-quality preference data, whether **simulated with controlled ground truth** or **collected directly from human comparative judgments**, is necessary for reliable measurement of pairwise preference models. PrefSQA is presented not only as a model contribution but also as an empirical argument for re-centering benchmark design around the target supervision signal.

## 6. Terminological overlap and adjacent preference-centered uses

Within the provided literature, the string **“PrefSQA”** is not completely unique to speech quality assessment. In the paper **"PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs"**, the term is used informally to denote a **preference-aware, personalised QA** setting built around PrefEval, where the same question may require different answers for different users depending on explicit or implicit preferences [2508.10028]. That work defines a three-stage reference-free evaluation pipeline—**coverage**, **preference**, and **scoring**—in which a general query-specific rubric is first constructed, then re-ranked or augmented using a user profile, and finally applied by an LLM judge to score candidate answers.

The overlap is conceptual rather than architectural. Speech-quality PrefSQA replaces scalar MOS regression with pairwise preference learning over utterance pairs [2606.19597], whereas PREF’s “PrefSQA” usage concerns user-conditional evaluation of candidate textual answers rather than speech signals [2508.10028]. The shared theme is that **relative or conditioned judgments can be more faithful than single universal scores**: in one case because pairwise comparisons are less noisy than MOS, and in the other because generic reference-based metrics fail to capture user-specific alignment.

This suggests a broader research pattern. Preference-centered formulations are being used to address mismatch between the target notion of quality and the available supervision signal. In speech quality assessment, the mismatch lies between human comparative perception and scalar MOS labels. In personalized QA, it lies between generic answer quality and user-specific utility. The term “PrefSQA” therefore names a specific speech-quality framework, but it also sits within a wider shift toward preference-driven evaluation and learning across modalities.

Source: https://www.emergentmind.com/topics/prefsqa