PrefSQA: MOS-Free Pairwise Speech Quality

• PrefSQA is a MOS-free framework that predicts pairwise preferences for speech quality assessment, avoiding the noise of scalar MOS ratings.
• It utilizes a dual-encoder design with wav2vec 2.0 and WavLM, incorporating uncertainty-aware logits and an impairment attention head to capture localized degradations.
• Empirical results across diverse datasets highlight PrefSQA’s robustness and emphasize the need for high-quality, preference-based supervision in speech evaluation.

PrefSQA most directly denotes a framework for MOS-free pairwise preference prediction in speech quality assessment. Rather than regressing a scalar Mean Opinion Score (MOS) for a single utterance, it predicts which signal in a pair is preferred, using a dual-encoder architecture with uncertainty-aware logits, an impairment attention head, and a non-matching-reference (NMR) comparison module (Fan et al., 17 Jun 2026). The framework is motivated by the claim that scalar MOS labels are sensitive to rater variability, listening-test protocol differences, label quantization, and corpus effects, whereas comparative judgments can be cleaner and more consistent. In the literature provided here, the same string also appears as an informal label for preference-aware personalized question answering, so the term has a limited but nontrivial terminological overlap across domains (Fu et al., 8 Aug 2025).

1. From MOS regression to pairwise preference prediction

Perceptual speech quality assessment has classically relied on Mean Opinion Scores, in which human raters assign scalar ratings and the scores are averaged. Modern MOS-based systems such as MOSNet, NISQA, and VoiceMOS-style models regress from acoustic features to these scalar labels. The central critique underlying PrefSQA is that MOS supervision is intrinsically noisy: different listeners interpret scales differently, listening tests vary in scale type and instructions, many protocols quantize judgments onto coarse discrete scales, and cross-corpus or cross-domain effects distort the mapping between perception and numeric labels (Fan et al., 17 Jun 2026).

PrefSQA reframes the task as pairwise preference prediction. The model receives a pair of utterances $(x,y)$ and predicts which one is preferred in perceived quality. This binary formulation is motivated by the observation that listeners are often more consistent when comparing stimuli than when assigning absolute scores. Comparative labels depend less on the exact score anchors used in a listening test, and they can preserve relative information even when small MOS differences would be dominated by annotation noise.

The paper is explicit that PrefSQA is MOS-free in its learning objective. Even when a dataset originally contains MOS values, those values are used only to derive pairwise labels during dataset construction; the model is never trained to predict MOS, and MOS is not used as a supervision signal at training or evaluation time. This keeps the method in preference space rather than treating pairwise prediction as a secondary view of scalar regression.

2. Core architecture and scoring mechanism

PrefSQA is built around a dual-encoder semantic–acoustic backbone. One branch is wav2vec 2.0, used as a semantic encoder, and the other is WavLM, used as an acoustic encoder. The wav2vec 2.0 branch uses the last hidden state as a sequence representation. The WavLM branch produces multiple layer outputs, which PrefSQA combines through a learnable layer-weighted sum with a temperature-controlled softmax; during training, a gate randomly drops some layers before softmax and renormalization so that the model does not rely too narrowly on a small subset of WavLM layers (Fan et al., 17 Jun 2026).

Each sequence then passes through a residual feature processor composed of two linear layers with GELU, residual addition, and LayerNorm. The processed semantic and acoustic sequences are concatenated and sent into a single-layer BLSTM with 256 hidden units per direction, followed by time-average pooling to obtain an utterance-level embedding $\mathbf{f}$. From this embedding, linear heads produce a latent score $s_0$ and a log variance $\log \sigma^2$.

For a pair $(x,y)$, PrefSQA computes an uncertainty-aware Bradley–Terry-style logit

$$z_{x,y} = \frac{s_x - s_y}{\sqrt{\sigma_x^{2} + \sigma_y^{2} + \epsilon}},$$

where the denominator acts as an uncertainty-dependent temperature. Large variances reduce the magnitude of the logit and move the preference probability toward $0.5$, while small variances yield sharper decisions. The temperature term is clamped empirically to the interval $a = 0.6, b = 2.0$ (Fan et al., 17 Jun 2026).

A second architectural component is the impairment attention head, which is designed to emphasize localized degradations that may be diluted by global pooling. It applies a 1-D convolution over time, a sigmoid-gated $1 \times 1$ convolution to produce a time-wise attention mask, an attention-weighted average of the features, and an MLP that outputs a residual score $s_r$. The final utterance quality score is

$\mathbf{f}$0

with $\mathbf{f}$1, so the attention head acts as a correction rather than a replacement for the global score.

The third distinctive module is the non-matching-reference comparison head. After pooling, all utterance embeddings in a batch are collected, and additional in-batch comparisons are formed regardless of whether the utterances have matching content. For each sampled pair $\mathbf{f}$2, the input to a small MLP is

$\mathbf{f}$3

This head predicts feature-level ranking relations using pseudo-targets derived from the model’s own scores, thereby encouraging a more globally consistent embedding geometry for non-matching comparisons.

3. Objectives, datasets, and training protocol

The main supervised target is binary: $\mathbf{f}$4 if $\mathbf{f}$5 is preferred over $\mathbf{f}$6, and $\mathbf{f}$7 otherwise. Ties are not modeled; equal-MOS pairs or otherwise tied cases are removed during dataset preparation. The principal objective is a Bradley–Terry logistic loss implemented as BCE with logits on the uncertainty-aware logit $\mathbf{f}$8. The NMR module adds an auxiliary BCE loss on in-batch comparison logits, with soft pseudo-targets derived from latent score differences. The total objective is

$\mathbf{f}$9

with $s_0$0 in the reported experiments (Fan et al., 17 Jun 2026).

The empirical study uses five main preference datasets plus an unseen test-only set. These span MOS-derived preferences, low-noise simulated preferences, and human-collected preferences.

Dataset	Label source	Content relation
NISQA	MOS-derived	Non-matching
SOMOS M / SOMOS NM	MOS-derived	Matching / non-matching
CHiLi M / CHiLi NM	Simulated SNR-based preference	Matching / non-matching
SpeechEval / SpeechJudge	Human preference	Matching
IUB-COSINE	MUSHRA-derived test-only set	Derived audio pairs

NISQA is constructed by generating pairs within each original condition and split; the preferred member is the utterance with higher MOS, and each utterance is used at most three times. SOMOS is built from SOMOS-clean and appears in both matching-content and non-matching-content variants. CHiLi is synthetically generated by mixing LibriSpeech speech with CHiME-3 noise. For each pair, a base SNR $s_0$1 is sampled uniformly from $s_0$2 dB, an SNR difference $s_0$3 dB is sampled, and the preferred item is defined as the signal with higher SNR. The paper treats CHiLi as a low-noise preference source because the label is determined by construction rather than by subjective scalar ratings (Fan et al., 17 Jun 2026).

The human preference datasets are SpeechEval and SpeechJudge, both using direct human preference labels and only matching-content pairs. Generalization is assessed on IUB-COSINE, derived from COSINE/MUSHRA tests and used only at test time.

Training uses signals resampled at 16 kHz, truncated or zero-padded to 6 seconds. The implementation uses AdamW, effective batch size 32, learning rate 0.001 for task heads and wav2vec2, learning rate 0.00003 for WavLM with layer-wise decay 0.95, weight decay 0.01, and gradient clipping at norm 1.0. The feature processors use 64-dimensional bottlenecks; the NMR head uses $s_0$4 partners per anchor, $s_0$5, label smoothing 0.03, and an MLP of size 256-128-1 (Fan et al., 17 Jun 2026).

4. Empirical performance across dataset regimes

The main evaluation metric is pairwise preference prediction accuracy. The strongest baseline families are SQAPP and UPPSQA, with an additional PrefSQA-Frozen variant in which wav2vec 2.0 and WavLM remain frozen during training (Fan et al., 17 Jun 2026).

On MOS-derived datasets, the gains of PrefSQA are present but small. On NISQA, test accuracy is 64.83 for SQAPP, 83.46 for UPPSQA, 82.80 for PrefSQA-Frozen, and 83.84 for PrefSQA. On SOMOS M, PrefSQA reaches 73.18 while PrefSQA-Frozen achieves 73.27. On SOMOS NM, PrefSQA achieves 74.72, slightly above 73.10 for UPPSQA and 73.48 for PrefSQA-Frozen. These margins are described as modest (Fan et al., 17 Jun 2026).

On the low-noise CHiLi datasets, the improvements are much clearer. On CHiLi M, SQAPP scores 94.78, UPPSQA 85.88, PrefSQA-Frozen 91.52, and PrefSQA 96.29. On CHiLi NM, SQAPP scores 86.90, UPPSQA 81.05, PrefSQA-Frozen 87.50, and PrefSQA 90.37. The non-matching CHiLi condition is especially informative because it stresses robustness to content mismatch while retaining deterministic preference labels (Fan et al., 17 Jun 2026).

On human preference datasets, the picture is more dataset-specific. On SpeechEval, UPPSQA reaches 86.31, PrefSQA-Frozen 86.85, and PrefSQA 84.32. On SpeechJudge, SQAPP performs best at 70.40, while PrefSQA is second-best at 68.20. These results indicate that the relative ranking among architectures depends on how closely the dataset matches each model’s inductive biases rather than on a uniform advantage for any one system (Fan et al., 17 Jun 2026).

Generalization to unseen IUB-COSINE is strong. When training on CHiLi NM and evaluating on IUB-COSINE, PrefSQA reaches 83.50, above 78.06 for UPPSQA. When training on SpeechEval and evaluating on IUB-COSINE, PrefSQA-Frozen is best at 91.61 and PrefSQA is second-best at 89.28. These results support the claim that preference-based learning can transfer across corpora, degradation types, and recording conditions, although the choice between full fine-tuning and frozen encoders remains dataset-dependent (Fan et al., 17 Jun 2026).

Ablation on CHiLi further isolates the architectural contributions. On CHiLi M, removing both attention and NMR yields 95.52, removing only attention yields 96.22, removing only NMR yields 96.25, and the full model reaches 96.29. On CHiLi NM, the corresponding values are 89.35, 88.86, 90.12, and 90.37. The gains are modest but consistent, and they are larger in the harder non-matching condition than in the easier matching condition (Fan et al., 17 Jun 2026).

5. The critical role of high-quality preference datasets

A central claim of the PrefSQA study is that dataset quality determines whether architectural gains are visible at all. The paper uses Concordance Correlation Coefficient (CCC) between lists of misclassified pairs ordered by absolute MOS or SNR difference to analyze whether different models fail on the same examples. On MOS-derived datasets such as NISQA and SOMOS, CCC values between models are very high, around 0.9–0.99, indicating that the different systems tend to make nearly the same errors in nearly the same margin regions. Combined with the small accuracy gaps, this is interpreted as evidence that MOS-derived preference labels inherit substantial noise from the underlying scalar ratings (Fan et al., 17 Jun 2026).

On the simulated CHiLi datasets, CCC is much lower: 0.30–0.56 on CHiLi M and 0.61–0.80 on CHiLi NM, depending on the model pair. This means the models are making different mistakes rather than being jointly constrained by the same noisy labels. The result is that genuine architectural differences become observable once label noise is reduced.

The paper also studies error distributions over absolute MOS or SNR margins. Across datasets, the median error occurs at small margins: roughly MOS differences around 0.3–0.4 or SNR differences around 1–2 dB. For MOS-derived preferences, this is precisely the region in which annotation noise and scale uncertainty are most likely to dominate. For CHiLi, the same region remains perceptually subtle, but the labels are deterministic rather than rater-dependent, making it possible to expose real model improvements (Fan et al., 17 Jun 2026).

This leads to a broad methodological conclusion: progress in speech quality assessment cannot be judged solely from benchmarks whose preference labels are inherited from noisy MOS scores. High-quality preference data, whether simulated with controlled ground truth or collected directly from human comparative judgments, is necessary for reliable measurement of pairwise preference models. PrefSQA is presented not only as a model contribution but also as an empirical argument for re-centering benchmark design around the target supervision signal.

6. Terminological overlap and adjacent preference-centered uses

Within the provided literature, the string “PrefSQA” is not completely unique to speech quality assessment. In the paper "PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs", the term is used informally to denote a preference-aware, personalised QA setting built around PrefEval, where the same question may require different answers for different users depending on explicit or implicit preferences (Fu et al., 8 Aug 2025). That work defines a three-stage reference-free evaluation pipeline—coverage, preference, and scoring—in which a general query-specific rubric is first constructed, then re-ranked or augmented using a user profile, and finally applied by an LLM judge to score candidate answers.

The overlap is conceptual rather than architectural. Speech-quality PrefSQA replaces scalar MOS regression with pairwise preference learning over utterance pairs (Fan et al., 17 Jun 2026), whereas PREF’s “PrefSQA” usage concerns user-conditional evaluation of candidate textual answers rather than speech signals (Fu et al., 8 Aug 2025). The shared theme is that relative or conditioned judgments can be more faithful than single universal scores: in one case because pairwise comparisons are less noisy than MOS, and in the other because generic reference-based metrics fail to capture user-specific alignment.

This suggests a broader research pattern. Preference-centered formulations are being used to address mismatch between the target notion of quality and the available supervision signal. In speech quality assessment, the mismatch lies between human comparative perception and scalar MOS labels. In personalized QA, it lies between generic answer quality and user-specific utility. The term “PrefSQA” therefore names a specific speech-quality framework, but it also sits within a wider shift toward preference-driven evaluation and learning across modalities.