PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets

Abstract: Mean opinion scores (MOS) are widely used for speech quality assessment, yet scalar labels are sensitive to rater variability and listening test differences. This introduces labeling noise, which limits the reliability of MOS prediction. Preference prediction reduces this variability as listeners compare signals directly, producing cleaner labels. We study MOS-free preference prediction and propose PrefSQA, which incorporates uncertainty-aware logits, an impairment attention head, and a module based on non-matching-reference comparisons. We use and refine five datasets, including MOS-derived and low-noise simulated sets with matching and non-matching content, experiment with human preference sets, and test on unseen data. Experiments show small improvements on MOS-derived data, while other sets reveal clear improvement over the baselines, highlighting the value of high-quality preference data and demonstrating the effectiveness of the proposed method.

• The paper presents a MOS-free pairwise preference model, PrefSQA, that uses dual semantic-acoustic encoders to improve speech quality prediction.
• It incorporates innovative components like uncertainty-aware logits, impairment attention, and a non-matching-reference head to reduce label noise impacts.
• Experimental results on simulated low-noise datasets show up to 11% accuracy improvement over baselines, highlighting the importance of high-quality data.

PrefSQA: Advancing MOS-Free Pairwise Preference Prediction in Speech Quality Assessment

Introduction

The paper "PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets" (2606.19597) addresses the limitations of mean opinion score (MOS)-based speech quality assessment (SQA) by proposing a preference-based framework that is strictly MOS-free. The work introduces the PrefSQA model, leveraging dual semantic-acoustic encoders and several novel architectural components to enhance pairwise preference prediction accuracy. The study systematically investigates the implications of high-quality preference datasets and demonstrates that conventional MOS-derived labels obscure true model improvements due to high labeling noise. Distinct experimental protocols, ablation analyses, and rigorous error studies provide a robust evaluation of both the model and the foundational importance of dataset quality.

Model Architecture

PrefSQA builds on the semantic-acoustic abstraction of UPPSQA, fusing representations from wav2vec 2.0 and WavLM encoders. This dual-encoder strategy extracts both semantic-linguistic and acoustic features from input waveforms. The encoders are complemented by a residual feature processor and BiLSTM aggregator to yield utterance-level embeddings, which are subsequently scored via two linear heads to output a latent quality score and uncertainty estimate per utterance.

Enhanced modeling capabilities are realized by three major architectural augmentations:

• Uncertainty-Aware Pairwise Preference Logits: PrefSQA employs a Bradley-Terry style logit calculation, where the difference in latent scores is adaptively tempered by the model’s own uncertainty estimates. The temperature parameter (denominator) is clamped to suppress degenerate behavior, and the final preference probability is fed into a binary cross-entropy loss.
• Impairment Attention Head: This lightweight head applies temporal convolution and gated attention over encoder features to emphasize local degradations. Its output corrects the global score, producing a composite metric sensitive to both holistic and localized impairments.
• Feature-Level Non-Matching-Reference (NMR) Head: Operating on utterance embeddings within a batch, this auxiliary module utilizes in-batch, pseudo-labeled preference comparisons to regularize the embedding space, enforcing a finer-grained ranking order.

  Figure 1: PrefSQA model architecture for input waveform $x$ with semantic-acoustic encoders, augmented with uncertainty-aware preference logits, a lightweight impairment attention head (purple blocks), and a feature-level non-matching-reference (NMR) head (blue blocks).

Dataset Construction and Evaluation Protocol

The study implements five distinct preference datasets across multiple sources and tasks, with special attention to controlling labeling noise:

• MOS-Derived Preference Sets: NISQA and SOMOS are processed to generate pairwise comparison labels strictly from MOS (never seen by the model), partitioned by content-matching and non-matching pairs.
• Simulated Low-Noise Datasets (CHiLi M/NM): Clean LibriSpeech utterances are mixed with CHiME-3 noise, manipulating SNRs systematically to define pairwise preferences without human annotation, yielding datasets with negligible labeling noise.
• Human Preference Datasets: Evaluation includes human-curated sets (SpeechEval, SpeechJudge), which provide a realistic benchmark for model generalization beyond simulated or MOS-derived data.

All datasets are constructed with balanced class distributions, and partitioned by absolute MOS or SNR margins to carefully examine model performance across varying degrees of pairwise difficulty.

Experimental Findings

The PrefSQA architecture consistently matches or outperforms strong baselines such as SQAPP and UPPSQA on both MOS-derived and simulated preference datasets. On MOS-derived sets (NISQA, SOMOS), all advanced models reach closely packed accuracy plateaus, with the accuracy gap between models rarely exceeding 2%. For example, on NISQA, PrefSQA achieves 83.84% accuracy—marginally higher than UPPSQA’s 83.46%. These small margins are attributed primarily to persistent labeling noise in MOS-derived supervision.

More distinct results emerge in simulated datasets with explicit, low-noise preference labeling. On CHiLi M, PrefSQA achieves 96.29% accuracy, up to 11% higher than UPPSQA. On the more difficult CHiLi NM task, PrefSQA reaches 90.37%, nearly 9% higher than baseline models. These large deltas unequivocally demonstrate the advantages of the proposed architecture when evaluated in a regime where model errors reflect genuine limitations, not label corruption.

The human labeled sets affirm robust generalization: on SpeechEval, PrefSQA achieves 84.32% accuracy, as compared to 86.31% for UPPSQA. On SpeechJudge, SQAPP (70.40%) slightly outperforms PrefSQA (68.20%), potentially reflecting better alignment with the per-sample annotation protocol.

Error analysis focusing on the concordance correlation coefficient (CCC) between lists of misclassified pairs reveals a critical distinction: MOS-derived datasets show high CCC (close to 1), indicating that all models err on the same pairs—evidence of labeling noise. In contrast, simulated datasets reveal much more diverse error patterns (CCC as low as 0.30), demonstrating that the PrefSQA model’s improvements are substantive and not noise-driven.

Component Ablation

An ablation study isolates the contributions of the impairment attention head and NMR head. For the more challenging CHiLi NM dataset, removing the attention head reduces accuracy to 88.86% and removing the NMR head yields 90.12%, both underperforming the full PrefSQA (90.37%). These findings indicate the two components provide synergistic, non-redundant gains, especially in difficult non-matching-content scenarios.

Implications and Future Perspectives

This work highlights the critical importance of dataset quality—specifically, the necessity of low-noise pairwise data for exposing meaningful architectural improvements in SQA models. The finding that widely used MOS-derived preferences mask genuine model advantages has both practical and theoretical import: (1) future research in SQA should prioritize direct human preference annotation or principled simulation pipelines; (2) benchmarks must account for labeling protocol and margin distributions, not only global accuracy.

The PrefSQA approach, by remaining MOS-free throughout, more closely approximates real-world scenarios where absolute ratings are unavailable. Its architectural innovations—uncertainty-aware scoring, impairment localization, and in-batch ranking—push the boundaries of pairwise prediction and demonstrate strong transferability to unseen, human-annotated data. The analysis suggests future SQA modeling should consider explicit tie-handling for near-equal quality pairs, finer granularity in uncertainty modeling, and further exploration of embedding regularization for robust generalization.

Conclusion

PrefSQA introduces a principled, MOS-free framework for pairwise preference prediction in speech quality assessment, enabled by semantic-acoustic dual encoders and a suite of task-targeted architectural modules. The study’s rigorous experimental paradigm demonstrates that only high-quality, low-noise preference datasets can faithfully expose the capabilities of advanced SQA models. The findings underscore the necessity of revisiting current practices in SQA benchmarking and dataset curation, and signal promising directions for the adoption of pairwise, comparison-based approaches in perceptual quality modeling for speech technologies.