---
title: VoiceMOS Challenge Overview
url: https://www.emergentmind.com/topics/voicemos-challenge
type: topic
---

# VoiceMOS Challenge Overview

The VoiceMOS Challenge is an international scientific initiative focused on advancing automatic, reference-free prediction of human mean opinion scores (MOS) for speech quality assessment across diverse domains, including text-to-speech (TTS), voice conversion (VC), singing voice synthesis/conversion (SVS/SVC), and speech enhancement (SE). Running annually since 2022, the challenge series systematically benchmarks data-driven approaches for emulating human listening test ratings, with particular emphasis on generalization, low-resource adaptation, and robust multi-domain performance. It occupies a central position in the empirical landscape of subjective speech evaluation research, establishing de facto community standards for both task formulation and system-level performance reporting.

## 1. Evolution and Scope of the VoiceMOS Challenge

The VoiceMOS Challenge emerged to address fundamental bottlenecks in speech quality assessment: human MOS tests are resource-intensive, slow, and fraught with inter-listener, inter-domain, and inter-protocol variability. The overarching goals are to stimulate the development and rigorous benchmarking of non-intrusive, data-driven models for MOS prediction, and to foster generalization to unseen systems, languages, and rater populations [2203.11389][2310.02640][2409.07001].

Challenge editions are marked by progressive complexity:
- **2022** introduced a dual-track format: a main "in-domain" track using English synthetic speech and an "out-of-domain" (OOD) track for domain adaptation using limited Chinese labeled data [2203.11389].
- **2023** emphasized zero-shot MOS prediction, explicitly withholding MOS labels for most evaluation splits and expanding to speech enhancement and singing voice scenarios [2310.02640].
- **2024** further diversified with three tracks: (1) "zoomed-in" high-quality TTS system discrimination, (2) multilingual, multi-system singing synthesis/conversion, and (3) semi-supervised speech quality prediction of noisy/clean/enhanced samples under strict data scarcity [2409.07001].

Each edition expands the empirical focus to broader domains, more challenging generalization settings, and new evaluation protocols.

## 2. Dataset Construction and Listening Test Protocols

Datasets underpinning VoiceMOS tracks are curated for diversity in system type, linguistic domain, and rater demographics:
- **Main Track Datasets:** Large-scale, lab-based listening tests from Blizzard TTS Challenges, Voice Conversion Challenges, ESPnet-TTS, and voicebank datasets—typically involving hundreds of systems and thousands of utterances rated on a 1–5 MOS scale [2203.11389][2204.02249][2409.09305][2409.07001].
- **Out-of-domain and Multilingual Data:** OOD splits in Chinese (BC2019), French, and multilingual singing (SingMOS, Japanese/Chinese) extend the coverage to new listening-test languages, system architectures, and recording conditions [2310.02640][2409.07001].
- **Noisy/Enhanced Speech:** ITU-T P.835 protocols with SIG (signal distortion), BAK (background noise), and OVRL (overall quality) provide multidimensional ratings under adverse and enhancement conditions [2409.07001][2506.00506].

New listening tests are routinely commissioned for "zoomed-in" subsets (top 12–25% of systems by MOS) and cross-system scenarios to probe fine-grained quality distinctions. Rater assignments are controlled, with each utterance rated by multiple listeners (typically 5–8), and, in some protocols, detailed listener and system metadata are captured to facilitate advanced modeling of individual and group differences [2209.06358][2311.10656].

## 3. Evaluation Metrics and Benchmarking

The challenge standardizes a rigorous set of performance metrics at both the utterance and system levels:
- **Mean Squared Error (MSE):**
  $$
  \mathrm{MSE} = \frac{1}{N}\sum_{i=1}^{N} (\hat y_i - y_i)^2
  $$
- **Pearson Linear Correlation Coefficient (PLCC/LCC):**
  $$
  \mathrm{PLCC} = \frac{\sum_i (\hat y_i - \bar{\hat y})(y_i - \bar y)}{\sqrt{\sum_i(\hat y_i - \bar{\hat y})^2} \sqrt{\sum_i(y_i - \bar y)^2}}
  $$
- **Spearman Rank Correlation Coefficient (SRCC):**
  $$
  \mathrm{SRCC} = 1 - \frac{6 \sum_i d_i^2}{N(N^2-1)},\quad d_i = \operatorname{rank}(\hat y_i) - \operatorname{rank}(y_i)
  $$
- **Kendall’s Tau (KTAU):**
  $$
  \mathrm{KTAU} = \frac{C-D}{0.5N(N-1)}
  $$
  with $C$ and $D$ concordant/discordant pairs.

Where applicable (notably in enhancement tracks), metrics are reported for all three ITU-T P.835 categories (SIG, BAK, OVRL). Primary challenge rankings use system-level SRCC to emphasize accurate system ranking—a practical priority in quality benchmarking [2203.11389][2409.07001].

## 4. Modeling Approaches and System Architectures

The trajectory of leading systems highlights convergence on self-supervised learning (SSL) backbones, feature fusion, retrieval, and meta-learning:
- **Fine-tuned SSL Predictors:** Dominant across all editions are models based on wav2vec 2.0, HuBERT, and WavLM, fine-tuned for MOS regression using MSE or multi-task objectives. Frame-wise pooling strategies, listener/domain metadata embeddings, and contrastive losses are recurrent enhancements [2204.02152][2310.02640][2409.07001].
- **Hybrid Feature and Ensemble Models:** Fused representations from SSL and spectrogram-image encoders (e.g., EfficientNetV2 on mel-spectrograms) capture complementary cues: SSL layers encode global structure and system-level ranking, while spectrogram-based features excel at detecting local artifacts and improving calibration (low MSE) [2409.09305].
- **Retrieval and Non-parametric Augmentation:** Retrieval-augmented models integrate k-nearest-neighbor ranking or prior-net weighting for robust zero-/few-shot generalization, particularly excelling in "zoom-in" discrimination among top-tier TTS systems [2409.07001].
- **Task-specific Pre-training:** Multi-stage pre-training (e.g., SNR→BAK mapping, spoof/natural for SIG) facilitates label-efficient adaptation to SE and noisy speech scenarios [2506.00506][2409.07001].
- **Metadata and Listener Modeling:** Listener-dependent architectures and the explicit use of system/rater metadata provide regularization against domain shift and permit per-listener prediction [2209.06358][2311.10656][2409.07001].
- **Model Ensembling:** Multi-stream, multi-backbone ensembles, and stacking of "strong" and "weak" learners deliver significant reductions in utterance- and system-level MSE [2204.02152][2409.09305].

A summary of distinctive model components from recent editions:

| Approach           | Key Feature Integration                      | Context                        |
|--------------------|---------------------------------------------|-------------------------------|
| SSL Fine-tuning    | Wav2vec2.0/HuBERT MOS regression            | All tracks/editions           |
| Hybrid Fusion      | SSL + spectrogram/image (EfficientNetV2)    | High-quality TTS, T05 [2409.09305] |
| Retrieval-based    | SSL backbone + kNN non-parametric head      | Zero/few-shot, zoom-in SRCC   |
| Listener Modeling  | Rater embeddings, multi-task regression     | Out-of-domain, singing, noisy speech |
| Metadata           | System/rater ID one-hot/binary injection    | Analysis, regularization      |
| Ensembling/Stacking| Multiple SSL backbones, handcrafted features| System-level MSE/SRCC         |

## 5. Major Findings and Comparative Results

Robust findings consistently validated across editions include:
- **SSL-based fine-tuned models dominate both in-domain and out-of-domain tracks**, with system-level SRCC routinely exceeding 0.93 on main tracks and reaching 0.979 in favorable OOD settings [2203.11389][2310.02640].
- **Fusion of SSL and spectral/image representations provides complementary gains**, crucial for both absolute-scoring (MSE) and high-MOS system discrimination [2409.09305][2409.07001].
- **Retrieval-augmented and task-specific pre-training approaches enable strong generalization with limited or no in-domain MOS labels**, especially in challenging zero-/few-shot and semi-supervised tracks [2409.07001][2506.00506].
- **Listener-dependent modeling is necessary for robust cross-domain generalization** in tracks where rater bias and distributional differences are substantial [2311.10656].

Significant performance gains are achieved by carefully matching training and evaluation data distributions and leveraging domain adaptation or self-training where allowed [2310.02640]. In data-scarce conditions (less than 100 labeled utterances), simulation-driven proxy tasks and multi-task architectures sustain competitive correlation and ranking metrics [2506.00506].

## 6. Open Challenges and Lessons Learned

The challenge series lays bare several persistent technical problems:
- **Domain Shift and Zero-shot Generalization:** No single model/dataset combination achieves universal performance across all domains (TTS, VC, SVS/SVC, enhancement, noisy speech). Generalization to ultra-high-quality, multi-lingual, and new architecture scenarios (e.g., zero-shot voice cloning, multi-sampling-rate inputs) remains unsolved [2409.07001][2507.14647].
- **Calibrated Utterance-level Prediction:** System-level ranking is reliably attainable, but utterance-level reliability (SRCC, MSE) and credible uncertainty estimates lag behind [2209.06358].
- **Dataset and Metric Design:** System-level performance metrics are sensitive to imbalance in utterance counts per condition; sufficient per-system data (>30 utterances) is recommended for robust error bounds [2209.06358].
- **Label Efficiency & Data Augmentation:** Semi-supervised/adaptively pre-trained models confirm that strong generalization is possible without large labeled datasets, but model and augmentation strategies must be tailored for small-data regimes [2506.00506][2204.02152].

Best practices arising from challenge analysis include:
- Leveraging diverse, high-quality MOS corpora and maintaining domain tags.
- Utilizing multi-stage and multi-task learning regimens.
- Combining ranking (contrastive) and regression (MSE) losses for stable optimization.
- Preferring utterance-level metrics for small-N or highly imbalanced test sets.

## 7. Prospective Directions

VoiceMOS organizers and participants highlight several future directions:
- **Multi-task and Multi-domain Prediction Architectures:** Simultaneously predict MOS, SIG/BAK/OVRL, and possibly finer-grained perceptual judgments (e.g., pairwise preference) [2409.07001].
- **Expansion to Broader Audio Domains:** Generalization across sampling rates, audio types (e.g., music, environmental sound), and zero-shot voice cloning is an active area [2409.07001][2507.14647].
- **Uncertainty Quantification:** Improved modeling of confidence in utterance- and system-level predictions.
- **Active/Domain-adaptive Learning:** Incorporating principled domain adaptation and active sampling under strict label budgets.
- **Advanced Feature Engineering:** SFI convolutional layers and neural analog filters have demonstrated promise for sampling-frequency invariance [2507.14647].

The VoiceMOS Challenge has thus defined the state of the art in MOS prediction, systematically pushed model development toward robust, domain-agnostic evaluators, and set high standards for reproducible, data-driven speech quality assessment [2409.07001][2310.02640][2203.11389][2507.14647].

Source: https://www.emergentmind.com/topics/voicemos-challenge