ConferencingSpeech 2022 Challenge

• The ConferencingSpeech 2022 Challenge is a benchmark that evaluates non-intrusive DNN models predicting subjective speech quality in online conferencing scenarios.
• It leverages a large-scale multilingual corpus (~200 hours) with a variety of real-world signal impairments to train and test robust models.
• Advanced architectures using polynomial mapping achieved high accuracy with PCC > 0.9 and RMSE_MAP < 0.25 on blind test sets.

The ConferencingSpeech 2022 Challenge is a competitive benchmark targeting the objective assessment of speech quality for online conferencing applications, emphasizing non-intrusive deep neural network (DNN) models that predict subjective quality directly from the impaired signal. The challenge introduces a large-scale multilingual dataset with high diversity in signal degradations and aims to establish rigorous evaluation methodologies, open-source baselines, and reproducible comparisons across state-of-the-art approaches (Yi et al., 2022).

1. Motivation and Context

Online conferencing (e.g., Zoom, Teams, Webex) has rendered high-quality remote speech communication essential. Real-world deployment is impeded by degradations such as background noise, reverberation, codec distortion, packet loss, jitter, and artifacts from noise suppression. Gold-standard speech quality assessment relies on subjective listening tests (ITU-T P.800/P.808), which are accurate but costly and impractical for customer data or real-time feedback. Full-reference objective measures (e.g., PESQ, POLQA) require a clean reference signal that is typically unavailable in deployment settings. Existing single-ended methods (e.g., ITU-T P.563) are inadequate for wideband or super-wideband speech common in modern conferencing.

The challenge motivates the development and benchmarking of non-intrusive, no-reference DNN models—specifically those predicting mean opinion scores (MOS) directly from impaired signals—for the robust and scalable evaluation of speech quality in contemporary conferencing scenarios (Yi et al., 2022).

2. Corpus Construction

The training and evaluation corpus for the challenge was curated to maximize diversity of language, degradation, and acoustic conditions.

Training/Development Corpus

• Volume: ≈ 86,000 speech samples (≈ 200 hours)
• Languages: Chinese, English, German
• Source Corpora:
  • Tencent Corpus: ≈ 10,000 non-reverberant, 4,000 reverberant/reverberated
  • NISQA Corpus: ≈ 14,000
  • PSTN Corpus: filtered Librivox (≈ tens of thousands)
  • IU Bloomington Corpus: ≈ 36,000, used as auxiliary/reference

Impairment types include:

• Additive noises (white, non-stationary, DNS challenge)
• Filters (high/low-pass), amplitude clipping
• Codec degradations (AMR, Opus), packet loss concealment (PLC), noise suppression
• Room reverberation (simulated and real), clean speech examples

For example, in the Tencent Corpus (non-reverberant branch): 60% clips exhibit non-stationary noise, 10% white noise, 5% codecs, 3.75% filtering, with combinatorial mixes (e.g., 60% with only first-step impairment, 10% add noise suppression).

Blind Test Sets

• Size: 4,372 clips
• Subsets:
  • Tencent blind: 20% of Tencent corpus
  • PSTN: 5% of PSTN samples
  • TUB: new subset (865 clips × 62 degradation conditions ≈ 4300 clips)

Each subset maintains the training corpus's diversity in language and degradation, but with unseen speakers and conditions (Yi et al., 2022).

3. Subjective Annotation Protocol

Crowdsourced absolute category rating (ACR) was conducted according to ITU-T P.808. Participants underwent qualification and modified training before rating clips. Each sample received at least 24 listener ratings, with rigorous cleaning to ensure ≥ 20 valid votes per clip (the PSTN training/test and NISQA corpora had lower vote counts as defined).

MOS was annotated on the standard 1–5 scale; some auxiliary references (IU Bloomington) used a 0–100 scale. The final MOS per clip was computed as the arithmetic mean of valid votes (Yi et al., 2022).

4. Evaluation Methodologies

Model outputs $\hat{y}$ are mapped to the subjective MOS scale using a monotonic cubic polynomial:

$\hat{y}' = a + b\hat{y} + c\hat{y}^2 + d\hat{y}^3$

Mapping coefficients are chosen to minimize MSE and ensure monotonicity.

Key metrics:

Metric	Definition / Formula	Significance
RMSE	$$\sqrt{\frac{1}{N}\sum_{i=1}^N (\mathrm{MOS}_i - \hat{\mathrm{MOS}}_i)^2}$$	Absolute accuracy of MOS prediction
PCC	Pearson correlation between prediction and truth	Linear association of predicted and true MOS
SRCC	$$\rho = 1 - \frac{6\sum_i (R(x_i)-R(y_i))^2}{N(N^2-1)}$$	Correlation of prediction and truth ranks
Outlier Ratio (OR)	Fraction of clips with $|\mathrm{MOS}_i - \hat{\mathrm{MOS}}_i|$ exceeding MOS's 95% CI	Robustness to extreme errors

This framework standardizes comparisons, accounting for scaling differences and statistical bias between lab and crowdsourcing settings (Yi et al., 2022).

5. Baseline and Submitted Systems

Baseline Models

• Baseline 1 (“NISQA-Lite”):
  • Input: 48-band log-mel spectrogram, context of ±7 frames (~150 ms)
  • CNN encoder, FFN layers, LSTM, temporal pooling, scalar MOS
  • MSE loss
• Baseline 2 (“NISQA full”):
  • Input: same log-mel features
  • CNN block, self-attention pooling, MLP, scalar MOS
  • End-to-end training, MSE loss

Both baselines provide reproducible reference points; Baseline 2 employs self-attention pooling.

Participating Systems

18 teams submitted non-intrusive DNN models. While specific models are not individually detailed in (Yi et al., 2022), high-performing trends included:

• Enhanced CNN/LSTM or hybrid CNN+Transformer/self-attention architectures
• Multi-scale feature extraction (diverse STFT windows)
• Feature fusion: log-mel, MFCC, self-supervised embeddings (wav2vec 2.0, CPC)
• Multi-task learning (predicting auxiliary attributes like noisiness, reverberation)
• Data augmentation (synthetic reverberation, new noise combinations)
• Loss functions: primarily MSE, some with Huber or rank-based losses

This suggests robust generalization depended on both architectural advances and training data heterogeneity.

6. Results and Comparative Analysis

• All 18 teams outperformed Baseline 1 (RMSE_MAP ≈ 0.55).
• 11 teams surpassed Baseline 2 (RMSE_MAP ≈ 0.28).

Top result: Team #1 achieved RMSE_MAP ≈ 0.24–0.26 and PCC ≈ 0.90–0.94 across blind test sets.

Blind-Set Performance (Best Systems)

Subset	RMSE_MAP	PCC
Tencent Corpus	≈ 0.23	≈0.96
TUB Corpus	≈ 0.25	≈0.90
PSTN Corpus	≈ 0.26	≈0.92

Self-attention pooling consistently surpassed LSTM or mean pooling. Systems integrating hand-crafted and learned features attained lower RMSE. Training on corpora with diverse degradations was critical; polynomial mapping was essential for bridging methodological gaps (lab-to-crowd).

7. Impact and Future Research Directions

The challenge demonstrates that non-intrusive DNN models can now robustly predict MOS for online conferencing speech with RMSE_MAP below 0.25 and PCC exceeding 0.9 on truly blind, real-world data. Crucially, independently generated blind evaluation sets are necessary to prevent overfitting and evaluate generalization.

Identified future research avenues include:

• Causal architectures for real-time monitoring
• Multi-dimensional quality prediction (beyond overall MOS, to include dimensions such as noisiness, coloration)
• Unsupervised or semi-supervised adaptation to novel codecs, noise, and room acoustics
• Improved uncertainty estimation for single-pass MOS predictions

All training and evaluation resources, including corpora and baselines, were open-sourced to further the community's research efforts (Yi et al., 2022).