---
title: ConferencingSpeech 2022 Challenge
url: https://www.emergentmind.com/topics/conferencingspeech-2022-challenge
type: topic
---

# ConferencingSpeech 2022 Challenge

The ConferencingSpeech 2022 Challenge is a competitive benchmark targeting the objective assessment of speech quality for online conferencing applications, emphasizing non-intrusive deep neural network (DNN) models that predict subjective quality directly from the impaired signal. The challenge introduces a large-scale multilingual dataset with high diversity in signal degradations and aims to establish rigorous evaluation methodologies, open-source baselines, and reproducible comparisons across state-of-the-art approaches [2203.16032].

## 1. Motivation and Context

Online conferencing (e.g., Zoom, Teams, Webex) has rendered high-quality remote speech communication essential. Real-world deployment is impeded by degradations such as background noise, reverberation, codec distortion, packet loss, jitter, and artifacts from noise suppression. Gold-standard speech quality assessment relies on subjective listening tests (ITU-T P.800/P.808), which are accurate but costly and impractical for customer data or real-time feedback. Full-reference objective measures (e.g., PESQ, POLQA) require a clean reference signal that is typically unavailable in deployment settings. Existing single-ended methods (e.g., ITU-T P.563) are inadequate for wideband or super-wideband speech common in modern conferencing.

The challenge motivates the development and benchmarking of non-intrusive, no-reference DNN models—specifically those predicting mean opinion scores (MOS) directly from impaired signals—for the robust and scalable evaluation of speech quality in contemporary conferencing scenarios [2203.16032].

## 2. Corpus Construction

The training and evaluation corpus for the challenge was curated to maximize diversity of language, degradation, and acoustic conditions.

### Training/Development Corpus

- **Volume**: ≈ 86,000 speech samples (≈ 200 hours)
- **Languages**: Chinese, English, German
- **Source Corpora**:
  - Tencent Corpus: ≈ 10,000 non-reverberant, 4,000 reverberant/reverberated
  - NISQA Corpus: ≈ 14,000
  - PSTN Corpus: filtered Librivox (≈ tens of thousands)
  - IU Bloomington Corpus: ≈ 36,000, used as auxiliary/reference

**Impairment types** include:
- Additive noises (white, non-stationary, DNS challenge)
- Filters (high/low-pass), amplitude clipping
- Codec degradations (AMR, Opus), packet loss concealment (PLC), noise suppression
- Room reverberation (simulated and real), clean speech examples

For example, in the Tencent Corpus (non-reverberant branch): 60% clips exhibit non-stationary noise, 10% white noise, 5% codecs, 3.75% filtering, with combinatorial mixes (e.g., 60% with only first-step impairment, 10% add noise suppression).

### Blind Test Sets

- **Size**: 4,372 clips
- **Subsets**:
  - Tencent blind: 20% of Tencent corpus
  - PSTN: 5% of PSTN samples
  - TUB: new subset (865 clips × 62 degradation conditions ≈ 4300 clips)

Each subset maintains the training corpus's diversity in language and degradation, but with unseen speakers and conditions [2203.16032].

## 3. Subjective Annotation Protocol

Crowdsourced absolute category rating (ACR) was conducted according to ITU-T P.808. Participants underwent qualification and modified training before rating clips. Each sample received at least 24 listener ratings, with rigorous cleaning to ensure ≥ 20 valid votes per clip (the PSTN training/test and NISQA corpora had lower vote counts as defined).

MOS was annotated on the standard 1–5 scale; some auxiliary references (IU Bloomington) used a 0–100 scale. The final MOS per clip was computed as the arithmetic mean of valid votes [2203.16032].

## 4. Evaluation Methodologies

Model outputs $\hat{y}$ are mapped to the subjective MOS scale using a monotonic cubic polynomial:

\[
\hat{y}' = a + b\hat{y} + c\hat{y}^2 + d\hat{y}^3
\]

Mapping coefficients are chosen to minimize MSE and ensure monotonicity.

Key metrics:

| Metric       | Definition / Formula                                                       | Significance                                    |
|--------------|----------------------------------------------------------------------------|--------------------------------------------------|
| RMSE         | $\sqrt{\frac{1}{N}\sum_{i=1}^N (\mathrm{MOS}_i - \hat{\mathrm{MOS}}_i)^2}$ | Absolute accuracy of MOS prediction              |
| PCC          | Pearson correlation between prediction and truth                           | Linear association of predicted and true MOS     |
| SRCC         | $\rho = 1 - \frac{6\sum_i (R(x_i)-R(y_i))^2}{N(N^2-1)}$                   | Correlation of prediction and truth ranks        |
| Outlier Ratio (OR) | Fraction of clips with $|\mathrm{MOS}_i - \hat{\mathrm{MOS}}_i|$ exceeding MOS's 95% CI      | Robustness to extreme errors                     |

This framework standardizes comparisons, accounting for scaling differences and statistical bias between lab and crowdsourcing settings [2203.16032].

## 5. Baseline and Submitted Systems

### Baseline Models

- **Baseline 1 (“NISQA-Lite”)**:
  - Input: 48-band log-mel spectrogram, context of ±7 frames (~150 ms)
  - CNN encoder, FFN layers, LSTM, temporal pooling, scalar MOS
  - MSE loss
  
- **Baseline 2 (“NISQA full”)**:
  - Input: same log-mel features
  - CNN block, self-attention pooling, MLP, scalar MOS
  - End-to-end training, MSE loss

Both baselines provide reproducible reference points; Baseline 2 employs self-attention pooling.

### Participating Systems

18 teams submitted non-intrusive DNN models. While specific models are not individually detailed in [2203.16032], high-performing trends included:

- Enhanced CNN/LSTM or hybrid CNN+Transformer/self-attention architectures
- Multi-scale feature extraction (diverse STFT windows)
- Feature fusion: log-mel, MFCC, self-supervised embeddings (wav2vec 2.0, CPC)
- Multi-task learning (predicting auxiliary attributes like noisiness, reverberation)
- Data augmentation (synthetic reverberation, new noise combinations)
- Loss functions: primarily MSE, some with Huber or rank-based losses

This suggests robust generalization depended on both architectural advances and training data heterogeneity.

## 6. Results and Comparative Analysis

- All 18 teams outperformed Baseline 1 (RMSE_MAP ≈ 0.55).
- 11 teams surpassed Baseline 2 (RMSE_MAP ≈ 0.28).

**Top result**: Team #1 achieved RMSE_MAP ≈ 0.24–0.26 and PCC ≈ 0.90–0.94 across blind test sets.

### Blind-Set Performance (Best Systems)

| Subset         | RMSE_MAP | PCC   |
|----------------|----------|-------|
| Tencent Corpus | ≈ 0.23   | ≈0.96 |
| TUB Corpus     | ≈ 0.25   | ≈0.90 |
| PSTN Corpus    | ≈ 0.26   | ≈0.92 |

Self-attention pooling consistently surpassed LSTM or mean pooling. Systems integrating hand-crafted and learned features attained lower RMSE. Training on corpora with diverse degradations was critical; polynomial mapping was essential for bridging methodological gaps (lab-to-crowd).

## 7. Impact and Future Research Directions

The challenge demonstrates that non-intrusive DNN models can now robustly predict MOS for online conferencing speech with RMSE_MAP below 0.25 and PCC exceeding 0.9 on truly blind, real-world data. Crucially, independently generated blind evaluation sets are necessary to prevent overfitting and evaluate generalization.

Identified future research avenues include:
- Causal architectures for real-time monitoring
- Multi-dimensional quality prediction (beyond overall MOS, to include dimensions such as noisiness, coloration)
- Unsupervised or semi-supervised adaptation to novel codecs, noise, and room acoustics
- Improved uncertainty estimation for single-pass MOS predictions

All training and evaluation resources, including corpora and baselines, were open-sourced to further the community's research efforts [2203.16032].

Source: https://www.emergentmind.com/topics/conferencingspeech-2022-challenge