ITU-T P.808: Crowdsourced Speech Quality

• ITU-T P.808 is a crowdsourcing framework that uses Absolute Category Rating to assess speech quality on a large scale.
• It integrates listener qualification, environment checks, training, and reliability screening to replicate and extend lab-based evaluations online.
• Empirical validations demonstrate high correlation with traditional laboratory measures and robust reproducibility in speech-enhancement benchmarks.

ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. In the research literature, it is treated as the standardized crowdsourcing counterpart to laboratory-based ITU-T P.800 testing: it structures listener qualification, environment and device checks, training, reliability control, and Mean Opinion Score (MOS) aggregation for large-scale online listening studies, and it has been used for open-source test systems, dataset curation, benchmark ranking, and the training of non-intrusive perceptual predictors such as DNSMOS (Naderi et al., 2020, Reddy et al., 2020, Reddy et al., 2021).

1. Position within subjective speech-quality evaluation

P.808 emerged from a practical mismatch between the perceptual target of speech-quality assessment and the operational constraints of modern experimentation. The laboratory procedures standardized in ITU-T P.800 provide controlled listening-opinion tests, but they are difficult to scale to large, diverse, and realistic test sets. By contrast, much speech-enhancement work had relied on objective measures such as PESQ, POLQA, ViSQOL, and SDR on synthetic mixtures, even though the DNS Challenge papers explicitly argue that such metrics do not correlate well with subjective tests and that performance on synthetic mixtures often degrades on real recordings (Reddy et al., 2020, Reddy et al., 2020).

Within that setting, P.808 is presented as the mechanism that makes large-scale perceptual evaluation operational. It allows the same subjective criterion—listener-rated quality—to be applied to both synthetic and real recordings, including cases where no clean reference exists. This was central to the INTERSPEECH 2020 Deep Noise Suppression Challenge, where P.808 was not an auxiliary validation method but the official mechanism for selecting winners on a blind test set containing both synthetic and real recordings (Reddy et al., 2020).

The literature also shows that P.808 is not limited to model ranking. In the DNS Challenge pipeline it was used to assess Librivox audiobook chapters by subjective quality: for each chapter, 10 randomly sampled 10-second clips were rated, chapter MOS was computed from those samples, and the upper quartile with $4.3 \le \mathrm{MOS} \le 5$ was selected as clean speech material (Reddy et al., 2020). P.808 therefore functions both as an evaluation protocol and as a perceptually grounded data-selection tool.

2. Operational structure and quality-control mechanisms

A P.808 listening session is typically described as having multiple phases: qualification, setup or environment check, training, and rating. In the multilingual URGENT 2025 work, qualification is done once per participant and is meant to ensure suitable hearing ability, equipment, and task comprehension; setup and environment checks verify playback conditions and a sufficiently quiet environment; training calibrates listeners to the expected quality range; and the rating phase collects the actual opinion scores. Setup and training are repeated at fixed intervals. After collection, ratings are subjected to reliability analysis; unreliable ratings may be discarded, and because P.808 requires at least eight ratings per stimulus, rejected items are resubmitted for further evaluation (Sach et al., 15 Jul 2025).

The open-source implementation described by Naderi and Cutler operationalizes these stages directly on Amazon Mechanical Turk. A master script takes URLs for stimuli, training material, trapping stimuli, and gold-standard stimuli, then generates the HTML HIT application, the input URL list, and the configuration file for post-processing. The generated task includes a hearing test, an environmental suitability test, a two-eared-headphone check, and trapping and gold-standard questions. Qualification is integrated into the first rating assignment rather than separated into an earlier stage, and the pass/fail result is stored in browser local storage. The same implementation introduced a 30-minute “temporal environment suitability certificate,” so a participant who passes the environment test need not repeat it on every assignment; this reduced overall participant working time by 40%, while integrated qualification reduced execution time by a factor of 4–5 in several test runs (Naderi et al., 2020).

The concrete checks used in published implementations are unusually rich for a crowdsourcing protocol. The multilingual URGENT description includes self-reported hearing information, a speech-in-noise transcription comprehension test, and a bandwidth test using speech mixed with bandpass-filtered white noise. Setup and environment checks include a playback-level test, a binaural test in which listeners transcribe alternating left/right digits, and a comparison test in which listeners must detect a small quality difference induced by just-noticeable noise. Training and rating jobs include “gold” clips and “trapping” clips. Gold clips have known extremal quality; trapping clips contain an explicit spoken instruction telling the worker which rating option to select (Sach et al., 15 Jul 2025).

A separate implementation-oriented refinement is the Just-Noticeable Difference of Quality (JNDQ) environment suitability test. Its laboratory staircase threshold is defined as

$$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$

where $t_{i,p}$ is the SNR at the $i^{\text{th}}$ reversal for participant $p$. For crowdsourcing, that full staircase was shortened to four pair-comparison questions. One proposed operational rule was that if the participant correctly answers $\frac{3}{4}$ of the questions, the participant passes the Environment Suitability Test (Naderi et al., 2020). This does not redefine P.808, but it shows how later work attempted to turn P.808’s environmental requirements into a short behavioral screen.

3. Validation, vote requirements, and reproducibility

The central empirical question around P.808 has been whether crowdsourced scores can approximate laboratory results closely enough for scientific comparison. In the open-source implementation paper, the authors validated their system against ITU-T Supplement 23, Experiment 3, a standard P.800 laboratory dataset. Averaged over ten repetitions of matched-condition sampling, they reported $PCC = 0.954$, $SRCC = 0.923$, and $RMSE = 0.237$, with RMSE reduced to $0.214$ after first-order mapping (Naderi et al., 2020). DNS Challenge validation work likewise reported Spearman correlations around $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$0 or $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$1 against Supplement 23 laboratory MOS, supporting the claim that P.808-style crowdsourcing can track lab judgments at condition level (Reddy et al., 2020, Reddy et al., 2020).

Reproducibility across repeated crowdsourcing runs has also been studied directly. In a five-run experiment on the DNS Challenge wideband test set, the open-source P.808 implementation reported average agreement between model MOS values of $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$2 and $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$3. Using the explicit hidden-reference differential score

$$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$4

the same study reported $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$5 for MOS and $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$6 for DMOS under a single-measure, absolute-agreement, two-way random model, which the authors described as good reliability for MOS and excellent reliability for DMOS (Naderi et al., 2020).

Filter design materially affects those outcomes. In the same reliability analysis, gold-stimulus and environment-test screening improved consistency across runs, whereas automated headset detection via WebRTC did not show measurable benefit. For “all criteria together,” the passed group had $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$7 and average $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$8, while the failed group had $$JND_p = 50 - T_p, \qquad T_p = \frac{1}{N-1}\sum_{i=2}^{N} t_{i,p},$$9 and $t_{i,p}$0 (Naderi et al., 2020). Later comparative work on crowdsourced DCR likewise found that raw P.808-style ratings preserved condition ordering fairly well but suffered from compressed scale usage and higher variance; postscreening based on gold-standard reference checks, rating span, and anchor ordering substantially improved agreement with P.800 (Treffehn et al., 26 Jun 2026).

A separate line of work addressed sample complexity. In a study explicitly framed as input to P.808, simulated resampling over three crowdsourcing experiments showed that validity and reliability curves flatten around roughly 60–100 votes per condition. The authors’ practical recommendation was to use at least 60 votes per test condition; if narrower confidence bounds are required, their confidence-interval analysis suggested about 115 votes per condition to keep MOS CI width below 0.3 (Naderi et al., 2020). This is one of the clearest operational recommendations attached to P.808-style study design.

4. Role in speech-enhancement benchmarks and MOS-based ranking

P.808 became especially prominent through community challenges in speech enhancement. In the INTERSPEECH 2020 DNS Challenge, it was used as the official blind-test evaluation protocol on a representative mix of synthetic and real recordings, and the challenge organizers released both the datasets and the P.808 framework in open-source form. In Phase 1, all 28 submissions plus the noisy test set were evaluated with 10 raters per clip, yielding a reported 95% confidence interval of 0.02 for each submission; in Phase 2, top systems were reevaluated and combined with Phase 1 ratings for 20 ratings per clip total, yielding a reported 95% confidence interval of 0.01 per model (Reddy et al., 2020).

The same challenge illustrates how P.808 scores enter ranking logic. The blind noisy test set had overall MOS 2.85. The winning non-real-time system achieved overall MOS 3.52 and overall dMOS 0.67, while the best real-time system achieved overall MOS 3.42 and dMOS 0.57. Statistical significance among top systems was assessed with ANOVA and pairwise $t_{i,p}$1-values, using $t_{i,p}$2 as the significance criterion; if systems were statistically indistinguishable, computational complexity was used as a tiebreaker (Reddy et al., 2020). P.808 therefore served not merely as a perceptual score source, but as the decision backend for challenge outcomes.

By ICASSP 2021, P.808 had become entrenched as the organizer-provided subjective criterion for DNS evaluation. In “ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network,” the authors state that they present “the subjective results of the submission with ITU-T P.808 criterion … which is provided by the organizer,” and that their system ranked top-1 for the real-time track in terms of MOS under the P.808 framework. The reported overall MOS values were 2.86 for Noisy, 3.21 for the NSnet2 baseline, and 3.38 for TSCN-PP, so the proposed system outperformed the baseline by 0.17 MOS and the noisy condition by 0.52 MOS (Li et al., 2021).

That paper is also instructive about what P.808 does not specify in downstream publications. The reported subjective table contains category-wise MOS for Singing, Tonal, Non-English, English, Emotional, and Overall, but the paper does not provide the number of raters, ratings per clip, crowdsourcing platform details, session structure, hidden references, confidence-interval computation, or aggregation procedure for the official test. P.808 there functions as the benchmark under which the challenge outcome is stated, while procedural specifics are delegated to the organizers (Li et al., 2021).

5. Relation to adjacent standards and to learned objective predictors

P.808 occupies a specific position in a broader standards ecosystem. P.800 remains the laboratory baseline. P.835 decomposes noise-suppressed speech into separate ratings for speech signal, background noise, and overall quality. P.804 introduces multi-dimensional perceptual dimensions such as noisiness, coloration, discontinuity, and loudness. Several papers treat P.808 not as a competing construct, but as the crowdsourcing process layer that can host these adjacent rating paradigms online (Naderi et al., 2020, Naderi et al., 2023).

Framework	Reported outputs	Reported role
P.800	ACR, CCR, DCR, lab MOS	Laboratory baseline
P.808	ACR overall MOS in crowdsourcing	Online process and quality control
P.835	SIG, BAK, OVRL	Noise-suppression decomposition
P.804	Noisiness, coloration, discontinuity, loudness	Multi-dimensional listening-phase diagnostics

The sharpest conceptual contrast in the literature is between P.808 and P.835. The DNSMOS P.835 paper states explicitly that the earlier DNSMOS model was trained using ITU-T Rec. P.808 subjective scores and that “the P.808 scores reflect the overall quality of the audio clip.” By contrast, P.835 gives the standalone quality scores of speech and background noise in addition to the overall quality. This is why DNSMOS P.835 was introduced: P.808-based DNSMOS predicts overall MOS only, whereas DNSMOS P.835 predicts SIG, BAK, and OVRL (Reddy et al., 2021).

The distinction also appears in data-centric modeling work. “Speech MOS multi-task learning and rater bias correction” uses 274,466 training samples rated following P.808, each with a single MOS label per file, and combines them with a P.835 dataset of 143,731 samples carrying OVR, SIG, and BAK labels. The paper does not claim equivalence between P.808 MOS and P.835 OVR/SIG/BAK, but shows that P.808-labeled data can improve prediction of richer P.835 dimensions through multi-task learning and semi-supervised pseudo-labeling (Akrami et al., 2022).

P.808 has also been extended operationally rather than replaced. One open-source toolkit follows P.835 for SIG, BAK, and OVRL while using P.808-style qualification, environment checks, headset checks, traps, gold questions, periodic retraining, and answer screening; another toolkit extends the P.808 codebase to a seven-dimensional framework comprising coloration, discontinuity, noisiness, loudness, reverberation, signal quality, and overall quality (Naderi et al., 2020, Naderi et al., 2023). These developments preserve the role of P.808 as the crowdsourcing infrastructure while broadening what can be measured within it.

6. Multilingual adaptation, screening challenges, and current limitations

Although the original open-source implementation is English-only, later work showed that P.808 can be localized systematically. In the URGENT 2025 multilingual speech-enhancement study, the localization pipeline consisted of manually transcribing all English audio instructions; translating all text and transcribed instructions into the target language; synthesizing target-language instructional audio with TTS; replacing recorded utterances used in qualification, setup, and training with real target-language speech from a disjoint database; asking workers for fluency in the target language rather than mother tongue; recreating the bandwidth test with lower cutoff frequencies of $t_{i,p}$3, $t_{i,p}$4, and $t_{i,p}$5; and recreating the comparison test with white Gaussian noise at SNRs of $t_{i,p}$6 and $t_{i,p}$7. In URGENT 2025, English used the original implementation and German, Chinese, and Japanese used localized variants. Each utterance received eight ratings, but acceptance rates after reliability analysis dropped sharply from 82.3% in English to 62.8% in German, 41.1% in Chinese, and 28.9% in Japanese during the first testing stage (Sach et al., 15 Jul 2025).

This multilingual work also exposed a deeper limitation of reference-free ACR MOS in the age of generative speech enhancement. In the same study, a generative system achieved overall MOS $t_{i,p}$8, the second-highest in the table, while also posting ESTOI $t_{i,p}$9 and $i^{\text{th}}$0, where

$i^{\text{th}}$1

The authors argue that reference-free objective metrics such as DNSMOS and NISQA, and even reference-free subjective ACR MOS, may fail to detect hallucinations when outputs sound natural but do not preserve the intended phonetic content. Their recommendation is to accompany P.808 ACR MOS with intelligibility or phone-fidelity metrics, especially LPS (Sach et al., 15 Jul 2025).

Listener population remains another nontrivial variable. In a P.835 study implemented in a crowdsourcing setting according to P.808, native English and native German listeners judged English, German, and Mandarin speech under noise and suppression distortions. After P.808-style screening, 86.7% of submissions in the English experiment and 74.5% in the German experiment were considered reliable. The main finding was that background-noise ratings were not affected by participants’ proficiency in the target language, whereas speech-signal ratings and consequently overall quality were affected in specific conditions (Naderi et al., 2020). A plausible implication is that multilingual P.808 studies cannot treat language matching as a mere recruitment convenience; it changes what is being measured on at least some scales.

More generally, later comparative screening work shows that not all reliability mechanisms contribute equally. In a P.808-versus-P.800 DCR codec study, pretests and questionnaires were weak predictors of agreement with laboratory results, while continuous and post hoc screening based on traps, minimum reference rating, rating span, and anchor ordering were much more effective. The authors therefore recommend a compact screening package centered on traps, gold-standard reference checks, rating span, and anchor ordering rather than generic outlier rules (Treffehn et al., 26 Jun 2026). This suggests that contemporary P.808 practice is increasingly interpreted as a family of implementation choices around a standard core, not a single frozen workflow.

Taken together, these studies define P.808 as a standardized crowdsourcing framework for subjective speech-quality evaluation whose strength lies in operational scalability, explicit screening, and reproducible MOS collection. They also show its boundaries: P.808 is strongest for overall perceptual assessment under large-scale online conditions, but its outputs are sensitive to screening design, listener population, language localization, and—especially for generative systems—to the difference between sounding good and preserving what was said.