MUSHRA-NMR: A Fair TTS Evaluation Protocol

• MUSHRA-NMR is a reference-fair TTS evaluation protocol that removes the explicit human reference label to prevent systematic underrating of advanced synthetic speech.
• The modified protocol allows synthetic outputs to score above the hidden human reference, aligning closely with pairwise CMOS evaluations and ensuring fairer comparisons.
• Empirical tests with native speakers in Hindi and Tamil demonstrated improved sensitivity and reduced bias, supporting robust benchmarking in TTS research.

MUSHRA-NMR (No Mentioned Reference) is a reference-fair human evaluation protocol for text-to-speech (TTS) systems, developed to address the inherent reference-matching bias present in the classical MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) paradigm. In standard MUSHRA, listeners compare multiple system outputs, a low-quality anchor, and an explicitly labeled human reference while rating perceived naturalness on a continuous scale. However, as TTS systems increasingly match or exceed human naturalness, the visibility of the “Reference” label artificially constrains rater judgments, leading to systematic underrating of advanced synthetic speech. MUSHRA-NMR eliminates this bias by omitting any mentioned reference during the evaluation, while otherwise retaining the established workflow and rigorous comparative structure of MUSHRA (Varadhan et al., 2024).

1. Classical MUSHRA and Its Limitations

The original MUSHRA protocol was designed to evaluate intermediate audio quality by anchoring system outputs against a known human reference. In each trial, listeners are presented with:

• The human reference, explicitly labeled as such (REF)
• A lower-quality anchor (ANC)
• Outputs from $N$ systems under test (e.g., FS2, VITS, ST2)

Raters score samples on a 0–100 perceived naturalness scale, with interpreted bands (Excellent: 80–100, Good: 60–80, etc.). However, modern high-fidelity TTS systems frequently approach or surpass the reference in certain perceptual qualities. Under these circumstances, listeners exhibit “reference-matching bias,” anchoring high scores to the reference label and compressing the rating range available to synthetic systems. This leads to the paradoxical outcome where truly superior synthetic samples cannot be rated above their reference exemplars, undermining reliable system ranking and progress tracking (Varadhan et al., 2024).

2. Protocol and Design of MUSHRA-NMR

MUSHRA-NMR is defined by a single core modification to the classic MUSHRA workflow: the “Reference” label is omitted entirely during stimuli presentation. Instead, each trial includes the hidden reference (HR), anchor (ANC), and $N$ TTS outputs, now shuffled and labeled generically (e.g., “Audio 1...Audio N+2”). Raters receive the following instruction: “Please rate each audio sample from 0 (Bad) to 100 (Excellent) on perceived naturalness. You will not be shown which sample is the human recording.”

• Stimuli structure: Hidden reference, anchor, $N$ system outputs, all unlabeled.
• Scoring: Raw scores $s_i$ on the uniform 0–100 scale, with no further nonlinearity or mapping function applied.
• Interpretation: Synthetic outputs can be assigned scores exceeding the (now-unlabeled) human sample if judged to possess higher perceived naturalness.

This structure directly addresses reference-matching bias by removing the explicit cognitive anchor (Varadhan et al., 2024).

3. Empirical Validation and Key Metrics

A comprehensive evaluation of MUSHRA-NMR was conducted with $n=492$ native listeners (236 Hindi, 256 Tamil) across 100 utterances and 5 stimuli per trial. The protocol was implemented via the WebMUSHRA platform, employing robust controls for forced listening, fatigue, and event logging. Key findings include:

• Gap reduction: In Tamil, the mean score ($\mu$) of the top TTS system increased from $71.4$ (classic MUSHRA) to $76.6$ (MUSHRA-NMR), while the hidden reference scored $\mu=78.7$. Thus, the system-reference gap was halved relative to the classic protocol.
• Alignment with pairwise CMOS: Scores now more closely match those obtained by costly pairwise comparison (CMOS), with the leading system versus reference producing a mean difference of $+0.24$ on a $N$0 scale.
• Sensitivity: A Spearman correlation of $N$1 to full-scale system rankings is achieved with as few as 40 listeners or 20 utterances.

These results confirm that MUSHRA-NMR provides both fairer scores and comparable reliability to established protocols, while reducing systematic rating artifacts (Varadhan et al., 2024).

4. Comparative Outcomes and Statistical Properties

MUSHRA-NMR was evaluated alongside classical MUSHRA and an ambiguity-reduced variant (MUSHRA-DG), with the following mean ($N$2) and variance ($N$3) statistics reported for Hindi:

Protocol	FS2 ($N$4, $N$5)	ST2 ($N$6, $N$7)	VITS ($N$8, $N$9)	REF ($N$0, $N$1)
Classic	64.2, 22.9	66.7, 21.7	67.7, 20.6	84.2, 15.5
NMR	62.0, 23.9	68.1, 22.0	68.8, 21.0	76.4, 18.1
DG	72.7, 11.7	73.4, 12.0	75.6, 11.0	90.9, 9.3

• Inference: The NMR protocol enables system scores to rise relative to the hidden reference, directly correcting the reference-matching bias, while maintaining statistical rigor in rankings.
• A plausible implication is that MUSHRA-NMR better supports comparative benchmarking as TTS systems approach or surpass human-level perceptual quality.

5. Broader Context: Integration with Fine-Grained and Reference-Free Evaluation

While MUSHRA-NMR neutralizes explicit reference bias, it does not resolve all reliability issues related to rater ambiguity. The ambiguity-reduced “MUSHRA-DG” variant introduces detailed guidelines, scoring rubrics, and explicit fault counts (e.g., mild/severe mispronunciation, digital artifacts, speed changes), resulting in a composite score:

$N$2

where $N$3 (liveliness), $N$4 (voice quality), and $N$5 (rhythm) are rated on 0–100 scales, and fault events are penalized accordingly.

Combining MUSHRA-NMR’s reference fairness with MUSHRA-DG’s detailed guidance yields further reduced variance ($N$6), maintains correct system ranks, and provides actionable diagnostics for system improvement (Varadhan et al., 2024).

6. The MANGO Dataset and Implications for TTS Evaluation

The MANGO (“MUSHRA Assessment using Native listeners and Guidelines Opinions”) dataset accompanies this work as the first large-scale, systematically rated corpus for Indian language TTS. Comprising 246,000 human ratings (492 listeners × 100 utterances × 5 stimuli), MANGO includes raw scores, rubric marks, system and trial metadata, and session logs for both Hindi and Tamil.

• Structure: Raw CSVs with per-listener and per-system scores, variant labels (Original, NMR, DG, DG-NMR), CMOS benchmarks, and rich demographic annotations.
• Purpose: Facilitates analysis of human rater behavior, benchmarking of system advances, and the development of reference-free, fine-grained automatic metrics for TTS (Varadhan et al., 2024).

7. Significance and Future Directions

By eliminating explicit reference anchoring, MUSHRA-NMR enables unbiased, future-proof evaluation as TTS systems continue to improve. Its adoption creates a more level benchmarking landscape, supporting claims of superhuman synthesis where warranted by subjective evaluation. In conjunction with datasets such as MANGO and fine-grained variants like MUSHRA-DG, reference-fair protocols constitute an essential foundation for robust, transparent, and reproducible TTS evaluation, especially in contexts where classical human-language references may not represent an upper bound on system quality (Varadhan et al., 2024).