MUSHRA-1S: Single-Stimulus Speech Evaluation

• MUSHRA-1S is a subjective speech quality evaluation protocol that combines the high sensitivity of traditional MUSHRA with the scalability and simplicity of ACR in a single-stimulus format.
• It employs a design with fixed low-quality anchors and high-quality references to minimize cognitive load and ensure consistent, calibrated ratings from listeners.
• The protocol enables fine-grained discrimination of high-quality speech systems and supports massively parallel evaluations, making it ideal for extensive benchmarking campaigns.

MUSHRA-1S (Single-Stimulus) is a subjective speech quality evaluation protocol that synthesizes the high sensitivity of the ITU-R BS.1534 MUSHRA method with the scalability and operational simplicity of Absolute Category Rating (ACR). MUSHRA-1S addresses critical limitations of standard multi-stimulus MUSHRA protocols—notably, their limited scalability due to cognitive load—by isolating the evaluation of each system under test (SUT) per trial and employing constant anchor and reference signals for context standardization. This methodology enables fine-grained system-level discrimination, particularly in the high-quality regime, while facilitating unbounded throughput suitable for modern benchmarking campaigns (Lechler et al., 23 Sep 2025).

1. Protocol Architecture and Test Workflow

In each MUSHRA-1S trial, listeners are presented with precisely three audio players: the SUT output, a fixed low-quality anchor (e.g., Opus encoded at 6 kbps), and a fixed high-quality reference (clean wide-band speech, typically 16 or 24 kHz). Only the SUT is rated using a continuous slider mapped to [0,100], with explicit anchoring: 0 corresponds to "Anchor quality" and 100 to "Reference quality." Hidden reference trials may be interleaved to assess rater consistency. Unlike standard MUSHRA, which simultaneously displays multiple systems, MUSHRA-1S presents one system per trial, thereby minimizing the cognitive complexity per judgment and eliminating direct inter-system contrast effects (Lechler et al., 23 Sep 2025).

In contrast to ACR—where a 1–5 discrete scale is used and no explicit reference/anchor is present—MUSHRA-1S maintains a continuous, contextually calibrated scale. This design preserves high sensitivity to subtle degradations and prevents the rating curve from saturating as system quality approaches that of the reference.

2. Rating Scale, Normalization, and Statistical Estimation

Listeners assign raw scores $s_{i,j}$ for file $i$ and listener $j$, constrained to $[0,100]$. Per-file mean scores are computed as

$\mu_i = \frac{1}{J} \sum_{j=1}^J s_{i,j}$

with $J$ ratings per file. Normalization proceeds by referencing to baseline anchor and reference means ($\mu_{anchor}$, $\mu_{ref}$) derived from a standard MUSHRA test:

$$Q_i = 100 \cdot \frac{\mu_i - \mu_{anchor}}{\mu_{ref} - \mu_{anchor}}$$

Alternate mapping to an absolute MOS scale can be obtained by replacing ($\mu_{anchor}$, $i$0) with constants or initial calibration means.

Confidence intervals for $i$1 are estimated via the standard t-distribution:

$i$2

where

$i$3

This formulation enables reliable reporting of mean system scores and uncertainty quantification in large-scale deployments (Lechler et al., 23 Sep 2025).

3. Anchor and Reference Selection: Mitigating Range Equalization Bias

Anchor selection is critical: the anchor is chosen to fall just below the lowest anticipated SUT quality (e.g., Opus at 6 kbps for mid-quality, 9 kbps for high-quality conditions), while the reference is set to an "ideal" clean wide-band speech file. By globally fixing both anchor and reference across all pages, the protocol enforces a common frame of reference for all raters across all systems and trials. This strategy directly combats the range-equalizing bias observed in ACR designs, where listeners adjust their use of a 1–5 scale to the presented system range, introducing rating drift and inconsistencies (Lechler et al., 23 Sep 2025).

4. Sensitivity, Scalability, and Statistical Power

Sensitivity ($i$4) is defined as the inverse of the smallest reliably detectable difference ($i$5) between two systems at a given statistical power ($i$6, typically 0.8) and significance ($i$7). In practice:

$i$8

Scalability is defined as the maximal number of systems ($i$9) evaluable per campaign at constant per-file listener cost. Standard MUSHRA’s cognitive load grows with $j$0 systems shown per page, limitating $j$1 to approximately 5–7. In MUSHRA-1S, as each system is evaluated in isolation, $j$2 becomes unbounded: per-system rater fatigue remains constant, allowing for arbitrary scaling in massive benchmarking campaigns (Lechler et al., 23 Sep 2025).

5. Experimental Validation and Comparative Performance

Crowdsourced MUSHRA-1S protocols utilized 100 clean, gender/accent-balanced speech files (adult/child, LRAC-2025 blind test set) rated by Prolific listeners (English L1, no hearing issues, $j$398% approval, $j$4110 prior studies). MUSHRA and MUSHRA-1S each used $j$5 ratings per file; ACR, $j$6 plus gold/catch references per ITU-P.808.

Key comparisons are as follows (MOS normalized to [0,100]):

Experiment	Max Δ(MUSHRA–M1S)	Max Δ(MUSHRA–ACR)	M1S vs. MUSHRA significant differences	ACR vs. MUSHRA significant differences	Typical 95% CI (M1S)	Typical 95% CI (ACR)
1a	3.38	~5–7	Only Opus LACE	All conditions	±2.5	±5.0
1b	0.53	~6–8	None (except B–C not discriminated)	All conditions	±2.0	±6.0
2a	1.4	~7–9	All except Bigcodec	All except ESC	±2.5	±7.0
2b	0.2	~8–9	None	All except DAC	±2.0	±8.0

These results establish that MUSHRA-1S closely tracks standard MUSHRA sensitivity across a range of quality regimes, especially in crowded high-end scenarios where ACR saturates and fails to distinguish adjacent systems (Lechler et al., 23 Sep 2025).

6. Implementation and Parallelization

MUSHRA-1S is amenable to highly parallelized deployment. For each SUT and test file, listener assignments are randomized. The core steps are:

1. Audio players for reference, anchor, and SUT output are loaded per page.
2. Listeners are instructed to rate only the SUT on a 0–100 continuous scale.
3. Slider scores are collected and aggregated.
4. Per-file scores are normalized as per the baseline MUSHRA anchor/reference statistics.

Final system scores and confidence intervals are published as:

$j$7

This process is readily parallelizable, allowing for simultaneous evaluation of hundreds of systems with minimal cognitive burden per rater and efficient data collection (Lechler et al., 23 Sep 2025).

7. Limitations, Contemporary Variants, and Best Practices

Anchor setting remains a sensitive variable; anchors set too distant from the SUT cluster (either too good or too bad) can induce non-linear anchoring effects. Pre-testing anchor candidates or using a dual-anchor scheme is advised for particularly broad system spreads. Explicit instruction is necessary to ensure raters score only the SUT.

By eliminating direct system–system ranking on identical pages, MUSHRA-1S forgoes contextual rank-contrast effects intrinsic to classic MUSHRA; however, for large portfolios the resulting throughput advantage is substantial. Future directions include automated anchor calibration, dynamic anchoring, and cross-calibration to MOS prediction models.

Related research on reference-induced bias and rater ambiguity has led to proposals that MUSHRA-1S can incorporate MUSHRA-NMR ("No Mentioned Reference") to remove reference-matching artifacts and MUSHRA-DG ("Detailed Guidelines") for reducing rater variance via structured error templates. These variants, originally assessed in the multi-stimulus MUSHRA setting, are directly transferrable to the MUSHRA-1S paradigm and, when combined, yield single-stimulus protocols that are free from reference bias, exhibit reduced inter-rater variance, and facilitate actionable feedback (Varadhan et al., 2024).

8. Summary and Positioning

MUSHRA-1S constitutes a robust, reference-anchored, single-stimulus subjective evaluation protocol optimized for top-tier speech system assessment. It bridges the granularity and top-end discrimination of standard MUSHRA with the scalability and simplicity of ACR, enabling statistically powerful, cost-effective, and large-scale benchmarking in both academic and industrial speech processing contexts (Lechler et al., 23 Sep 2025).