---
title: Speech Restoration Challenge
url: https://www.emergentmind.com/topics/speech-restoration-challenge
type: topic
---

# Speech Restoration Challenge

Searching arXiv for the challenge paper and closely related speech restoration work.
The Speech Restoration Challenge denotes a class of benchmark tasks in which a model must recover high-quality, intelligible speech from signals corrupted by multiple interacting degradations rather than a single impairment in isolation. In the formulation introduced in “The CCF AATC 2025: Speech Restoration Challenge” [2509.12974], the task explicitly targets composite distortions arising from acoustic conditions, codec processing, and secondary artifacts introduced by prior enhancement systems. This framing places the challenge within a broader shift from single-task speech enhancement toward universal or general speech restoration, as seen in systems such as VoiceFixer [2109.13731], SelfRemaster [2203.12937], Miipher-2 [2505.04457], and Sidon [2509.17052], all of which address restoration under mixed or open-world degradations with different architectural assumptions.

## 1. Definition and problem formulation

In the CCF AATC 2025 formulation, the degraded input $\hat{x}$ is produced by a three-stage degradation operator
$$
\hat{x} = D_3 \circ D_2 \circ D_1(x),
$$
where $x$ is clean speech and $D_1$, $D_2$, and $D_3$ correspond to acoustic, codec, and secondary processing degradations, respectively [2509.12974]. The stated motivation is that real-world speech pipelines often incur multiple, interacting distortions: a single utterance may be recorded in reverberant, noisy environments, then passed through low-bitrate codecs, and finally further “enhanced” by imperfect onboard algorithms, compounding artifacts [2509.12974].

This definition departs from traditional enhancement settings that isolate denoising, dereverberation, declipping, or codec artifact removal. The challenge description explicitly contrasts this composite task with prior single-degradation challenges such as DNS Challenge for denoising and REVERB Challenge or ICASSP dereverberation tasks, arguing that real signals rarely suffer only one degradation type [2509.12974]. A plausible implication is that the task is intended to measure robustness under non-linear cascades of distortions rather than performance on any one canonical corruption.

The broader literature uses related terminology. VoiceFixer defines “general speech restoration” as learning a single restoration function for combinations of additive noise, reverberation, low-resolution distortion, and clipping [2109.13731]. SelfRemaster frames restoration as inversion of an explicit recording process without paired clean–distorted corpora [2203.12937]. More recent work such as Miipher-2 emphasizes conditioning-free multilingual restoration at million-hour scale [2505.04457], while Sidon treats restoration as multilingual dataset cleansing with a two-stage parametric resynthesis pipeline [2509.17052]. These systems suggest that the Speech Restoration Challenge sits at the intersection of robustness, generative modeling, and data-centric speech pipeline design.

## 2. Composite degradation model

The challenge partitions degradation into three categories. The first, complex acoustic degradations $D_1$, includes Room Impulse Responses drawn from a library to simulate reverberation, microphone saturation simulated via hard clipping, low-pass filtering to mimic limited system bandwidth, and stationary and non-stationary noise such as babble, traffic, and white noise added at multiple SNR levels [2509.12974]. The second, signal-chain artifacts $D_2$, consists of MP3 compression and decompression using FFmpeg at varying bitrates such as 16 kbps–128 kbps, introducing spectral “pre-echo” and “burbly” noise typical of low-bitrate lossy codecs [2509.12974]. The third, secondary artifacts $D_3$, passes already degraded signals through ten off-the-shelf enhancement models to simulate residual processing artifacts [2509.12974].

The ten secondary processors are divided into discriminative baselines and generative baselines. The discriminative baselines are Demucs, FRCRN, VoiceFixer, NSNet2, and TF-GridNet. The generative baselines are Sgmse+, Storm, AnyEnhance, MaskSR, and LLASE-G1 [2509.12974]. Because these models are used as degradation generators rather than only as comparison points, the challenge treats enhancement artifacts themselves as part of the corruption process. This is a notable design choice: many restoration benchmarks assume a clean target and a physically degraded input, whereas here model-induced artifacts are explicitly embedded in the data-generation chain.

Several related works illuminate why this matters. MaskSR2 shows that generative full-band restoration can attain high quality but still benefit substantially from explicit semantic guidance to improve intelligibility, with WER reduced from 4.74% to 3.18% when scaling from MaskSR-L to MaskSR2-L (Avg-feature) on the simulated VCTK full-band test [2409.09357]. RestSE argues that lightweight SE modules can over-suppress speech, motivating a progressive SE-plus-codec design [2410.01150]. ReverbMiipher shows that conventional SR may completely remove reverberation, even though reverberation encodes spatial information regarding the acoustic source environment [2505.05077]. Together, these results suggest that “restoration” is not a single objective; preserving, removing, or regenerating signal attributes depends on task definition.

## 3. Dataset construction and multilingual scope

The clean speech sources for the challenge are VCTK, AISHELL-3, and EARS [2509.12974]. VCTK provides multi-speaker English speech with various accents, AISHELL-3 provides multi-speaker Mandarin Chinese, and EARS provides anechoic, fullband English recordings [2509.12974]. Each clean utterance is processed through $D_1 \rightarrow D_2 \rightarrow D_3$ to produce a degraded counterpart [2509.12974].

The dataset creation pipeline specifies randomized perturbations at each stage. Reverberation is applied by convolving the clean signal with a randomly selected RIR. Clipping thresholds are varied to simulate different microphone headrooms. The low-pass filter cutoff is randomly varied in a predetermined band such as 3–7 kHz. Noise SNR levels are sampled uniformly between 0 dB and 20 dB using multiple noise types. MP3 bitrates are uniformly sampled, for example 16, 32, 64, and 128 kbps. Each utterance is then processed by one of the ten enhancement systems used for secondary processing [2509.12974].

While the paper does not enumerate exact counts, it states that the training set comprises tens of thousands of utterances drawn from all three corpora, ensuring diversity in speakers, languages, recording conditions, and degradation permutations; a dev set of several thousand utterances is held out for leaderboard scoring; and a blind test set of comparable size and composition is reserved for final evaluation [2509.12974]. This bilingual English–Mandarin construction is narrower than large multilingual restoration systems, but it still extends beyond monolingual benchmark design.

A comparison with recent restoration datasets clarifies the design space. Miipher-2 is trained on 3,195 h of studio-quality speech spanning 44 languages and 1,642 speakers, with synthetic degradation formed by reverberation, additive noise, and optional codec artifacts [2505.04457]. Sidon trains on approximately 2,219 h covering 104 languages, with a noising pipeline that applies reverberation, background noise, bandlimiting, clipping, MP3 codec, and packet loss in random order, yielding approximately 9,000 h of paired data [2509.17052]. By contrast, the Speech Restoration Challenge emphasizes a more tightly specified but operationally realistic cascade, especially through its use of secondary enhancement artifacts [2509.12974].

## 4. Evaluation protocol and ranking procedure

The evaluation protocol is divided into a preliminary round and a final round [2509.12974]. In the preliminary round, objective metrics account for 40 points total and are equally weighted: Word Accuracy for intelligibility, measured via an automatic speech recognizer; DNSMOS for non-intrusive perceptual quality estimation; and PESQ (ITU-T P.862.2) for waveform fidelity [2509.12974]. Model complexity contributes 20 additional points via parameter-count tiers: fewer than 10 M parameters receives 20 points, 10–20 M receives 16 points, 20–50 M receives 12 points, 50–100 M receives 10 points, and more than 100 M receives 8 points [2509.12974]. Teams are ranked on the composite objective score, with a maximum of 60 points, and assigned to four performance tiers [2509.12974].

In the final round, the top six teams from preliminaries advance [2509.12974]. The objective score carries over, and a 40-point subjective score is added. This subjective component includes Solution Innovation worth 20 points, judged by novelty and algorithmic contributions, and a Formal Listening Test worth 20 points based on MOS tests with four performance tiers [2509.12974]. A technical presentation and defense before an expert panel finalizes the rankings [2509.12974].

The challenge therefore combines signal quality, intelligibility, efficiency, and expert assessment rather than reducing ranking to a single metric. This differs from challenges that rely only on ASR performance or only on subjective listening. For example, the Helsinki Speech Challenge 2024 uses Character Error Rate from Mozilla DeepSpeech v0.9.3 as its official quantitative metric, although PESQ and STOI are recommended as supplementary metrics [2406.04123]. KS-Net, which won the ICASSP 2024 SSI Challenge, reports ITU-T P.804 MOS and Word Accuracy Rate as headline evaluation measures [2402.01808]. Gesper, which ranked first in both tracks of the ICASSP 2023 SSI Challenge, was evaluated with ITU-T P.835, P.804, DNSMOS, and NISQA [2306.08454]. The Speech Restoration Challenge inherits this multi-axis evaluation logic but adds an explicit model-complexity score [2509.12974].

## 5. Baselines and system paradigms

The official baseline is AnyEnhance V1, described as a non-causal masked generative model with two stages: semantic enhancement, which restores high-level content, and acoustic enhancement, which refines waveform details [2509.12974]. It is pre-trained on the full training pipeline, and its code and models are released through the cited repository [2509.12974]. On the validation set, the paper reports illustrative baseline figures of PESQ around 2.6, DNSMOS around 3.4, WAcc around 80%, and a parameter count of 15 M, corresponding to a complexity score of 16/20 [2509.12974]. The listed strengths are joint handling of complex distortions and competitive PESQ and DNSMOS, while weaknesses are that the non-causal design limits real-time deployment and that word accuracy can still lag on severe degradations [2509.12974].

The challenge’s degradation generators and the wider literature expose several distinct architectural paradigms:

| Paradigm | Representative systems | Defining property |
|---|---|---|
| Analysis–synthesis / parametric resynthesis | VoiceFixer, Miipher-2, Sidon, ReverbMiipher | Predict clean intermediate features, then vocode |
| Generative token or codec restoration | MaskSR2, RestSE | Restore via masked acoustic modeling or generative codec |
| Direct complex-domain restoration | Gesper, KS-Net | Restore complex STFT directly |
| Self-supervised or self-supervised-adapted restoration | SelfRemaster, Sidon | Use real degraded data or frozen SSL backbones |

VoiceFixer separates an analysis stage predicting mel-spectrogram features from a synthesis stage implemented by a neural vocoder [2109.13731]. Miipher-2 uses a frozen Universal Speech Model as a conditioning-free feature extractor, parallel adapters for feature cleaning, and a modified WaveFit vocoder, achieving a real-time factor of 0.0078 on a single TPU v4i [2505.04457]. Sidon similarly adopts a two-stage “parametric resynthesis” flow with a w2v-BERT 2.0 feature predictor and a HiFi-GAN vocoder, and reports inference approximately 3,390 times faster than real time on a single NVIDIA H200 [2509.17052]. ReverbMiipher extends this paradigm by adding a dedicated ReverbEncoder and reverb-feature control [2505.05077].

Alternative designs remain competitive. MaskSR2 keeps the same inference-time compute and model size as MaskSR while using semantic knowledge distillation during training to improve intelligibility [2409.09357]. Gesper performs restoration then enhancement in the complex STFT domain and achieves 3.27 P.804 overall MOS and 3.35 P.835 overall MOS on the blind test set of the ICASSP 2023 SSI Challenge [2306.08454]. KS-Net uses a complex-domain GAN followed by fine-grained multi-band fusion and achieves an overall P.804 MOS of 3.49 and a WAcc of 0.78 for the real-time track of the ICASSP 2024 SSI Challenge [2402.01808]. This suggests that the Speech Restoration Challenge is not architecturally prescriptive; the task can be approached through vocoder-based resynthesis, token generation, complex spectral mapping, or hybrid cascades.

## 6. Research significance, limitations, and emerging directions

The challenge is presented as a step toward universal, “all-in-one” restoration systems that can generalize across noise, reverberation, codec artifacts, and residual processing distortions [2509.12974]. That emphasis aligns with a broader research trajectory. VoiceFixer argued early for general speech restoration rather than separate SSR systems [2109.13731]. SelfRemaster addressed the domain-mismatch problem of synthetic paired training by using an analysis-by-synthesis framework trained with real degraded speech data in a self-supervised manner [2203.12937]. Miipher-2 and Sidon then reframed restoration as a scalable infrastructure problem for cleaning training corpora for ASR, TTS, and LLM-adjacent data pipelines [2505.04457] [2509.17052].

A recurring tension in this literature concerns fidelity versus intelligibility, and suppression versus regeneration. MaskSR2 explicitly demonstrates that generative restoration quality does not guarantee optimal intelligibility without semantic guidance [2409.09357]. RestSE argues that in challenging environments traditional SE can create over-suppressed speech and harm downstream tasks, motivating progressive restoration through SE and codec modules [2410.01150]. ReverbMiipher introduces a further complication: removing reverberation can improve classical enhancement scores while destroying spatial characteristics that may be desirable to preserve [2505.05077]. This suggests that “better restoration” depends on whether the target is anechoic speech, perceptually improved speech, studio-quality speech, or speech that preserves some environmental attributes.

Another emerging direction is multilingual and cross-domain generalization. Sidon covers dozens of languages and is designed for large-scale dataset cleansing [2509.17052]. Miipher-2 supports over 300 languages through a frozen USM backbone [2505.04457]. DisSR, a later cross-domain diffusion-based approach, introduces speaker-invariant degradation priors and hierarchical MMD alignment to improve adaptability on unseen speakers and languages [2602.12701]. Although DisSR is not part of the challenge design itself, it suggests one route for systems attempting to generalize beyond the bilingual English–Mandarin data of the competition.

The official organization details underscore the challenge’s role as a community benchmark. The timeline covers dataset release and baseline code publication from February to May 2024, preliminary submission in June 2024, finalist announcement in July 2024, and final submissions, listening tests, and technical presentations in August 2024 [2509.12974]. Submissions consist of restored waveforms and model metadata such as parameter count and run-time specifications; automated scoring scripts and submission instructions are provided in the open-source toolkit; and leaderboard updates occur weekly during preliminaries [2509.12974]. In this sense, the Speech Restoration Challenge is both a benchmark specification and an organizational mechanism for consolidating work on composite, multi-stage speech degradation.

A plausible implication is that the challenge formalizes a transition already underway in speech restoration research: from narrowly defined enhancement tasks toward models evaluated simultaneously for intelligibility, perceptual quality, robustness to cascaded artifacts, and deployability. Whether future systems converge on universal parametric resynthesis, direct complex-domain mapping, diffusion-based conditional generation, or hybrid progressive pipelines remains unresolved, but the challenge makes that comparison experimentally tractable [2509.12974].

Source: https://www.emergentmind.com/topics/speech-restoration-challenge