UniPASE: Universal Speech Enhancement Model

• The paper demonstrates that UniPASE is a purely generative universal speech enhancement system that restores corrupted speech with low hallucination across diverse distortion types.
• It employs a dual-stream architecture using enhanced phonetic and acoustic representations via modules like DeWavLM-Omni, Adapter, Vocoder, and PostNet for multi-rate processing.
• Experimental evaluations show robust performance in packet loss concealment and non-intrusive perceptual quality, advancing the state of universal speech enhancement.

to=arxiv_search.search 《凤凰大参考json {"query":"(Rong et al., 16 Apr 2026) UniPASE A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations", "max_results": 5} to=arxiv_search qq上json {"query":"(Rong et al., 16 Apr 2026) UniPASE A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations","max_results":5} to=search_arxiv գործումայժմ ՞ւjson {"query":"UniPASE (Rong et al., 16 Apr 2026)","max_results":10} UniPASE is a purely generative universal speech enhancement (USE) system designed to restore speech corrupted by many different distortion types while also handling multiple input and output sampling rates. It extends the earlier PASE framework into the USE setting defined by the URGENT challenge series by combining phonologically anchored representation enhancement, explicit acoustic refinement, waveform synthesis at 16 kHz, and bandwidth extension to 48 kHz before resampling to the original rate. Its central design objective is to preserve the perceptual advantages of generative restoration while reducing hallucination, defined as either incorrect spoken content or inconsistent speaker characteristics and acoustics (Rong et al., 16 Apr 2026).

1. Problem setting and conceptual position

UniPASE addresses a USE formulation aligned with the URGENT 2025 Challenge. In that setup, a single model must process seven distortion types—additive noise, reverberation, clipping, bandwidth limitation, codec artifacts, packet loss, and wind noise—and support input/output sampling rates of 8, 16, 22.05, 24, 32, 44.1, and 48 kHz (Rong et al., 16 Apr 2026). The task is therefore simultaneously multi-distortion, multi-rate, and expected to preserve authenticity of content and speaker identity.

The model is explicitly framed around three goals: high-fidelity enhancement, low linguistic hallucination, and universal handling of different input/output sampling rates. The paper distinguishes two forms of hallucination. Linguistic hallucination denotes output with incorrect spoken content, whereas acoustic hallucination denotes inconsistent speaker traits or acoustics. This distinction is central because the target is not only perceptual plausibility, but faithfulness to the source utterance (Rong et al., 16 Apr 2026).

UniPASE inherits the core logic of PASE, the Phonologically Anchored Speech Enhancer. PASE performs enhancement in a representation space derived from WavLM and uses dual-stream representations: a high-level phonetic representation $R_P$ and a low-level acoustic representation $R_A$. In UniPASE, that philosophy is retained and extended in three stated ways: DeWavLM becomes DeWavLM-Omni for broader distortion coverage, an explicit Adapter is inserted to enhance acoustic representations before waveform synthesis, and a PostNet is added for sampling-rate-flexible output. The resulting system is therefore not presented as a separate paradigm, but as a full-stack USE version of PASE (Rong et al., 16 Apr 2026).

A common misconception is that a purely generative USE model must generate waveform content without strong intermediate constraints. UniPASE is instead organized around intermediate representations anchored by a pretrained WavLM prior. Another common misconception is that the reported URGENT 2026 result refers to standalone UniPASE; the paper states more narrowly that UniPASE served as the backbone of a hybrid extension integrating predictive TF-GridNet, and that extended system achieved 1st place in the objective evaluation (Rong et al., 16 Apr 2026).

2. End-to-end architecture and signal flow

The complete UniPASE pipeline begins with a degraded waveform at arbitrary sampling rate, resamples it to 16 kHz, performs packet loss detection to obtain a binary packet-loss mask $M_T$, and feeds the degraded 16-kHz waveform together with $M_T$ into DeWavLM-Omni. DeWavLM-Omni outputs an enhanced phonetic representation $R_P$ and a degraded acoustic representation $R_A$. The degraded $R_A$, conditioned on the enhanced $R_P$, is then passed through the Adapter to produce an enhanced acoustic representation $\hat{R}_A$. The Vocoder synthesizes an enhanced 16-kHz waveform from $\hat{R}_A$. If the original desired output rate exceeds 16 kHz, a PostNet converts the waveform to 48 kHz, after which the system downsamples to the original target rate (Rong et al., 16 Apr 2026).

The paper identifies four key components and assigns each a specific role:

Component	Input/Output	Role
DeWavLM-Omni	degraded waveform $R_A$0	core enhancement in the phonetic representation domain
Adapter	degraded $R_A$1 conditioned on enhanced $R_A$2 $R_A$3	explicit acoustic enhancement
Vocoder	$R_A$4 16-kHz waveform	waveform synthesis
PostNet	16-kHz waveform $R_A$5 48-kHz waveform	bandwidth extension / super-resolution

The notation is fixed throughout the model. The degraded waveform is $R_A$6, the clean waveform is $R_A$7, the packet-loss mask is $R_A$8, $R_A$9 denotes the number of time frames, and $M_T$0 denotes feature dimensionality. Both streams satisfy

$M_T$1

The phonetic representation is taken from the final Transformer layer of WavLM-like processing, while the acoustic representation is taken from the first Transformer layer (Rong et al., 16 Apr 2026).

The multi-rate logic is deliberately asymmetric. Internal enhancement and synthesis are centered on 16 kHz, while higher-rate restoration is deferred to the PostNet. This means that support for multiple I/O rates is achieved by front-end resampling to 16 kHz and back-end conversion to 48 kHz followed by resampling to the original rate when needed (Rong et al., 16 Apr 2026).

3. DeWavLM-Omni, WavLM priors, and hallucination suppression

DeWavLM-Omni is the core enhancement model in UniPASE. It extends the earlier DeWavLM idea from PASE to universal speech enhancement by mapping a degraded waveform to a clean, linguistically faithful phonetic representation under a broader multi-distortion training setup. The backbone is not changed into a new architecture; rather, the denoising representation distillation (DRD) strategy is retrained with diverse distortions so that the model produces degradation-invariant phonetic representations corresponding to clean speech (Rong et al., 16 Apr 2026).

The system uses two copies of WavLM: a frozen teacher and a trainable student, both initialized from pretrained WavLM weights. The student is trained to map degraded input to the teacher’s clean representation target using the DRD objective

$M_T$2

The paper states that this loss is computed over all frames rather than only masked ones. DeWavLM-Omni adopts the WavLM-Large configuration, and during distillation training all model parameters are updated (Rong et al., 16 Apr 2026).

WavLM is used because it provides a strong phonological prior learned through self-supervised pretraining. The final-layer features are treated as phonetic representations because they contain abstract, context-dependent phonetic information, while the first-layer features are treated as acoustic representations because they preserve finer low-level detail useful for speaker identity and prosody. The paper explicitly notes that “phonetic” and “acoustic” are not strictly disentangled factors; they are names based on dominant information observed in prior analysis (Rong et al., 16 Apr 2026).

Packet loss concealment is integrated through an explicit packet loss detection (PLD) algorithm. For audio $M_T$3, packet length and packet count are defined as

$M_T$4

For packet $M_T$5,

$M_T$6

and packet $M_T$7 is flagged as lost if $M_T$8. The returned binary mask is $M_T$9. The paper emphasizes that packets per second match WavLM embedding frames per second, so the mask can be applied directly without interpolation. For detected missing packets, the corresponding CNN output frames inside WavLM are replaced with a shared learnable mask embedding (Rong et al., 16 Apr 2026).

The anti-hallucination function of the WavLM prior is supported by ablations. On the URGENT 2025 non-blind set, removing the prior yields CER $M_T$0, LPS $M_T$1, and SBS $M_T$2, compared with CER $M_T$3, LPS $M_T$4, and SBS $M_T$5 with the prior. A separate prior ablation reports CER $M_T$6, LPS $M_T$7, and SpkSim $M_T$8 without prior, versus CER $M_T$9, LPS $R_P$0, and SpkSim $R_P$1 with prior. The paper interprets this as evidence that the phonological prior is the main anti-linguistic-hallucination mechanism. PLD also contributes: removing it changes UTMOS from 3.30 to 3.19, SBS from 0.88 to 0.86, LPS from 0.83 to 0.79, and CER from $R_P$2 to $R_P$3 (Rong et al., 16 Apr 2026).

4. Adapter, MSRD, and waveform reconstruction

The Adapter is introduced because direct dual-stream waveform reconstruction, as used in PASE, can permit noise and reverberation leakage, especially at low SNR. In UniPASE, the Adapter takes degraded acoustic representation $R_P$4, conditions on enhanced phonetic representation $R_P$5, and predicts an enhanced acoustic representation whose training target is the clean $R_P$6 obtained from DeWavLM-Omni when fed clean speech. Conditioning is implemented by element-wise summation, a choice described as “simple yet effective” and consistent with ablation findings in PASE (Rong et al., 16 Apr 2026).

The Adapter predicts acoustic representations rather than mel-spectrograms, codec tokens, or a separate SSL-token vocabulary. Its backbone is the improved Vocos architecture proposed in WavTokenizer, with hidden dimension 1024, 4 ResNet blocks, 1 attention module, 12 ConvNeXt blocks, and shared intermediate dimension 3072. To avoid over-smoothing under a pure regression objective, the model adds a representation-domain discriminator, MSRD, the Multi-Scale Representation Discriminator. MSRD uses six sub-discriminators with hidden channels

$R_P$7

each implemented as a stack of 1D convolutions with Leaky ReLU, with the first layer projecting the input to a hidden size corresponding to the scale (Rong et al., 16 Apr 2026).

Adapter training follows LS-GAN. The stated losses are

$R_P$8

$R_P$9

$R_A$0

$R_A$1

with final objectives

$R_A$2

$R_A$3

and

$R_A$4

Here $R_A$5 and $R_A$6 are ground-truth and generated acoustic representations in $R_A$7 (Rong et al., 16 Apr 2026).

The Vocoder reconstructs the 16-kHz waveform from the enhanced acoustic representation. It uses the same improved Vocos / WavTokenizer-style backbone as the Adapter, plus an iSTFT head with FFT size 1280 and hop size 320. The Vocoder is trained independently on clean speech, then frozen and integrated into UniPASE; there is no joint fine-tuning with the other modules. Its objective includes multi-scale Mel-spectrogram reconstruction loss, adversarial loss, and feature matching loss, with multi-period discriminator (MPD) and multi-band multi-scale STFT discriminator (MBMSD) following DAC. The multi-scale Mel loss uses window lengths $R_A$8, Mel bins $R_A$9, and hop lengths equal to one quarter of each window length. The loss weights are reconstruction 30, adversarial 1, and feature matching 1 (Rong et al., 16 Apr 2026).

A key design result is that $R_A$0 is substantially better than $R_A$1 for synthesis. On clean-speech probing, a vocoder trained on $R_A$2 obtains PESQ 1.29 and SpkSim 0.62, while a vocoder trained on $R_A$3 obtains PESQ 3.47 and SpkSim 0.94. In ablations on the URGENT 2025 non-blind set, direct dual-stream reconstruction gives UTMOS 2.64, PESQ 2.00, and SpkSim 0.77; adding the Adapter without MSRD gives UTMOS 2.98, PESQ 2.15, and SpkSim 0.76; adding MSRD yields NISQA 4.26 and SpkSim 0.80. A separate subjective result reports CMOS $R_A$4 for MSRD (Rong et al., 16 Apr 2026).

5. PostNet, training data, and optimization

Because DeWavLM-Omni and the Vocoder are intrinsically 16-kHz / 8-kHz-bandwidth modules, UniPASE adds a PostNet to support higher output rates such as 22.05, 24, 32, 44.1, and 48 kHz. PostNet takes the 16-kHz vocoder output, performs bandwidth extension to 48 kHz, and the system then resamples to the original target rate. It is used only when the original desired rate exceeds 16 kHz. The architecture is the STFT-domain CWS-TF-GridNet adopted from TS-URGENet, with FFT size 1536, hop size 768, embedding dimension 48, LSTM hidden dimension 100, 4 attention heads, and 5 blocks (Rong et al., 16 Apr 2026).

The paper describes an inference-time low-frequency copying mechanism intended to prevent bandwidth extension from altering the already reliable low-frequency band. Let $R_A$5 be the spectrogram derived from the 16-kHz vocoder output and resampled to 48 kHz, and let $R_A$6 be the network-predicted full-band spectrogram. The final full-band spectrogram is

$R_A$7

where

$R_A$8

with $R_A$9 kHz and $R_P$0 Hz. Below $R_P$1, the original low band is kept entirely; in the transition band, the system blends gradually; above $R_P$2, it relies on the generated high band (Rong et al., 16 Apr 2026).

Training is fully supervised and staged. The clean speech pool is built from the LibriVox subset of DNS5, LibriTTS, VCTK, EARS, MLS, and Common Voice 19.0. All corpora except EARS are filtered using DNSMOS scores (OVRL, SIG, BAK, and P.808) with threshold 3.0; EARS is exempt because DNSMOS is unreliable for atypical speech such as whisper or extreme pitch, and manual inspection confirmed quality. The final clean set size is approximately 2,360 hours. Noise sources are DNS5, WHAM!, FSD50K, FMA, and a simulated wind noise database, and RIRs are openSLR26 and openSLR28 (Rong et al., 16 Apr 2026).

Training mixtures are generated on the fly. For each sample, the pipeline starts from the original clean utterance as target, convolves with a random RIR with probability 0.5, mixes with a random noise clip at SNR uniformly sampled from $R_P$3 dB, uses simulated wind noise with probability 0.05 and ordinary noise addition otherwise, and then applies additional distortions. The additional distortions are sampled as zero augmentations 0.25, one augmentation 0.40, two augmentations 0.20, and three augmentations 0.15. The four extra distortion types—clipping, bandwidth limitation, codec artifact, and packet loss—are selected with equal probability. Detailed settings include bandwidth limitation at 4 kHz, codec format in $R_P$4 with qscale in $R_P$5, and packet loss duration 20 ms with rate in $R_P$6 and max continuous loss 10 (Rong et al., 16 Apr 2026).

The modules are trained in four stages: Vocoder on clean speech; DeWavLM-Omni on degraded speech with PLD mask; Adapter on top of frozen DeWavLM-Omni; and PostNet on top of frozen DeWavLM-Omni, Adapter, and Vocoder. All models use AdamW, linear warmup over the first 10% of steps from zero to peak learning rate, and cosine decay to $R_P$7. Training uses four NVIDIA 4090 GPUs. The paper states a general weighting philosophy in which the reconstruction term dominates and adversarial and feature-matching terms contribute roughly $R_P$8 of the total loss (Rong et al., 16 Apr 2026).

6. Evaluation, empirical behavior, and limitations

Evaluation covers DNS 2020 for denoising and dereverberation, the PLC 2024 validation set for packet loss concealment, the VoiceFixer GSR test set for denoising, dereverberation, declipping, and bandwidth extension, and the URGENT 2025 non-blind test set as the main USE benchmark. Baselines include TF-GridNet, StoRM, LLaSE-G1, AnyEnhance, PASE, UNIVERSE++, VoiceFixer, BSRNN-FAN, TS-URGENet, FUSE, USEMamba, the TF-GridNet official baseline, and the wataru9871 pure generative system. Metrics span non-intrusive scores such as DNSMOS, UTMOS, NISQA, and PLCMOS; intrusive scores such as PESQ and ESTOI; representation-similarity-based scores such as SpkSim, LPS, and SBS; and ASR-based scores such as WER, CER, dWER, and dCER (Rong et al., 16 Apr 2026).

On DNS 2020 no-reverb, UniPASE reports DNSMOS 3.40, UTMOS 4.06, PESQ 3.05, ESTOI 0.93, SBS 0.94, LPS 0.97, SpkSim 0.96, and dWER 2.17\%. On DNS 2020 with-reverb, it reports DNSMOS 3.33, UTMOS 3.62, PESQ 1.74, ESTOI 0.76, SBS 0.87, LPS 0.93, SpkSim 0.79, and dWER 8.16\%. On PLC 2024 validation, it reports DNSMOS 3.39, NISQA 4.34, UTMOS 3.65, PLCMOS 4.30, PESQ 2.53, ESTOI 0.85, SBS 0.93, LPS 0.92, SpkSim 0.94, and WER 13.55\%. On the VoiceFixer GSR test set, it reports DNSMOS 3.09, NISQA 4.37, UTMOS 3.89, PESQ 2.47 (tied with TF-GridNet), ESTOI 0.69, SBS 0.89, LPS 0.91, SpkSim 0.81, and dWER 8.21\% (Rong et al., 16 Apr 2026).

The URGENT 2025 non-blind set is the clearest USE comparison. UniPASE reports DNSMOS 3.26, NISQA 4.18, UTMOS 2.97, PESQ 2.12, ESTOI 0.70, SBS 0.89, LPS 0.84, SpkSim 0.81, and CER 12.90\%. Relative to the listed purely generative competitor wataru9871, it improves CER from $R_P$9 to $\hat{R}_A$0 and SpkSim from 0.51 to 0.81. The paper therefore positions UniPASE not as a system that dominates every predictive or hybrid model on every intrusive metric, but as evidence that a purely generative USE model can achieve strong perceptual quality without severe hallucination (Rong et al., 16 Apr 2026).

Packet loss concealment is a particularly strong case. On PLC 2024, the paper reports lossy WER $\hat{R}_A$1, TF-GridNet $\hat{R}_A$2, UNIVERSE++ $\hat{R}_A$3, LLaSE-G1 $\hat{R}_A$4, and UniPASE $\hat{R}_A$5. The robustness analysis further states that the model remains strong across packet-loss fractions and generalizes beyond its training maximum continuous loss of 10 packets to 25 packets, while degrading more substantially beyond 50 packets. The paper concludes that it is robust in realistic scenarios with bursts $\hat{R}_A$6 packets and loss fractions $\hat{R}_A$7 (Rong et al., 16 Apr 2026).

The study also reports cross-language behavior on multilingual URGENT 2025 data. Acoustic representations generalize well across languages, with probing results of PESQ approximately 3.37–3.60 and SpkSim approximately 0.93–0.95. Phonetic representations are strongest for English but still useful in other languages, and the phonological prior improves LPS and CER consistently across tested languages. This suggests reasonable cross-lingual transfer from an English-pretrained WavLM prior, although the paper describes that interpretation as likely rather than formally proved (Rong et al., 16 Apr 2026).

The system is large and computationally heavy. DeWavLM-Omni has 315.44M parameters and 18.08 G/s; the Adapter has 113.73M parameters and 5.69 G/s; the Vocoder has 113.73M parameters and 5.69 G/s; and the PostNet has 2.77M parameters and 49.73 G/s. The total is 545.7M parameters and 79.2 GMACs/s. The PostNet is therefore unusual in having comparatively few parameters but high compute. The paper does not claim real-time processing or report latency. It also notes that very long packet-loss bursts can make outputs effectively unusable, and on URGENT 2025 the model remains weaker than leading predictive or hybrid systems on PESQ and ESTOI. Within those limits, UniPASE is presented as a unified framework for many distortions and sample rates, with particularly strong packet loss concealment, strong non-intrusive perceptual quality, and comparatively low hallucination for a purely generative USE model (Rong et al., 16 Apr 2026).