WhisperVC: Whisper-to-Normal Speech Conversion

• WhisperVC is a family of whispered-speech conversion approaches that restore intelligibility by recovering pitch and vocal characteristics from muted input.
• It encompasses diverse methodologies, including vocoder-free waveform generation, spectrogram translation with MaskCycleGAN, and a three-stage modular Mandarin pipeline.
• These systems leverage non-parallel training, advanced masking strategies, and alignment techniques to achieve near ground-truth naturalness and speaker similarity.

Searching arXiv for relevant WhisperVC papers and closely related work. WhisperVC is a name used in recent arXiv literature for several distinct systems centered on whispered-speech conversion, especially whispered-to-normal or whispered-to-speech reconstruction. Across these uses, the underlying problem is the recovery of intelligible, natural-sounding voiced speech from whispered input, which lacks vocal-fold excitation and exhibits reduced energy and shifted formant frequencies (Liu et al., 2 Nov 2025). The term has been attached to at least three technically different paradigms: a vocoder-free non-parallel masked cycle-consistent GAN that operates directly on waveforms (Wagner et al., 2023), a MaskCycleGAN-based spectrogram-to-spectrogram system for whisper-to-normal conversion (Gupta et al., 2024), and a three-stage target speaker-controllable Mandarin framework combining Whisper-large V3, a Conformer VAE, FastSpeech 2, and HiFi-GAN (Liu et al., 2 Nov 2025). Related work such as WESPER addresses the same conversion problem with zero-shot, real-time self-supervised learning (Rekimoto, 2023).

1. Terminological scope and research lineage

A common misconception is that WhisperVC denotes a single canonical architecture. The literature instead uses the label for multiple systems with different assumptions about supervision, synthesis, alignment, and speaker control.

Paper	Primary task	Core formulation
(Wagner et al., 2023)	Whispered-to-voiced conversion	Vocoder-free non-parallel masked cycle-consistent GAN with direct waveform generation
(Gupta et al., 2024)	Whisper-to-normal speech conversion	Spectrogram translation as an image-to-image task with MaskCycleGAN
(Liu et al., 2 Nov 2025)	Target speaker-controllable Mandarin whisper-to-speech conversion	Three-stage Whisper-large V3 encoder, Conformer VAE, Length–Channel Aligner, FastSpeech 2, HiFi-GAN

The 2023 vocoder-free system emphasizes non-parallel training and elimination of a separate vocoder by generating raw waveforms directly from masked mel-spectrogram inputs (Wagner et al., 2023). The 2024 MaskCycleGAN-based system keeps the non-parallel cycle-consistent adversarial formulation but reverts to spectrogram translation, focusing on mask tuning and a voice activity detector as the key sources of improvement (Gupta et al., 2024). The 2025 Mandarin system departs more substantially from the GAN-only lineage by decomposing the problem into content extraction, latent alignment, target-speaker-conditioned acoustic prediction, and waveform synthesis (Liu et al., 2 Nov 2025).

This progression suggests a shift from purely adversarial domain translation toward modular architectures that explicitly separate content, speaker identity, timing, and vocoding.

2. Task definition and signal characteristics

Whispered-speech conversion seeks to reconstruct normally phonated speech from a source signal in which periodic vocal-fold excitation is absent. In the Mandarin WhisperVC formulation, whispered speech is described as having reduced energy and shifted formant frequencies, making natural and intelligible voice reconstruction highly challenging (Liu et al., 2 Nov 2025). In the MaskCycleGAN-based system, the task is framed as spectrogram translation, with the spectrogram treated as an image and the conversion learned as an image-to-image mapping (Gupta et al., 2024).

Two recurrent technical consequences follow from this problem definition. First, the recovery of pitch-related structure is central. The 2024 MaskCycleGAN system states that the generator learns to “fill-in” missing spectral regions, encouraging it to recover pitch harmonics in voiced parts while preserving unvoiced regions (Gupta et al., 2024). Second, temporal alignment is a major design choice. The vocoder-free 2023 system explicitly targets non-parallel voice conversion and avoids temporal alignments by learning source-target mappings through cycle consistency (Wagner et al., 2023), whereas the 2025 Mandarin system introduces soft-DTW alignment inside a Conformer-VAE and then uses a deterministic Length–Channel Aligner to bridge 16 kHz latent representations and 22.05 kHz mel frames (Liu et al., 2 Nov 2025).

Within the masked spectrogram formulation, the basic corruption mechanism is

$X_{\text{masked}}(t,f) = M(t,f)\cdot X(t,f),$

where $M\in\{0,1\}^{T\times F}$ hides a random subset of frames (Gupta et al., 2024). In the 2024 system, the mask is non-consecutive-frame filling; $K$ non-adjacent frames are selected uniformly, with mask ratio $\rho$ controlling how much of the spectrogram is hidden (Gupta et al., 2024). In the 2023 vocoder-free system, a contiguous block of at most 25 frames is randomly zeroed in mel-spectrogram space during each forward pass (Wagner et al., 2023).

3. Masked cycle-consistent adversarial formulations

The two masked cycle-consistent GAN variants share the same high-level premise: learn bidirectional mappings between whispered and voiced domains without explicit parallel alignment, while using masking as a self-supervised or auxiliary reconstruction pressure.

In the 2024 MaskCycleGAN-based whisper-to-normal system, the generators are the “2-1-2D CNN” from CycleGAN-VC2: a 2D convolutional encoder downsamples the spectrogram, 1D residual blocks operate along the time axis, and a symmetric 2D deconvolutional decoder reconstructs the spectrogram (Gupta et al., 2024). The discriminators are PatchGAN classifiers that operate on local patches such as $70\times70$ (Gupta et al., 2024). The full objective combines adversarial, cycle-consistency, identity, and mask-regularization terms:

$$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$

with $\lambda_{cyc}=10$, $\lambda_{id}=5$, and $\lambda_{mask}=2$ (Gupta et al., 2024). The mask loss is defined as

$$\mathcal{L}_{mask}(G) = \mathbb{E}_{x,M}\bigl[\| (1-M)\odot\bigl(G(M\odot x) - x\bigr)\|_1 \bigr].$$

The 2023 vocoder-free WhisperVC changes the synthesis target and the generator family. Its generator is based on the HiFi-GAN vocoder generator and takes as input a masked mel-spectrogram $M\in\{0,1\}^{T\times F}$0 together with its binary mask $M\in\{0,1\}^{T\times F}$1, concatenated along the channel dimension (Wagner et al., 2023). A small 2D GLU-based feature encoder processes $M\in\{0,1\}^{T\times F}$2, the representation is reshaped to a 1-D sequence, and $M\in\{0,1\}^{T\times F}$3 transposed-convolution upsampling blocks with strides $M\in\{0,1\}^{T\times F}$4 and kernels $M\in\{0,1\}^{T\times F}$5 produce the waveform, each followed by a Multi-Receptive-Field Fusion module (Wagner et al., 2023). The discriminator stack comprises two Multi-Scale Discriminators and two Multi-Period Discriminators for each forward and backward mapping, following HiFi-GAN-style 1-D convolutional blocks with Leaky-ReLU (Wagner et al., 2023).

Its training objective follows least-squares GAN and HiFi-GAN losses, adds cycle-consistency in mel-spectrogram space, identity losses, and a second cycle adversarial loss to counter over-smoothing (Wagner et al., 2023). Because the forward generator upsamples masked mel-spectrograms directly to raw audio, no separate neural vocoder is needed (Wagner et al., 2023).

The principal architectural divergence between the two systems is therefore not the use of masking or cycle consistency, which both share, but the representation at which conversion and synthesis are coupled: spectrogram translation with downstream reconstruction in the 2024 work, versus end-to-end waveform generation in the 2023 work.

4. Three-stage controllable Mandarin WhisperVC

The 2025 Mandarin WhisperVC introduces a substantially more modular architecture organized into three stages (Liu et al., 2 Nov 2025).

Stage 1 uses a fine-tuned OpenAI Whisper-large V3 encoder as a content encoder. The input is a 16 kHz waveform converted to 80-dimensional mel-filterbank features and processed by Conv/Subsampling and Transformer blocks. The resulting content features are

$M\in\{0,1\}^{T\times F}$6

for the whisper and normal branches respectively (Liu et al., 2 Nov 2025). A Conformer-based VAE then employs two parallel Conformer encoders $M\in\{0,1\}^{T\times F}$7 and $M\in\{0,1\}^{T\times F}$8 and a shared Conformer decoder $M\in\{0,1\}^{T\times F}$9. The latent variables satisfy

$K$0

with prior $K$1 (Liu et al., 2 Nov 2025). The per-batch loss contains KL terms, an $K$2 reconstruction term on normal speech, and a soft-DTW term aligning reconstructed whisper to normal content (Liu et al., 2 Nov 2025).

Stage 2 converts the Stage 1 outputs into target mel-spectrograms. The Length–Channel Aligner first temporally upsamples the sequence using

$K$3

then applies linear interpolation and channel projection using Conv1d$K$4 with ReLU and Conv1d$K$5 to produce $K$6 (Liu et al., 2 Nov 2025). A modified FastSpeech 2 removes duration modeling, predicts pitch $K$7 and energy $K$8, and conditions on a 256-dimensional speaker vector $K$9 from a SimAM–ResNet34 encoder; the decoder outputs $\rho$0 (Liu et al., 2 Nov 2025).

Stage 3 fine-tunes a HiFi-GAN vocoder. The generator $\rho$1 synthesizes waveform $\rho$2 from predicted mel $\rho$3, while standard multi-period and multi-scale discriminators $\rho$4 provide adversarial supervision (Liu et al., 2 Nov 2025). The generator loss combines adversarial, feature-matching, and STFT-magnitude terms, and the discriminator loss sums real and fake least-squares terms across discriminators (Liu et al., 2 Nov 2025).

Quantitatively, on AISHELL6-Whisper the reported results are as follows (Liu et al., 2 Nov 2025):

Method	DNSMOS	UTMOS
Whispered input	1.10	1.30
Proposed (WhisperVC)	3.11	2.52
Ground truth (GT)	3.14	2.87

The same evaluation reports CER of 25.78% for whispered input and 18.67% for WhisperVC, together with timbre cosine similarity of 0.58 for whispered input and 0.76 for WhisperVC (Liu et al., 2 Nov 2025). The paper characterizes these results as near ground-truth quality while maintaining speaker similarity under whisper-only inference (Liu et al., 2 Nov 2025).

5. Data regimes, training procedures, and empirical results

The 2024 MaskCycleGAN-based system uses the wTIMIT US subset with speaker-dependent conversion, training and testing one speaker at a time (Gupta et al., 2024). Its reported split is 10,934 normal and 10,245 whispered utterances for training, and 725 normal and 723 whispered utterances for testing (Gupta et al., 2024). Preprocessing windows audio into 20 ms frames with 5 ms shifts and converts it to magnitude spectrograms resized to $\rho$5 images (Gupta et al., 2024). A simple frame-level VAD based on zero-crossing rate and band-energy ratio removes leading and trailing silence, with zero-crossing thresholds of approximately 12/sec for voiced and approximately 50/sec for unvoiced, energy-ratio thresholds of 0.6 for normal and 0.2 for whispered speech, and a final 0.5 s median filter (Gupta et al., 2024). Training uses Adam with $\rho$6, $\rho$7, initial learning rate $\rho$8, batch size 1, and 400 epochs, with linear decay after 200 epochs (Gupta et al., 2024).

Its objective evaluation uses PESQ and Generator Loss, where

$\rho$9

and lower is better (Gupta et al., 2024). Tuning $70\times70$0 from 25% to 50% and $70\times70$1 from 64 to 128 improves PESQ from 1.786 to 2.328 and reduces G-Loss from 10.656 to 6.432 without VAD; adding VAD yields the best reported setting, with PESQ 3.159 and G-Loss 5.542 (Gupta et al., 2024). A subjective evaluation with 43 listeners rating 6 random test utterances on a 1–5 scale gives a final MOS of 4.1 for the setting $70\times70$2 (Gupta et al., 2024).

The 2023 vocoder-free system also evaluates on wTIMIT, training 10 speaker-dependent models over 5 female and 5 male speakers with two accents, using approximately 430 utterances per speaker for training and 20 held-out utterances per speaker for testing (Wagner et al., 2023). Audio is represented at 22.05 kHz with 80-dimensional mel features using window size 1024 and hop 256 (Wagner et al., 2023). Training uses batch size 8 for 50k iterations with Adam, $70\times70$3, $70\times70$4, initial learning rate $70\times70$5, decay 0.999/epoch, $70\times70$6, $70\times70$7, max mask length 25 frames, 3 MSD and 5 MPD blocks, and $70\times70$8 upsampling stages (Wagner et al., 2023). Against MaskCycleGAN-VC, its full model reports MCD $70\times70$9 dB versus $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$0 dB, fwSNRseg $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$1 dB versus $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$2, and RMSE(log F0) $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$3 versus $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$4 (Wagner et al., 2023). MOS prediction gives $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$5 versus $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$6, while subjective ACR listening tests report naturalness 3.54 versus 3.32 and intelligibility 3.66 versus 3.43 (Wagner et al., 2023).

The 2025 Mandarin WhisperVC uses AISHELL6-Whisper, a 30 h paired whisper/normal Mandarin corpus with 167 speakers recorded at 48 kHz (Liu et al., 2 Nov 2025). Training uses 110 speakers and approximately 20 h for all three stages, while evaluation uses 28 unseen speakers and approximately 5 h (Liu et al., 2 Nov 2025). The reported metric suite differs from the wTIMIT-based papers: DNSMOS and UTMOS for naturalness, Character Error Rate from Whisper-largeV3-turbo ASR for intelligibility, and cosine similarity between generated and reference speaker embeddings from Resemblyzer for timbre similarity (Liu et al., 2 Nov 2025).

Taken together, these results show that empirical comparison across WhisperVC systems is constrained by differing datasets, sampling rates, output representations, and evaluation protocols. Direct numerical ranking across papers would therefore be unwarranted.

6. Related systems, limitations, and open questions

WESPER is a closely related, but separately named, whisper-to-normal voice conversion system that emphasizes zero-shot, speaker-independent, real-time operation (Rekimoto, 2023). Its two-stage pipeline consists of a Speech-to-Unit encoder producing one 256-dimensional vector per 20 ms frame and a Unit-to-Speech decoder reconstructing a target mel-spectrogram, followed by a HiFi-GAN vocoder (Rekimoto, 2023). The STU encoder is pretrained on unpaired normal and whispered data, including LibriSpeech 960 h, mechanically converted whispered LibriSpeech, and the wTIMIT whispered/normal data, while the UTS decoder is trained per target speaker using only unlabeled normal speech (Rekimoto, 2023). The system reports approximately $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$7 real-time on GPU and approximately $$\mathcal{L} = \mathcal{L}_{GAN}(G,D_Y) + \mathcal{L}_{GAN}(F,D_X) + \lambda_{cyc}\mathcal{L}_{cyc} + \lambda_{id}\mathcal{L}_{id} + \lambda_{mask}\mathcal{L}_{mask},$$8 on Apple M1 Max CPU, with total latency of approximately 70 ms (Rekimoto, 2023). In listening evaluations, 50 English listeners rated normal speech at approximately 4.15 MOS, raw whispered speech at approximately 3.27, and WESPER-converted speech at approximately 3.79; MUSHRA scores were approximately 51.3 for whisper and approximately 63.6 for WESPER, and Google Cloud STT WER on wTIMIT improved from 44.7% for whispered speech to 26.7% for WESPER-converted speech (Rekimoto, 2023). The same work reports that prosodic naturalness was not significantly different from the raw whispered source (Rekimoto, 2023).

An adjacent line of work addresses the reverse direction: conversion of normally phonated speech into whispered speech for whispered speech synthesis (Cotescu et al., 2019). That study investigates a hand-crafted DSP recipe and statistical voice-conversion models based on GMMs and DNNs trained on parallel normal/whispered corpora, and reports that VC techniques are significantly better than rule-based signal processing methods and can be indistinguishable from copy-synthesis of natural whisper recordings (Cotescu et al., 2019). It also states that excluding the target speaker from the training set has little or no impact on perceived naturalness and speaker similarity, and that the DNN method is used in Amazon Alexa’s Whisper Mode (Cotescu et al., 2019). Although this is not whisper-to-normal reconstruction, it remains relevant as evidence that whispered-speech conversion problems can support both speaker-dependent and generalized multi-speaker regimes.

Several limitations recur across the WhisperVC literature. The 2024 MaskCycleGAN paper states that speaker-independent modeling remains to be studied (Gupta et al., 2024). The 2023 vocoder-free model similarly notes that extension to multi-speaker or zero-shot conversion remains open (Wagner et al., 2023). The 2025 Mandarin system demonstrates robust whisper-only inference and target-speaker control on unseen speakers (Liu et al., 2 Nov 2025), but its architecture is tied to a staged 16 kHz-to-22.05 kHz pipeline and Mandarin corpus setting (Liu et al., 2 Nov 2025). Proposed future directions in the GAN-based literature include phase reconstruction or end-to-end waveform GANs such as MelGAN, adaptive mask schedules or learnable masks, diffusion or flow-based architectures, and perceptual losses based on learned speech representations (Gupta et al., 2024, Wagner et al., 2023).

The most defensible synthesis is therefore not that WhisperVC denotes one settled method, but that it names a family of whispered-speech conversion approaches spanning masked non-parallel GANs, spectrogram translation systems, and modular controllable pipelines. Their shared objective is the restoration of voicing, intelligibility, and speaker characteristics from whispered input; their main differences concern representation level, supervision regime, speaker generalization, and whether waveform synthesis is integrated or delegated to a separate vocoder.