VoiceGrad: Score-Based Voice Conversion

• VoiceGrad is a non-parallel voice conversion framework that transforms a source mel-spectrogram to match a target speaker while preserving linguistic content.
• It employs conditioned score-based generation with techniques like annealed Langevin dynamics, reverse diffusion, and L1 objective training to guide conversion.
• Recent iterations—FastVoiceGrad, FasterVoiceGrad, and LatentVoiceGrad—optimize efficiency and audio quality through one-step distillation and latent diffusion strategies.

VoiceGrad is a non-parallel voice conversion framework that treats conversion as conditioned score-based generation in mel-spectrogram space: a source utterance is transformed so as to preserve linguistic content while approaching the target speaker distribution. The original formulation introduced non-parallel any-to-many voice conversion with weighted denoising score matching, annealed Langevin dynamics, and a reverse-diffusion variant (Kameoka et al., 2020). Later work recast the conditioning for zero-shot any-to-any conversion, distilled the iterative model into one-step students, and moved the reverse process into latent diffusion and flow-matching formulations to improve latency and audio quality (Kaneko et al., 2024, Kaneko et al., 25 Aug 2025, Kameoka et al., 10 Sep 2025).

1. Problem setting and representational scope

Voice conversion (VC) is the task of transforming a source speaker’s voice to sound like a specified target speaker while preserving the linguistic content. In the VoiceGrad line, the task is explicitly non-parallel: training does not require parallel utterances or transcriptions. The original system is any-to-many, in which the source speaker at test time can be arbitrary and conversion is performed to one of multiple target speakers represented during training (Kameoka et al., 2020). Later systems adopt speaker embeddings rather than speaker IDs, enabling one-shot any-to-any VC and zero-shot unseen-to-unseen VC (Kaneko et al., 2024, Kameoka et al., 10 Sep 2025).

A persistent misconception is that VoiceGrad is a waveform-generation model. It is not. Across the reported variants, the conversion target is the mel-spectrogram, typically with 80 channels, and waveform synthesis is delegated to a neural vocoder such as HiFi-GAN (Kameoka et al., 2020, Kaneko et al., 2024). This design separates speaker/content conversion from waveform rendering.

Conditioning evolves across the literature. The original VoiceGrad conditions on a target speaker identity $k$, a noise level $l$, and optionally a 144-dimensional bottleneck feature (BNF) sequence extracted by a pretrained ASR bottleneck model (Kameoka et al., 2020). Later DDPM-style formulations use a target speaker embedding $s$ and a content or phoneme embedding $p$, where $s$ is obtained either from a pretrained speaker encoder or a d-vector extractor, and $p$ is obtained from the source utterance through a BNF, phoneme feature extractor, or content encoder (Kaneko et al., 2024, Kameoka et al., 10 Sep 2025). In the FastVoiceGrad line, pitch conditioning is not used, although VoiceGrad/FastVoiceGrad can, in principle, incorporate $F0$ as an additional conditioning channel (Kaneko et al., 2024).

2. Original VoiceGrad: score matching, annealed Langevin dynamics, and reverse diffusion

The original VoiceGrad is a score-based, diffusion-inspired approach that trains a score approximator to predict the gradient of the log density of speech feature sequences from multiple speakers, then performs conversion by iteratively updating the source feature sequence toward a nearest stationary point of the target distribution (Kameoka et al., 2020). Two formulations are reported: a DSM variant using annealed Langevin dynamics, and a faster DPM variant using reverse diffusion.

In the DDPM-style formulation later adopted in the VoiceGrad family, the forward diffusion process is

$$x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),$$

and the conditional reverse mean is

$$\mu_\theta(x_t,t,s,p)=\frac{1}{\sqrt{\alpha_t}}\left[x_t-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\epsilon_\theta(x_t,t,s,p)\right].$$

VoiceGrad trains the denoiser with an L1 objective rather than the canonical L2 objective:

$$L_{\mathrm{DDPM}}(\theta)=\mathbb{E}\big[\lVert \epsilon-\epsilon_\theta(x_t,t,s_{\mathrm{src}},p_{\mathrm{src}})\rVert_1\big].$$

In the original paper’s notation, the DPM training loss is written with diffusion step $l$0 and speaker condition $l$1, but the operative idea is the same: predict the injected Gaussian noise from a corrupted mel-spectrogram (Kameoka et al., 2020).

The defining inference choice is the initialization. Unlike unconditional diffusion generation, VoiceGrad does not initialize from pure noise; it starts from the source mel-spectrogram itself. In the DSM variant, the source spectrogram is iteratively updated by annealed Langevin dynamics. In the DPM variant, the source spectrogram is treated as a partially diffused sample and updated through the reverse chain. This “warm start” is central to content preservation (Kameoka et al., 2020).

The original paper shows a marked efficiency difference between DSM and DPM. DSM uses $l$2, $l$3, $l$4 inner iterations per level, for approximately 576 iterations, whereas DPM uses $l$5, $l$6, for 11 reverse steps (Kameoka et al., 2020). On CMU ARCTIC, VoiceGrad DPM+BNF reports MCD 6.27, LFC 0.55, CER 2.24%, and pMOS 3.59, outperforming StarGAN-VC, AutoVC, and PPG-VC on the reported closed-set averages; subjective MOS for audio quality and speaker similarity are 3.77/3.81 and 3.58/3.67 for closed/open settings, respectively (Kameoka et al., 2020).

3. Conditioning pathways and architectural organization

The original VoiceGrad uses a fully convolutional U-Net-like architecture operating on mel-spectrograms, with 1D Conv/Deconv, skip connections, GLU activations, and weight normalization (Kameoka et al., 2020). Target-speaker and noise-level embeddings are repeated along time and concatenated at each block. When BNF conditioning is enabled, a strided Conv1d processes the 144-dimensional BNF to match the temporal resolution of each block and appends it along the channel dimension (Kameoka et al., 2020).

Later VoiceGrad-family implementations standardize several architectural elements. In FastVoiceGrad and FasterVoiceGrad, the reverse diffusion network is a U-Net-like 1D CNN (12 layers, 512 channels), with two down/upsampling stages, GLU activations, weight normalization, operating in the 80-dim log-mel spectrogram domain (Kaneko et al., 2024, Kaneko et al., 25 Aug 2025). These versions use sinusoidal positional or time embeddings and condition on a target speaker embedding $l$7 plus a content embedding $l$8. A pretrained GE2E-style speaker encoder is used both for conditioning and, in FasterVoiceGrad, for evaluation (Kaneko et al., 25 Aug 2025).

The treatment of content features is a major axis of variation. FastVoiceGrad retains a heavy, frozen content encoder, with examples including Conformer BNE, HuBERT, XLS-R, to compute $l$9 (Kaneko et al., 25 Aug 2025). FasterVoiceGrad replaces this with a lightweight 1D CNN (3 layers, 512 channels) with GLU, instance normalization, and weight normalization, denoted $s$0, and trains it jointly with the one-step diffusion student (Kaneko et al., 25 Aug 2025). This replacement is not a small implementation detail; it is the central mechanism by which FasterVoiceGrad reduces end-to-end latency.

LatentVoiceGrad modifies the computational bottleneck more radically. It introduces an autoencoder with encoder $s$1 and decoder $s$2, with architecture FC(256) + LReLU, BiLSTM×2(256), FC(32) for the encoder and FC(256) + LReLU, BiLSTM×2(256), FC(80) for the decoder, yielding a 32-channel latent representation $s$3 (Kameoka et al., 10 Sep 2025). The score or vector-field network then operates in this bottleneck rather than in mel space. This suggests a shift from direct feature-space denoising toward low-dimensional latent transport, while preserving the same conditioning interface $s$4.

Across the family, the waveform pathway is consistently mediated by a pretrained vocoder. The reported systems use HiFi-GAN variants, while the adversarial discriminators are described as multiresolution, or as multi-period + multi-resolution, depending on the paper (Kaneko et al., 2024, Kaneko et al., 25 Aug 2025, Kameoka et al., 10 Sep 2025).

4. One-step distillation: FastVoiceGrad and FasterVoiceGrad

The most immediate limitation of VoiceGrad is iterative sampling. Multi-step reverse diffusion incurs high inference latency and compute relative to feed-forward VC models. FastVoiceGrad addresses this by distilling the multi-step VoiceGrad teacher into a one-step student using Adversarial Conditional Diffusion Distillation (ACDD) (Kaneko et al., 2024).

A key change is the sampling initialization. Rather than starting one-step conversion from the clean source mel-spectrogram, FastVoiceGrad starts from a diffused source:

$s$5

with $s$6 in the one-step setting (Kaneko et al., 2024). The student then predicts the converted mel in a single reverse step, while training combines three losses: a waveform-domain LSGAN adversarial loss using a frozen vocoder, a feature matching loss, and a score distillation loss aligning the student output to the teacher’s denoised output (Kaneko et al., 2024). On VCTK, FastVoiceGrad reports qMOS 3.86±0.09, sMOS 2.68±0.16, UTMOS 3.96, DNSMOS 3.77, CER 1.89%, and SVA 83.0% (Kaneko et al., 2024).

FasterVoiceGrad begins from the observation that one-step reverse diffusion alone does not remove the latency imposed by the heavy content encoder. It therefore introduces Adversarial Diffusion Conversion Distillation (ADCD), which simultaneously distills the one-step reverse diffusion model $s$7 and the lightweight content encoder $s$8 (Kaneko et al., 25 Aug 2025). The distillation is performed in the conversion process while leveraging adversarial and score distillation training, not in reconstruction.

ADCD has four principal components. First, the student performs one-step conversion as

$s$9

with $p$0 (Kaneko et al., 25 Aug 2025). Second, conversion score distillation aligns this converted output with the teacher’s denoised output under the same conversion condition. Third, reconversion score distillation applies conversion again to the already converted output and distills the result toward the teacher, with the stated intuition that this trains $p$1 to preserve content across conversions. Fourth, inverse score distillation repels the student output from teacher denoising under a randomly sampled non-target speaker, emphasizing target identity (Kaneko et al., 25 Aug 2025).

A notable design decision is negative: FasterVoiceGrad does not introduce an explicit content matching term of the form $p$2. The paper reports that direct content encoder distillation of this form degrades performance when $p$3 has small capacity (Kaneko et al., 25 Aug 2025). The final reported weights are $p$4, $p$5, and $p$6 (Kaneko et al., 25 Aug 2025).

On VCTK, FasterVoiceGrad reports UTMOS 4.03, DNSMOS 3.82, CER 1.2, SECS 0.853, RTFGPU 0.00085, and RTFCPU 0.0747, compared with FastVoiceGrad’s UTMOS 3.96, DNSMOS 3.77, CER 1.3, SECS 0.847, RTFGPU 0.00560, and RTFCPU 0.1347 (Kaneko et al., 25 Aug 2025). The stated speed gain is 6.6–6.9× on GPU and 1.8× on CPU over FastVoiceGrad (Kaneko et al., 25 Aug 2025).

5. LatentVoiceGrad: latent diffusion and flow matching

LatentVoiceGrad addresses two stated difficulties in the original VoiceGrad: speech quality still needs improvement, and conversion remains slower than modern high-speed VC methods. The proposed response is twofold: perform reverse diffusion in a learned autoencoder bottleneck, and replace diffusion with flow matching as an alternative transport model (Kameoka et al., 10 Sep 2025).

The autoencoder is trained with a composite objective that includes mel-domain reconstruction L1, adversarial waveform losses with a frozen universal HiFi-GAN and discriminator combining MPD and MSD, auxiliary mel L1, feature matching L1, and KL regularization to make the latent approximately standard normal. The final autoencoder objective is

$p$7

Reverse diffusion is then run in the 32-channel latent $p$8, with final synthesis given by decoding and vocoding $p$9 (Kameoka et al., 10 Sep 2025).

The flow-matching alternative learns a time-dependent vector field $s$0 satisfying

$s$1

and trains it with a conditional flow matching objective based on straight-line Gaussian conditionals (Kameoka et al., 10 Sep 2025). At inference, the latent can be initialized by optional noise mixing $s$2, followed by Euler steps. The paper states that $s$3 gives a favorable speed-quality balance, while $s$4 sharply degrades quality and intelligibility (Kameoka et al., 10 Sep 2025).

On VCTK v0.92, the reported mel-spectrogram-setting results are VoiceGrad-DPM: pMOS 3.86±0.02, CER 2.88, SECS 0.830±0.05; LatentVoiceGrad-DPM: pMOS 3.93±0.01, CER 2.99, SECS 0.844±0.04; VoiceGrad-FM: pMOS 3.85±0.02, CER 2.95, SECS 0.817±0.05; and LatentVoiceGrad-FM: pMOS 3.92±0.02, CER 3.21, SECS 0.827±0.05 (Kameoka et al., 10 Sep 2025). Subjective listening reports qMOS 4.09±0.08; sMOS 3.05±0.15 for LatentVoiceGrad-DPM and qMOS 4.16±0.09; sMOS 3.02±0.17 for LatentVoiceGrad-FM, with ground truth qMOS 4.21±0.19 (Kameoka et al., 10 Sep 2025).

Latency is correspondingly reduced. Excluding HiFi-GAN time, reported mel-conversion RTFs are VoiceGrad-DPM (L=20): GPU 0.045, CPU 0.227; LatentVoiceGrad-DPM (L=20): GPU 0.034, CPU 0.166; VoiceGrad-FM, L=10: GPU 0.023, CPU 0.114; and LatentVoiceGrad-FM, L=10: GPU 0.022, CPU 0.111 (Kameoka et al., 10 Sep 2025).

6. Empirical patterns, trade-offs, and limitations

Across the literature, VoiceGrad exhibits a consistent pattern: stronger conditioning improves intelligibility, while reduced sampling depth improves speed only if the content pathway is appropriately controlled. In the original paper, adding BNF yields a large reduction in CER and improvements in MCD, LFC, and pMOS for both DSM and DPM (Kameoka et al., 2020). In the one-step line, replacing iterative denoising with a distilled student is viable, but naive training of a trainable content encoder under reconstruction can degenerate toward an identity-preserving content path (Kaneko et al., 25 Aug 2025).

The FasterVoiceGrad ablations make this explicit. The baseline “FastVoiceGrad + $s$5” reports UTMOS 3.45, DNSMOS 3.64, CER 0.4, and SECS 0.718, which the paper interprets as overfitting the content path: intelligibility is high, but speaker conversion is poor (Kaneko et al., 25 Aug 2025). Adding conversion (ADCD) raises SECS to 0.847; adding reconversion yields UTMOS 4.03, DNSMOS 3.79, CER 1.3, SECS 0.844; adding inverse produces UTMOS 4.03, DNSMOS 3.82, CER 1.2, SECS 0.853, mitigating the content/speaker trade-off (Kaneko et al., 25 Aug 2025).

A second recurring pattern is the non-equivalence of objective and perceptual speaker similarity. FasterVoiceGrad reports higher SECS than FastVoiceGrad, yet lower subjective sMOS (2.66±0.12 versus 2.84±0.11) while improving qMOS (3.81±0.09 versus 3.60±0.09) (Kaneko et al., 25 Aug 2025). The authors attribute this gap to residual source characteristics in $s$6 not completely disentangled perceptually, and further note that residual timbre may be undetected by speaker encoders yet audible to humans (Kaneko et al., 25 Aug 2025). This is one of the clearest empirical cautions in the VoiceGrad literature.

LatentVoiceGrad introduces a different trade-off. In the flow-matching setting, larger noise-mix $s$7 can increase speaker similarity but degrade intelligibility, and very few ODE steps sharply reduce quality (Kameoka et al., 10 Sep 2025). In FastVoiceGrad, choosing the one-step initialization too close to pure noise can also degrade intelligibility and quality (Kaneko et al., 2024). The broader implication is that acceleration in VoiceGrad is constrained not only by denoising capacity but also by how much source structure is retained at initialization and in the content pathway.

The published limitations are correspondingly concrete. Original VoiceGrad has no explicit $s$8 conversion module, and explicit $s$9 modeling is identified as a possible improvement for cross-gender conversion (Kameoka et al., 2020). FasterVoiceGrad evaluates only on English datasets (VCTK, LibriTTS), and robustness under multilingual, code-switching, accents, or noisy data remains to be established (Kaneko et al., 25 Aug 2025). LatentVoiceGrad reports that cross-lingual VC was not investigated in this work (Kameoka et al., 10 Sep 2025). One-shot scenarios with very short or noisy enrollment utterances may degrade, and all of the adversarial variants depend on a pretrained vocoder whose artifacts or domain mismatch can influence optimization (Kaneko et al., 25 Aug 2025).

Taken together, these papers define VoiceGrad less as a single static model than as a research program in conditioned score-based VC. The original contribution is the formulation of non-parallel conversion as score-guided transport in mel-spectrogram space (Kameoka et al., 2020). FastVoiceGrad and FasterVoiceGrad show that this transport can be compressed into a single reverse step if teacher guidance and adversarial objectives are arranged carefully (Kaneko et al., 2024, Kaneko et al., 25 Aug 2025). LatentVoiceGrad shows that the same conditioning logic can be embedded in a bottleneck latent space and even recast as flow matching, yielding further efficiency gains without abandoning the central VoiceGrad premise (Kameoka et al., 10 Sep 2025).