---
title: 'VoiceGrad: Score-Based Voice Conversion'
url: https://www.emergentmind.com/topics/voicegrad
type: topic
---

# VoiceGrad: Score-Based Voice Conversion

VoiceGrad is a non-parallel voice conversion framework that treats conversion as conditioned score-based generation in mel-spectrogram space: a source utterance is transformed so as to preserve linguistic content while approaching the target speaker distribution. The original formulation introduced non-parallel any-to-many voice conversion with weighted denoising score matching, annealed Langevin dynamics, and a reverse-diffusion variant [2010.02977]. Later work recast the conditioning for zero-shot any-to-any conversion, distilled the iterative model into one-step students, and moved the reverse process into latent diffusion and flow-matching formulations to improve latency and audio quality [2409.02245], [2508.17868], [2509.08379].

## 1. Problem setting and representational scope

Voice conversion (VC) is the task of transforming a source speaker’s voice to sound like a specified target speaker while preserving the linguistic content. In the VoiceGrad line, the task is explicitly **non-parallel**: training does not require parallel utterances or transcriptions. The original system is **any-to-many**, in which the source speaker at test time can be arbitrary and conversion is performed to one of multiple target speakers represented during training [2010.02977]. Later systems adopt speaker embeddings rather than speaker IDs, enabling **one-shot any-to-any VC** and **zero-shot unseen-to-unseen VC** [2409.02245], [2509.08379].

A persistent misconception is that VoiceGrad is a waveform-generation model. It is not. Across the reported variants, the conversion target is the **mel-spectrogram**, typically with **80 channels**, and waveform synthesis is delegated to a neural vocoder such as **HiFi-GAN** [2010.02977], [2409.02245]. This design separates speaker/content conversion from waveform rendering.

Conditioning evolves across the literature. The original VoiceGrad conditions on a target speaker identity \(k\), a noise level \(l\), and optionally a **144-dimensional bottleneck feature (BNF)** sequence extracted by a pretrained ASR bottleneck model [2010.02977]. Later DDPM-style formulations use a target speaker embedding \(s\) and a content or phoneme embedding \(p\), where \(s\) is obtained either from a pretrained speaker encoder or a d-vector extractor, and \(p\) is obtained from the source utterance through a BNF, phoneme feature extractor, or content encoder [2409.02245], [2509.08379]. In the FastVoiceGrad line, pitch conditioning is not used, although VoiceGrad/FastVoiceGrad can, in principle, incorporate \(F0\) as an additional conditioning channel [2409.02245].

## 2. Original VoiceGrad: score matching, annealed Langevin dynamics, and reverse diffusion

The original VoiceGrad is a score-based, diffusion-inspired approach that trains a score approximator to predict the gradient of the log density of speech feature sequences from multiple speakers, then performs conversion by iteratively updating the source feature sequence toward a nearest stationary point of the target distribution [2010.02977]. Two formulations are reported: a **DSM** variant using annealed Langevin dynamics, and a faster **DPM** variant using reverse diffusion.

In the DDPM-style formulation later adopted in the VoiceGrad family, the forward diffusion process is

$$
x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon,\qquad \epsilon \sim \mathcal{N}(0,I),
$$

and the conditional reverse mean is

$$
\mu_\theta(x_t,t,s,p)=\frac{1}{\sqrt{\alpha_t}}\left[x_t-\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\epsilon_\theta(x_t,t,s,p)\right].
$$

VoiceGrad trains the denoiser with an **L1** objective rather than the canonical L2 objective:

$$
L_{\mathrm{DDPM}}(\theta)=\mathbb{E}\big[\lVert \epsilon-\epsilon_\theta(x_t,t,s_{\mathrm{src}},p_{\mathrm{src}})\rVert_1\big].
$$

In the original paper’s notation, the DPM training loss is written with diffusion step \(l\) and speaker condition \(k\), but the operative idea is the same: predict the injected Gaussian noise from a corrupted mel-spectrogram [2010.02977].

The defining inference choice is the initialization. Unlike unconditional diffusion generation, VoiceGrad does **not** initialize from pure noise; it starts from the **source mel-spectrogram itself**. In the DSM variant, the source spectrogram is iteratively updated by annealed Langevin dynamics. In the DPM variant, the source spectrogram is treated as a partially diffused sample and updated through the reverse chain. This “warm start” is central to content preservation [2010.02977].

The original paper shows a marked efficiency difference between DSM and DPM. DSM uses \(L=21\), \(L' = 4\), \(T=32\) inner iterations per level, for approximately **576 iterations**, whereas DPM uses \(L=20\), \(L' = 11\), for **11 reverse steps** [2010.02977]. On CMU ARCTIC, **VoiceGrad DPM+BNF** reports **MCD 6.27**, **LFC 0.55**, **CER 2.24%**, and **pMOS 3.59**, outperforming StarGAN-VC, AutoVC, and PPG-VC on the reported closed-set averages; subjective MOS for audio quality and speaker similarity are **3.77/3.81** and **3.58/3.67** for closed/open settings, respectively [2010.02977].

## 3. Conditioning pathways and architectural organization

The original VoiceGrad uses a **fully convolutional U-Net-like architecture** operating on mel-spectrograms, with **1D Conv/Deconv**, **skip connections**, **GLU activations**, and **weight normalization** [2010.02977]. Target-speaker and noise-level embeddings are repeated along time and concatenated at each block. When BNF conditioning is enabled, a **strided Conv1d** processes the 144-dimensional BNF to match the temporal resolution of each block and appends it along the channel dimension [2010.02977].

Later VoiceGrad-family implementations standardize several architectural elements. In FastVoiceGrad and FasterVoiceGrad, the reverse diffusion network is a **U-Net-like 1D CNN (12 layers, 512 channels), with two down/upsampling stages, GLU activations, weight normalization**, operating in the **80-dim log-mel spectrogram** domain [2409.02245], [2508.17868]. These versions use **sinusoidal positional or time embeddings** and condition on a target speaker embedding \(s\) plus a content embedding \(p\). A pretrained **GE2E-style** speaker encoder is used both for conditioning and, in FasterVoiceGrad, for evaluation [2508.17868].

The treatment of content features is a major axis of variation. FastVoiceGrad retains a **heavy, frozen content encoder**, with examples including **Conformer BNE, HuBERT, XLS-R**, to compute \(p_{\mathrm{src}}\) [2508.17868]. FasterVoiceGrad replaces this with a **lightweight 1D CNN (3 layers, 512 channels) with GLU, instance normalization, and weight normalization**, denoted \(p_\phi\), and trains it jointly with the one-step diffusion student [2508.17868]. This replacement is not a small implementation detail; it is the central mechanism by which FasterVoiceGrad reduces end-to-end latency.

LatentVoiceGrad modifies the computational bottleneck more radically. It introduces an autoencoder with encoder \(f_\phi\) and decoder \(g_\phi\), with architecture **FC(256) + LReLU, BiLSTM×2(256), FC(32)** for the encoder and **FC(256) + LReLU, BiLSTM×2(256), FC(80)** for the decoder, yielding a **32-channel** latent representation \(z\) [2509.08379]. The score or vector-field network then operates in this bottleneck rather than in mel space. This suggests a shift from direct feature-space denoising toward low-dimensional latent transport, while preserving the same conditioning interface \(c=(s,p)\).

Across the family, the waveform pathway is consistently mediated by a pretrained vocoder. The reported systems use **HiFi-GAN** variants, while the adversarial discriminators are described as **multiresolution**, or as **multi-period + multi-resolution**, depending on the paper [2409.02245], [2508.17868], [2509.08379].

## 4. One-step distillation: FastVoiceGrad and FasterVoiceGrad

The most immediate limitation of VoiceGrad is iterative sampling. Multi-step reverse diffusion incurs high inference latency and compute relative to feed-forward VC models. FastVoiceGrad addresses this by distilling the multi-step VoiceGrad teacher into a **one-step** student using **Adversarial Conditional Diffusion Distillation (ACDD)** [2409.02245].

A key change is the sampling initialization. Rather than starting one-step conversion from the clean source mel-spectrogram, FastVoiceGrad starts from a **diffused source**:

$$
x_{S_K}^{\mathrm{src}}=\sqrt{\bar{\alpha}_{S_K}}\,x_0^{\mathrm{src}}+\sqrt{1-\bar{\alpha}_{S_K}}\,\epsilon,
$$

with **\(S_K=950\)** in the one-step setting [2409.02245]. The student then predicts the converted mel in a single reverse step, while training combines three losses: a waveform-domain **LSGAN adversarial loss** using a frozen vocoder, a **feature matching** loss, and a **score distillation** loss aligning the student output to the teacher’s denoised output [2409.02245]. On VCTK, FastVoiceGrad reports **qMOS 3.86±0.09**, **sMOS 2.68±0.16**, **UTMOS 3.96**, **DNSMOS 3.77**, **CER 1.89%**, and **SVA 83.0%** [2409.02245].

FasterVoiceGrad begins from the observation that one-step reverse diffusion alone does not remove the latency imposed by the heavy content encoder. It therefore introduces **Adversarial Diffusion Conversion Distillation (ADCD)**, which simultaneously distills the one-step reverse diffusion model \(\mu_\phi\) and the lightweight content encoder \(p_\phi\) [2508.17868]. The distillation is performed **in the conversion process while leveraging adversarial and score distillation training**, not in reconstruction.

ADCD has four principal components. First, the student performs one-step conversion as

$$
x_{\mathrm{cv}}^\phi=\mu_\phi(x_{t'},t',s_{\mathrm{tgt}},p_\phi^{\mathrm{src}}),
$$

with **\(t'=950\)** [2508.17868]. Second, **conversion score distillation** aligns this converted output with the teacher’s denoised output under the same conversion condition. Third, **reconversion score distillation** applies conversion again to the already converted output and distills the result toward the teacher, with the stated intuition that this trains \(p_\phi\) to preserve content across conversions. Fourth, **inverse score distillation** repels the student output from teacher denoising under a randomly sampled non-target speaker, emphasizing target identity [2508.17868].

A notable design decision is negative: FasterVoiceGrad **does not introduce an explicit content matching term** of the form \(L_{\mathrm{content}} = E[\|p_\phi(x)-p_{\mathrm{teacher}}(x)\|_p]\). The paper reports that direct content encoder distillation of this form degrades performance when \(p_\phi\) has small capacity [2508.17868]. The final reported weights are **\(\lambda_{\mathrm{FM}}=2\)**, **\(\lambda_{\mathrm{dist}}=45\)**, and **\(\lambda_{\mathrm{inv-dist}}=22.5\)** [2508.17868].

On VCTK, FasterVoiceGrad reports **UTMOS 4.03**, **DNSMOS 3.82**, **CER 1.2**, **SECS 0.853**, **RTFGPU 0.00085**, and **RTFCPU 0.0747**, compared with FastVoiceGrad’s **UTMOS 3.96**, **DNSMOS 3.77**, **CER 1.3**, **SECS 0.847**, **RTFGPU 0.00560**, and **RTFCPU 0.1347** [2508.17868]. The stated speed gain is **6.6–6.9×** on GPU and **1.8×** on CPU over FastVoiceGrad [2508.17868].

## 5. LatentVoiceGrad: latent diffusion and flow matching

LatentVoiceGrad addresses two stated difficulties in the original VoiceGrad: speech quality still needs improvement, and conversion remains slower than modern high-speed VC methods. The proposed response is twofold: perform reverse diffusion in a learned autoencoder bottleneck, and replace diffusion with **flow matching** as an alternative transport model [2509.08379].

The autoencoder is trained with a composite objective that includes **mel-domain reconstruction L1**, **adversarial waveform losses** with a frozen universal HiFi-GAN and discriminator combining **MPD and MSD**, **auxiliary mel L1**, **feature matching L1**, and **KL regularization** to make the latent approximately standard normal. The final autoencoder objective is

$$
\mathcal{J}_{\rm ae}(\phi)=
\lambda\big(\mathcal{J}_{\rm rec}+\mathcal{J}_{\rm mel}+\mathcal{J}_{\rm KL}\big)
+\mathcal{J}_{\rm adv}^{(a)}+\mathcal{J}_{\rm feat},
\qquad \lambda=45.
$$

Reverse diffusion is then run in the 32-channel latent \(z\), with final synthesis given by decoding and vocoding \(y=h(g_\phi(z))\) [2509.08379].

The flow-matching alternative learns a time-dependent vector field \(v_\theta(x,t,c)\) satisfying

$$
\frac{dx_t}{dt}=v_\theta(x_t,t,c),\qquad c=(s,p),
$$

and trains it with a **conditional flow matching** objective based on straight-line Gaussian conditionals [2509.08379]. At inference, the latent can be initialized by optional noise mixing \(z \leftarrow (1-r)z + r\epsilon\), followed by **Euler steps**. The paper states that **\(L \approx 10\)** gives a favorable speed-quality balance, while **\(L \le 2\)** sharply degrades quality and intelligibility [2509.08379].

On VCTK v0.92, the reported mel-spectrogram-setting results are **VoiceGrad-DPM: pMOS 3.86±0.02, CER 2.88, SECS 0.830±0.05**; **LatentVoiceGrad-DPM: pMOS 3.93±0.01, CER 2.99, SECS 0.844±0.04**; **VoiceGrad-FM: pMOS 3.85±0.02, CER 2.95, SECS 0.817±0.05**; and **LatentVoiceGrad-FM: pMOS 3.92±0.02, CER 3.21, SECS 0.827±0.05** [2509.08379]. Subjective listening reports **qMOS 4.09±0.08; sMOS 3.05±0.15** for LatentVoiceGrad-DPM and **qMOS 4.16±0.09; sMOS 3.02±0.17** for LatentVoiceGrad-FM, with **ground truth qMOS 4.21±0.19** [2509.08379].

Latency is correspondingly reduced. Excluding HiFi-GAN time, reported mel-conversion RTFs are **VoiceGrad-DPM (L=20): GPU 0.045, CPU 0.227**; **LatentVoiceGrad-DPM (L=20): GPU 0.034, CPU 0.166**; **VoiceGrad-FM, L=10: GPU 0.023, CPU 0.114**; and **LatentVoiceGrad-FM, L=10: GPU 0.022, CPU 0.111** [2509.08379].

## 6. Empirical patterns, trade-offs, and limitations

Across the literature, VoiceGrad exhibits a consistent pattern: stronger conditioning improves intelligibility, while reduced sampling depth improves speed only if the content pathway is appropriately controlled. In the original paper, adding BNF yields a large reduction in CER and improvements in MCD, LFC, and pMOS for both DSM and DPM [2010.02977]. In the one-step line, replacing iterative denoising with a distilled student is viable, but naive training of a trainable content encoder under reconstruction can degenerate toward an identity-preserving content path [2508.17868].

The FasterVoiceGrad ablations make this explicit. The baseline **“FastVoiceGrad + \(p_\phi\)”** reports **UTMOS 3.45**, **DNSMOS 3.64**, **CER 0.4**, and **SECS 0.718**, which the paper interprets as overfitting the content path: intelligibility is high, but speaker conversion is poor [2508.17868]. Adding **conversion (ADCD)** raises **SECS to 0.847**; adding **reconversion** yields **UTMOS 4.03**, **DNSMOS 3.79**, **CER 1.3**, **SECS 0.844**; adding **inverse** produces **UTMOS 4.03**, **DNSMOS 3.82**, **CER 1.2**, **SECS 0.853**, mitigating the content/speaker trade-off [2508.17868].

A second recurring pattern is the non-equivalence of objective and perceptual speaker similarity. FasterVoiceGrad reports higher **SECS** than FastVoiceGrad, yet lower subjective **sMOS** (**2.66±0.12** versus **2.84±0.11**) while improving **qMOS** (**3.81±0.09** versus **3.60±0.09**) [2508.17868]. The authors attribute this gap to **residual source characteristics in \(p_\phi\) not completely disentangled perceptually**, and further note that **residual timbre may be undetected by speaker encoders yet audible to humans** [2508.17868]. This is one of the clearest empirical cautions in the VoiceGrad literature.

LatentVoiceGrad introduces a different trade-off. In the flow-matching setting, larger noise-mix \(r\) can increase speaker similarity but degrade intelligibility, and very few ODE steps sharply reduce quality [2509.08379]. In FastVoiceGrad, choosing the one-step initialization too close to pure noise can also degrade intelligibility and quality [2409.02245]. The broader implication is that acceleration in VoiceGrad is constrained not only by denoising capacity but also by how much source structure is retained at initialization and in the content pathway.

The published limitations are correspondingly concrete. Original VoiceGrad has no explicit \(F0\) conversion module, and explicit \(F0\) modeling is identified as a possible improvement for cross-gender conversion [2010.02977]. FasterVoiceGrad evaluates only on **English datasets (VCTK, LibriTTS)**, and robustness under **multilingual, code-switching, accents, or noisy data remains to be established** [2508.17868]. LatentVoiceGrad reports that **cross-lingual VC was not investigated in this work** [2509.08379]. One-shot scenarios with very short or noisy enrollment utterances may degrade, and all of the adversarial variants depend on a pretrained vocoder whose artifacts or domain mismatch can influence optimization [2508.17868].

Taken together, these papers define VoiceGrad less as a single static model than as a research program in conditioned score-based VC. The original contribution is the formulation of non-parallel conversion as score-guided transport in mel-spectrogram space [2010.02977]. FastVoiceGrad and FasterVoiceGrad show that this transport can be compressed into a single reverse step if teacher guidance and adversarial objectives are arranged carefully [2409.02245], [2508.17868]. LatentVoiceGrad shows that the same conditioning logic can be embedded in a bottleneck latent space and even recast as flow matching, yielding further efficiency gains without abandoning the central VoiceGrad premise [2509.08379].

Source: https://www.emergentmind.com/topics/voicegrad