---
title: 'SemAlignVC: Zero-shot Voice Conversion'
url: https://www.emergentmind.com/topics/semalignvc
type: topic
---

# SemAlignVC: Zero-shot Voice Conversion

Searching arXiv for SemAlignVC and closely related zero-shot voice conversion work.
SemAlignVC is a zero-shot voice conversion architecture designed to prevent timbre leakage by using **SemAlign**, a method that aligns text and audio representations to enforce speaker-independent semantic encoding. In the reported formulation, this disentangled representation conditions an autoregressive transformer for high-fidelity conversion without explicit speaker embeddings. The method is positioned within neural codec- and LLM-based voice conversion, where quantized representations often entangle linguistic content, paralinguistic information, and speaker identity; SemAlignVC is presented as a privacy-preserving and generalizable solution for any-to-any conversion [2507.09070].

## 1. Problem formulation and motivation

SemAlignVC addresses **zero-shot voice conversion**, also described as **any-to-any VC**: given a source utterance, the system synthesizes speech that preserves the source content while adopting the timbre of an unseen target speaker, without requiring paired source-target utterances. The central difficulty is **timbre leakage**, namely the persistence of source-speaker traits in the converted output. The paper identifies this as a particular challenge in **neural codec-based VC** and **LLM-based VC**, because the discrete or quantized speech units used in such systems can entangle semantic content, paralinguistic content, and speaker timbre [2507.09070].

The notation used in the model description decomposes an utterance into semantic and timbre components. For an utterance \(X_{a_s,a_t}\), where \(a_s\) denotes semantics and \(a_t\) denotes timbre, the reconstruction objective is written as
\[
\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).
\]
For voice conversion, the desired inference combines the semantics of one speaker with the timbre of another:
\[
\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).
\]
The paper stresses that the model is **not trained on such mixed combinations**, which makes disentanglement necessary for generalization.

A second motivation concerns privacy and generalization. The paper notes that approaches relying on **explicit speaker embeddings** may expose identifiable speaker traits and may generalize poorly in any-to-any settings. SemAlignVC therefore attempts to remove speaker identity from the content pathway rather than representing it directly in a dedicated embedding.

## 2. Architecture and representational pipeline

SemAlignVC is organized as **four individually trained components**: an **audio tokenizer**, a **semantic language model (semantic-LLM)**, an **acoustic model**, and a **vocoder**. The architecture is described as having a training view for the semantic LLM and a separate inference pipeline for VC [2507.09070].

| Component | Role | Key details |
|---|---|---|
| Audio tokenizer | Converts speech to discrete tokens | Pretrained BEST-RQ-based VQ-VAE |
| Semantic encoder \(Q_\phi\) | Extracts speaker-independent semantics | Operates on audio tokens |
| Semantic LLM | Predicts audio tokens from semantics and prosody | LLaMA-style autoregressive transformer |
| Acoustic model | Converts tokens to mel spectrograms | Conditional flow matching-based transformer |
| Vocoder | Converts mel to waveform | BigVGAN |

The tokenizer maps an utterance to discrete audio tokens:
\[
z_a = H_{Enc}(X_{a_s, a_t}),
\]
where \(H_{Enc}\) is the audio tokenizer encoder and \(z_a\) is the discrete token sequence. The paper explicitly emphasizes that \(z_a\) still entangles timbre and content. A semantic encoder \(Q_\phi\) is then introduced to extract semantic content from these tokens:
\[
\hat{a}_s = Q_\phi(z_a).
\]
Here, \(\hat{a}_s\) is the predicted semantic representation, and the encoder is trained to filter out timbre.

Prosody is handled separately. The system extracts **pitch** \(f_0\) and **energy** \(e\) from the input audio, mean-normalizes them at the utterance level to remove speaker-related information, and feeds them to the LLM so that prosodic and paralinguistic cues are preserved. The semantic LLM models
\[
P_\theta(\hat{z}_a \mid \hat{a}_s, f_0, e),
\]
and the paper states that ideally \(\hat{z}_a \approx z\), where \(z\) denotes speaker-independent tokens containing the semantic and linguistic information of the input.

The acoustic model is described as a **conditional flow matching-based transformer** that converts audio tokens into mel spectrograms. The paper states that it models the conditional distribution
\[
p(\mu_a^{\text{mask} \mid \mu_a^{\text{ctx}, z_a)}.
\]
The source text explicitly notes a formatting issue in this expression, but the intended role is clear: the model reconstructs masked mel regions conditioned on contextual mel and audio tokens. A **BigVGAN** vocoder then synthesizes waveform audio from the predicted mel spectrogram.

## 3. SemAlign: text-audio alignment for speaker-independent semantics

The defining mechanism of SemAlignVC is **SemAlign**, a semantic alignment objective intended to make the speech content representation speaker-independent. Its basic premise is that **text carries semantic and linguistic information but no timbre**, whereas audio-derived semantic representations often retain speaker cues. Aligning audio semantics to text semantics is therefore used as an explicit pressure against timbre leakage [2507.09070].

The method does not rely on alignment by **CTC loss** alone. CTC is used as an auxiliary objective to encourage the semantic encoder to retain semantic information, in a manner similar to PPG-like prediction. However, the paper reports that **CTC alone did not sufficiently disentangle timbre**. The stated hypothesis is that CTC blank tokens can still carry unwanted speaker characteristics when modeled by a strong decoder. This distinction is central to the method’s rationale: SemAlign is presented as stronger than CTC because it ties the audio semantic representation to a text-only semantic space rather than only to token-sequence transduction.

The text-side signal is constructed from the utterance transcript. The transcript is tokenized with the **BERT tokenizer** and passed through **BERT** to obtain a text semantic embedding \(\tau_s\). The source text renders this as
\[
\tau_s = \operatorname{BERT}(),
\]
and explicitly notes a formatting omission in the expression. Because \(\tau_s\) and \(\hat{a}_s\) may have different temporal lengths and channels, \(\tau_s\) is **upsampled by repetition** to match the temporal dimension of \(\hat{a}_s\). The paper then minimizes the mean squared error alignment objective \(\mathcal{L}_{sem}\) between the aligned text semantics and the predicted audio semantics, using **Monotonic Alignment Search (MAS)**.

A further design choice is **gradient restriction** from the LLM decoder to the content encoder. Following prior work, the method restricts this gradient propagation to reduce the chance that speaker information leaks into the learned semantic representation. This suggests a deliberate separation between the semantic pathway and the autoregressive decoder pathway: the decoder should consume a clean semantic representation rather than shape it in a way that reintroduces speaker identity.

## 4. Conditioning, training objectives, and inference procedure

During semantic-LLM training, each utterance is split into a **primary input audio** and a **smaller excerpt** comprising about **25%** of the utterance. The mel spectrogram of this shorter excerpt is denoted \(\mu_a^{ref}\) and serves as the **reference timbre**. The semantic LLM is trained with the prompt
\[
\left[ [\text{SOS}], \hat{a}_s, [\text{SEP}], \mu_{a}^{ref}, [\text{SEP}], z_a, [\text{EOS}] \right].
\]
The cross-entropy loss \(\mathcal{L}_{LM}\) is computed **only for** \(z_a\) and \([\text{EOS}]\). In effect, the LLM is conditioned on the semantic representation \(\hat{a}_s\) and a timbre reference \(\mu_a^{ref}\), and is trained to predict the audio token sequence [2507.09070].

At inference time, the target-speaker reference \(\mu_b^{ref}\) is fed into both the **semantic-LLM** and the **acoustic model context**. The system then produces a converted mel spectrogram \(\hat{\mu}_b\), followed by the converted waveform \(\hat{X}_{a_s,b_t}\) through the vocoder. Although the method does not use explicit speaker embeddings, it still conditions generation on a target-speaker **reference mel** and on mean-normalized prosody features \(f_0\) and \(e\). A common misconception would be to equate “without explicit speaker embeddings” with “without target-speaker conditioning”; the architecture does not remove speaker conditioning, but relocates it to a reference-audio pathway.

Three training objectives are explicitly identified. The first is **CTC loss**, used to encourage semantic retention in \(Q_\phi\). The second is the **SemAlign loss** \(\mathcal{L}_{sem}\), the MSE-based text-audio alignment objective. The third is the **language modeling loss** \(\mathcal{L}_{LM}\), implemented as cross-entropy on the LLM output. The overall training is described as **modular**, with components trained separately rather than end-to-end.

The implementation details reported in the paper are specific. The semantic encoder uses **4 layers of Conformer** with hidden dimension \(d\), trained with a **beta-binomial prior** and alignment losses from **One TTS Alignment to Rule Them All**. The semantic LLM uses the same architecture as **LLaMA**, with hidden dimension \(d = 2048\), **8** layers, and a total size of about **0.5B parameters**. Optimization uses **AdamW**, a peak learning rate of **1e-4**, **2000 steps** of warmup, and decay after warmup. The acoustic model is identical to **VoiceBox**, uses a **Midpoint ODE solver**, and is run for **12 steps** with \(\text{NFE}=24\). The audio tokenizer is a BEST-RQ-based quantizer trained on a large internal dataset.

## 5. Evaluation protocol and empirical findings

The empirical analysis combines diagnostic experiments on representation quality with end-to-end voice conversion evaluation. A key diagnostic test is a **speaker classification experiment** designed to measure how much speaker information remains in different representations. The classifier is trained on **LibriHeavy** for **5 epochs** using **8 H100 GPUs**, and lower classification accuracy is better if the goal is to remove speaker identity [2507.09070].

The reported accuracies are:

- **EnCodec**: **96.7%**
- **HuBERT (layer 9)**: **71.7%**
- **Ours\(_{tok}\)**: **82.05%**
- **WavLM x-vectors**: **88.4%**
- **Ours\(_{Q_\phi}\)**: **2.84%**

These results show a large gap between the semantic encoder output \(Q_\phi\) and the alternative discrete or continuous speech representations. In the paper’s interpretation, the learned semantic encoder nearly eliminates speaker-class information. A companion **PCA analysis** on \(Q_\phi\) audio embeddings and text embeddings \(\tau_s\) for two utterances is reported to show strong alignment between the principal components of the two representations.

The subjective evaluation uses the **VCTK test set** with **10 unseen speakers** and **4 sentences per speaker**. Because the utterances were short, **two sentences were concatenated** to form longer inputs, producing **20** input utterances. References were taken from **6 unseen speakers** selected randomly from the VCTK test set, with **3 male** and **3 female** speakers, for a total of **120** audio samples. The metric is **SMOS**, a speaker-similarity mean opinion score on a **5-point Likert scale**. The study includes **120 participants**, **10 audio samples per model**, and **1,200 ratings per model**.

| Model | SMOS |
|---|---|
| KNNVC | \(2.77 \pm 0.09\) |
| HierSpeech++ | \(3.16 \pm 0.12\) |
| UniAudio | \(2.56 \pm 0.10\) |
| SemAlignVC | \(3.29 \pm 0.09\) |

The objective evaluation uses the **LibriHeavy test set**, with **50 utterances** from unseen speakers and **10 reference utterances** from different unseen speakers, for **50 × 10** samples per model. The metrics are **DNSMOS** for naturalness, **WER** via **Whisper Large v3**, **FPC** as the Pearson correlation of \(f_0\), and speaker-similarity scores from **WavLM**, **ECAPA-TDNN**, and **Resemblyzer**. The paper notes an input WER of **6.77%**.

For **naturalness**, SemAlignVC achieves **SIG 3.63**, **BAK 4.13**, and **OVRL 3.38**, which the paper reports as the best overall quality score. For **consistency**, the reported FPC values are **0.632** for KNNVC, **0.622** for SemAlignVC, **0.614** for HierSpeech++, and **0.610** for UniAudio, placing SemAlignVC close to the best result. For **intelligibility**, WER is **18.85%** for KNNVC, **8.24%** for HierSpeech++, **9.98%** for UniAudio, and **12.31%** for SemAlignVC. The method is therefore better than KNNVC on WER, but not as low as HierSpeech++ and UniAudio. For **speaker similarity**, SemAlignVC is reported as best on all three similarity metrics: **WavLM 0.95**, **ECAPA 0.82**, and **Resemblyzer 0.89**.

## 6. Interpretation, scope, and limitations

The reported results support a specific interpretation of how SemAlignVC operates. Standard speech representations such as EnCodec tokens, HuBERT layer-9 tokens, WavLM x-vectors, and the paper’s tokenizer outputs retain substantial speaker information, whereas the learned semantic encoder \(Q_\phi\) does not. In that sense, SemAlignVC treats **semantic alignment with text** as the primary disentanglement mechanism. This suggests that text can function as an anchor for content semantics in codec- and LLM-based VC, rather than merely as a supervision target [2507.09070].

The method’s main advantages, as stated or directly supported by the reported experiments, are reduced timbre leakage, strong target-speaker similarity, and improved naturalness without explicit speaker embeddings. The paper also presents the system as **privacy-preserving** and **generalizable** to unseen speakers and mismatched source-target combinations. At the same time, these strengths are qualified by several limitations. **Intelligibility is not best-in-class**, since WER remains higher than HierSpeech++ and UniAudio. **Prosodic consistency is close to, but not the highest among, the compared systems**. The paper also notes that **BERT/text-embedding artifacts may cause synonym-like substitutions**, observed in pure TTS experiments and suspected to contribute to errors.

Two structural constraints further define the scope of the method. First, SemAlignVC **depends on transcripts**, because SemAlign uses text alignment; it is therefore not a text-free system. Second, its components are **trained separately**, not as an end-to-end unified architecture. These properties matter for interpretation: SemAlignVC is best understood not as a universal disentanglement proof, but as a modular VC system in which text-audio semantic alignment is used to suppress speaker identity in the content representation while target timbre is supplied through a reference-audio pathway.

Source: https://www.emergentmind.com/topics/semalignvc