SemAlignVC: Zero-shot Voice Conversion

• SemAlignVC is a zero-shot voice conversion system that disentangles speaker timbre from linguistic content using semantic alignment between text and audio.
• It employs a modular pipeline consisting of an audio tokenizer, semantic encoder, autoregressive LLM, acoustic transformer, and BigVGAN vocoder to achieve robust conversion.
• The approach enhances privacy and target-speaker similarity by conditioning on reference mel spectrograms while mitigating timbre leakage through text grounding.

Searching arXiv for SemAlignVC and closely related zero-shot voice conversion work. SemAlignVC is a zero-shot voice conversion architecture designed to prevent timbre leakage by using SemAlign, a method that aligns text and audio representations to enforce speaker-independent semantic encoding. In the reported formulation, this disentangled representation conditions an autoregressive transformer for high-fidelity conversion without explicit speaker embeddings. The method is positioned within neural codec- and LLM-based voice conversion, where quantized representations often entangle linguistic content, paralinguistic information, and speaker identity; SemAlignVC is presented as a privacy-preserving and generalizable solution for any-to-any conversion (Mehta et al., 11 Jul 2025).

1. Problem formulation and motivation

SemAlignVC addresses zero-shot voice conversion, also described as any-to-any VC: given a source utterance, the system synthesizes speech that preserves the source content while adopting the timbre of an unseen target speaker, without requiring paired source-target utterances. The central difficulty is timbre leakage, namely the persistence of source-speaker traits in the converted output. The paper identifies this as a particular challenge in neural codec-based VC and LLM-based VC, because the discrete or quantized speech units used in such systems can entangle semantic content, paralinguistic content, and speaker timbre (Mehta et al., 11 Jul 2025).

The notation used in the model description decomposes an utterance into semantic and timbre components. For an utterance $X_{a_s,a_t}$, where $a_s$ denotes semantics and $a_t$ denotes timbre, the reconstruction objective is written as

$\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$

For voice conversion, the desired inference combines the semantics of one speaker with the timbre of another: $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$ The paper stresses that the model is not trained on such mixed combinations, which makes disentanglement necessary for generalization.

A second motivation concerns privacy and generalization. The paper notes that approaches relying on explicit speaker embeddings may expose identifiable speaker traits and may generalize poorly in any-to-any settings. SemAlignVC therefore attempts to remove speaker identity from the content pathway rather than representing it directly in a dedicated embedding.

2. Architecture and representational pipeline

SemAlignVC is organized as four individually trained components: an audio tokenizer, a semantic LLM (semantic-LLM), an acoustic model, and a vocoder. The architecture is described as having a training view for the semantic LLM and a separate inference pipeline for VC (Mehta et al., 11 Jul 2025).

Component	Role	Key details
Audio tokenizer	Converts speech to discrete tokens	Pretrained BEST-RQ-based VQ-VAE
Semantic encoder $Q_\phi$	Extracts speaker-independent semantics	Operates on audio tokens
Semantic LLM	Predicts audio tokens from semantics and prosody	LLaMA-style autoregressive transformer
Acoustic model	Converts tokens to mel spectrograms	Conditional flow matching-based transformer
Vocoder	Converts mel to waveform	BigVGAN

The tokenizer maps an utterance to discrete audio tokens: $z_a = H_{Enc}(X_{a_s, a_t}),$ where $H_{Enc}$ is the audio tokenizer encoder and $z_a$ is the discrete token sequence. The paper explicitly emphasizes that $z_a$ still entangles timbre and content. A semantic encoder $a_s$0 is then introduced to extract semantic content from these tokens: $a_s$1 Here, $a_s$2 is the predicted semantic representation, and the encoder is trained to filter out timbre.

Prosody is handled separately. The system extracts pitch $a_s$3 and energy $a_s$4 from the input audio, mean-normalizes them at the utterance level to remove speaker-related information, and feeds them to the LLM so that prosodic and paralinguistic cues are preserved. The semantic LLM models

$a_s$5

and the paper states that ideally $a_s$6, where $a_s$7 denotes speaker-independent tokens containing the semantic and linguistic information of the input.

The acoustic model is described as a conditional flow matching-based transformer that converts audio tokens into mel spectrograms. The paper states that it models the conditional distribution

$a_s$8

The source text explicitly notes a formatting issue in this expression, but the intended role is clear: the model reconstructs masked mel regions conditioned on contextual mel and audio tokens. A BigVGAN vocoder then synthesizes waveform audio from the predicted mel spectrogram.

3. SemAlign: text-audio alignment for speaker-independent semantics

The defining mechanism of SemAlignVC is SemAlign, a semantic alignment objective intended to make the speech content representation speaker-independent. Its basic premise is that text carries semantic and linguistic information but no timbre, whereas audio-derived semantic representations often retain speaker cues. Aligning audio semantics to text semantics is therefore used as an explicit pressure against timbre leakage (Mehta et al., 11 Jul 2025).

The method does not rely on alignment by CTC loss alone. CTC is used as an auxiliary objective to encourage the semantic encoder to retain semantic information, in a manner similar to PPG-like prediction. However, the paper reports that CTC alone did not sufficiently disentangle timbre. The stated hypothesis is that CTC blank tokens can still carry unwanted speaker characteristics when modeled by a strong decoder. This distinction is central to the method’s rationale: SemAlign is presented as stronger than CTC because it ties the audio semantic representation to a text-only semantic space rather than only to token-sequence transduction.

The text-side signal is constructed from the utterance transcript. The transcript is tokenized with the BERT tokenizer and passed through BERT to obtain a text semantic embedding $a_s$9. The source text renders this as

$a_t$0

and explicitly notes a formatting omission in the expression. Because $a_t$1 and $a_t$2 may have different temporal lengths and channels, $a_t$3 is upsampled by repetition to match the temporal dimension of $a_t$4. The paper then minimizes the mean squared error alignment objective $a_t$5 between the aligned text semantics and the predicted audio semantics, using Monotonic Alignment Search (MAS).

A further design choice is gradient restriction from the LLM decoder to the content encoder. Following prior work, the method restricts this gradient propagation to reduce the chance that speaker information leaks into the learned semantic representation. This suggests a deliberate separation between the semantic pathway and the autoregressive decoder pathway: the decoder should consume a clean semantic representation rather than shape it in a way that reintroduces speaker identity.

4. Conditioning, training objectives, and inference procedure

During semantic-LLM training, each utterance is split into a primary input audio and a smaller excerpt comprising about 25% of the utterance. The mel spectrogram of this shorter excerpt is denoted $a_t$6 and serves as the reference timbre. The semantic LLM is trained with the prompt

$a_t$7

The cross-entropy loss $a_t$8 is computed only for $a_t$9 and $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$0. In effect, the LLM is conditioned on the semantic representation $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$1 and a timbre reference $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$2, and is trained to predict the audio token sequence (Mehta et al., 11 Jul 2025).

At inference time, the target-speaker reference $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$3 is fed into both the semantic-LLM and the acoustic model context. The system then produces a converted mel spectrogram $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$4, followed by the converted waveform $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$5 through the vocoder. Although the method does not use explicit speaker embeddings, it still conditions generation on a target-speaker reference mel and on mean-normalized prosody features $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$6 and $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$7. A common misconception would be to equate “without explicit speaker embeddings” with “without target-speaker conditioning”; the architecture does not remove speaker conditioning, but relocates it to a reference-audio pathway.

Three training objectives are explicitly identified. The first is CTC loss, used to encourage semantic retention in $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$8. The second is the SemAlign loss $\hat{X}_{a_s, a_t} = P_\theta(X_{a_s, a_t}).$9, the MSE-based text-audio alignment objective. The third is the language modeling loss $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$0, implemented as cross-entropy on the LLM output. The overall training is described as modular, with components trained separately rather than end-to-end.

The implementation details reported in the paper are specific. The semantic encoder uses 4 layers of Conformer with hidden dimension $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$1, trained with a beta-binomial prior and alignment losses from One TTS Alignment to Rule Them All. The semantic LLM uses the same architecture as LLaMA, with hidden dimension $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$2, 8 layers, and a total size of about 0.5B parameters. Optimization uses AdamW, a peak learning rate of 1e-4, 2000 steps of warmup, and decay after warmup. The acoustic model is identical to VoiceBox, uses a Midpoint ODE solver, and is run for 12 steps with $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$3. The audio tokenizer is a BEST-RQ-based quantizer trained on a large internal dataset.

5. Evaluation protocol and empirical findings

The empirical analysis combines diagnostic experiments on representation quality with end-to-end voice conversion evaluation. A key diagnostic test is a speaker classification experiment designed to measure how much speaker information remains in different representations. The classifier is trained on LibriHeavy for 5 epochs using 8 H100 GPUs, and lower classification accuracy is better if the goal is to remove speaker identity (Mehta et al., 11 Jul 2025).

The reported accuracies are:

• EnCodec: 96.7%
• HuBERT (layer 9): 71.7%
• Ours$$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$4: 82.05%
• WavLM x-vectors: 88.4%
• Ours$$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$5: 2.84%

These results show a large gap between the semantic encoder output $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$6 and the alternative discrete or continuous speech representations. In the paper’s interpretation, the learned semantic encoder nearly eliminates speaker-class information. A companion PCA analysis on $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$7 audio embeddings and text embeddings $$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$8 for two utterances is reported to show strong alignment between the principal components of the two representations.

The subjective evaluation uses the VCTK test set with 10 unseen speakers and 4 sentences per speaker. Because the utterances were short, two sentences were concatenated to form longer inputs, producing 20 input utterances. References were taken from 6 unseen speakers selected randomly from the VCTK test set, with 3 male and 3 female speakers, for a total of 120 audio samples. The metric is SMOS, a speaker-similarity mean opinion score on a 5-point Likert scale. The study includes 120 participants, 10 audio samples per model, and 1,200 ratings per model.

Model	SMOS
KNNVC	$$\hat{X}_{a_s, b_t} = P_\theta(X_{a_s, a_t}, X_{b_s, b_t}).$$9
HierSpeech++	$Q_\phi$0
UniAudio	$Q_\phi$1
SemAlignVC	$Q_\phi$2

The objective evaluation uses the LibriHeavy test set, with 50 utterances from unseen speakers and 10 reference utterances from different unseen speakers, for 50 × 10 samples per model. The metrics are DNSMOS for naturalness, WER via Whisper Large v3, FPC as the Pearson correlation of $Q_\phi$3, and speaker-similarity scores from WavLM, ECAPA-TDNN, and Resemblyzer. The paper notes an input WER of 6.77%.

For naturalness, SemAlignVC achieves SIG 3.63, BAK 4.13, and OVRL 3.38, which the paper reports as the best overall quality score. For consistency, the reported FPC values are 0.632 for KNNVC, 0.622 for SemAlignVC, 0.614 for HierSpeech++, and 0.610 for UniAudio, placing SemAlignVC close to the best result. For intelligibility, WER is 18.85% for KNNVC, 8.24% for HierSpeech++, 9.98% for UniAudio, and 12.31% for SemAlignVC. The method is therefore better than KNNVC on WER, but not as low as HierSpeech++ and UniAudio. For speaker similarity, SemAlignVC is reported as best on all three similarity metrics: WavLM 0.95, ECAPA 0.82, and Resemblyzer 0.89.

6. Interpretation, scope, and limitations

The reported results support a specific interpretation of how SemAlignVC operates. Standard speech representations such as EnCodec tokens, HuBERT layer-9 tokens, WavLM x-vectors, and the paper’s tokenizer outputs retain substantial speaker information, whereas the learned semantic encoder $Q_\phi$4 does not. In that sense, SemAlignVC treats semantic alignment with text as the primary disentanglement mechanism. This suggests that text can function as an anchor for content semantics in codec- and LLM-based VC, rather than merely as a supervision target (Mehta et al., 11 Jul 2025).

The method’s main advantages, as stated or directly supported by the reported experiments, are reduced timbre leakage, strong target-speaker similarity, and improved naturalness without explicit speaker embeddings. The paper also presents the system as privacy-preserving and generalizable to unseen speakers and mismatched source-target combinations. At the same time, these strengths are qualified by several limitations. Intelligibility is not best-in-class, since WER remains higher than HierSpeech++ and UniAudio. Prosodic consistency is close to, but not the highest among, the compared systems. The paper also notes that BERT/text-embedding artifacts may cause synonym-like substitutions, observed in pure TTS experiments and suspected to contribute to errors.

Two structural constraints further define the scope of the method. First, SemAlignVC depends on transcripts, because SemAlign uses text alignment; it is therefore not a text-free system. Second, its components are trained separately, not as an end-to-end unified architecture. These properties matter for interpretation: SemAlignVC is best understood not as a universal disentanglement proof, but as a modular VC system in which text-audio semantic alignment is used to suppress speaker identity in the content representation while target timbre is supplied through a reference-audio pathway.