MaskVCT: Zero-Shot Multi-Factor Voice Conversion

• The paper introduces a masked generative Transformer model that achieves zero-shot voice conversion by integrating speaker, linguistic, and prosodic conditions within a unified framework.
• It employs a joint classifier-free guidance mechanism to dynamically balance accent conversion, intelligibility, and speaker similarity through adjustable weighting of conditioning signals.
• Experimental results show competitive metrics including high speaker similarity (SS-MOS ≈ 3.69) and effective prosody tracking, demonstrating its practical advantages over traditional VC systems.

MaskVCT refers to a generative Transformer model for zero-shot voice conversion, designed to enable multi-factor controllability through joint classifier-free guidance (CFG) over speaker identity, linguistic content, and prosodic features. Departing from prior VC systems that rely on fixed conditioning pipelines, MaskVCT incorporates multiple types of conditioning signals in a unified masked generative framework, allowing robust, adjustable conversion of source speech to a desired target speaker—optionally with accent and prosody manipulation—without any speaker-specific fine-tuning (Lee et al., 21 Sep 2025).

1. Model Architecture and Conditioning Scheme

MaskVCT operates on discrete acoustic tokens produced by a residual vector quantization (RVQ) neural codec. The architecture comprises a Transformer encoder of 16 PreLN layers (16 heads, 1024-dim hidden size, 4096-dim FFN) utilizing rotary positional embeddings. Speech tokens are augmented by three principal conditioning sources:

• Continuous linguistic embeddings (for enhanced intelligibility)
• Quantized syllabic tokens (from SylBoost, promoting timbre/identity retention and minimizing pitch leakage through the linguistic channel)
• Pitch embeddings, encoded with log-scale sinusoidal functions for prosody control
• Speaker prompt embedding: a 3-second target utterance is encoded, providing explicit speaker identity guidance.

The input token stream is masked according to a binary mask $M_{u,c}$ applied over temporal and codebook axes. Reconstruction is cast as a classification problem over the masked tokens only:

$$\mathcal{L}_{\text{mask}} = \mathbb{E}_{t | M_{u,c}[t,c]=0} \left[ -\log p_\theta(A_0[t,c] | A_{u,c}, C) \right],$$

where $A_0$ denotes the original acoustic tokens, $A_{u,c} = A_0 \odot M_{u,c}$ is the masked input, and $C$ is the set of conditions.

All conditioning signals are merged pre-Transformer via column-wise vector addition, maintaining compatibility with PreLN architectures.

2. Joint Classifier-Free Guidance (CFG) Mechanism

MaskVCT extends the CFG concept from text-to-image synthesis to multi-condition voice conversion. During training, various conditioning combinations are sampled (full, speaker-only, linguistic-only, null). Inference is performed with a triple-guidance logit interpolation scheme: $$\begin{aligned} \log \tilde{p}_\theta(A_n | A_{n+1}, C) =\; &\log p_\theta(A_n | A_{n+1}, L) \ &+ \omega_{\text{all}} \left[ \log p_\theta(A_n | A_{n+1}, A_p, L, P) - \log p_\theta(A_n | A_{n+1}, L) \right] \ &+ \omega_{\text{spk}} \left[ \log p_\theta(A_n | A_{n+1}, A_p, L, \varnothing) - \log p_\theta(A_n | A_{n+1}, L) \right] \ &+ \omega_{\text{ling}} \left[ \log p_\theta(A_n | A_{n+1}, L) - \log p_\theta(A_n | A_{n+1}, \varnothing) \right]. \end{aligned}$$ Here, $$\omega_{\text{all}}, \omega_{\text{spk}}, \omega_{\text{ling}}$$ are user-adjustable weights controlling the influence of pitch, speaker, and linguistic factors, respectively. This scheme enables dynamic navigation of the conversion trade-off: strong accent/speaker matching, intelligibility, and prosody can be independently tuned per utterance.

3. Conditioning Feature Encodings

MaskVCT leverages parallel paths for linguistic information and explicit pitch/prosody control:

• Continuous linguistic features are extracted from a self-supervised speech model (e.g., HuBERT), promoting accurate phonetic content and intelligibility.
• Quantized syllabic tokens (SylBoost): protect target timbre and accent, suppress pitch leakage in the linguistic channel, and enhance speaker similarity.
• Pitch embeddings utilize a sinusoidal code with log-frequency normalization:

$$P(f)_i = \begin{cases} \sin\left(\frac{\log(1 + f)}{10000^{2 i/d}}\right), & i < d/2 \ \cos\left(\frac{\log(1 + f)}{10000^{2(i-d/2)/d}}\right), & i \geq d/2 \end{cases}$$

for $i = 0, \ldots, d-1$ and $d$ is the embedding size. This representation is extractor-agnostic regarding pitch resolution.

The speaker prompt embedding is generated by encoding a brief (e.g., 3s) target reference utterance, forming the speaker identity component.

4. Experimental Outcomes and Evaluation

MaskVCT was benchmarked against contemporary VC models. Key findings:

• Subjective metrics: MaskVCT-Spk yields the highest speaker similarity (SS-MOS ≈ 3.69), competitive accent MOS, and strong Q-MOS and UTMOS (naturalness/quality).
• Objective metrics: Word Error Rate (WER) and Character Error Rate (CER) for MaskVCT variants remain competitive with state-of-the-art intelligibility-focused models like FACodec.
• Prosody tracking: The All-conditioning mode achieves the highest F0 Pearson correlation (FPC), closely reproducing source pitch; in contrast, omitting pitch conditioning (Spk mode) prioritizes speaker timbre over prosody.
• Audio demos: Samples at https://maskvct.github.io/ illustrate trade-offs across CFG settings, with accent conversion and speaker identity matching verified through qualitative analysis and reported MOS scores.

A plausible implication is that MaskVCT simultaneously advances target speaker similarity and accent control while offering flexible intelligibility by balancing CFG weights per task requirements.

5. Practical Implications and Applications

MaskVCT’s architecture—with zero-shot conversion, multi-factor CFG, and dual-conditioning paths—supports real-world deployment in several domains:

• Entertainment/dubbing: Flexible accent/style conversion for film/game audio without speaker-specific pre-training.
• Telecom and assistants: Customization of digital identities with dynamic speaker/accent switching.
• Assistive technology: Personalized voice restoration from brief target prompts.
• Multilingual VC and TTS: Accent, timbre, and prosody control with no external per-speaker adaptation.

Its zero-shot design, i.e., lack of required fine-tuning for each target speaker, facilitates scalability for large catalogues and rapid deployment in personalized or privacy-preserving scenarios.

6. Limitations and Future Research Directions

While MaskVCT attains state-of-the-art speaker and accent similarity with good intelligibility, the trade-off between pitch tracking, intelligibility, and timbre remains a core research challenge. The CFG formulation represents a flexible solution for dynamic adjustment but also demands careful calibration to suit differing application contexts.

Future research directions include:

• Automated CFG weight selection, possibly via learned heuristics or reinforcement learning.
• Integration of additional linguistic or semantic control signals.
• Further improvements in accent conversion and cross-lingual generalization.
• Exploration of larger-scale speaker prompt embeddings or fine-grained style/expressivity manipulation.
• Robustness analysis under noisy/reverberant source inputs, and adversarial conditioning scenarios.

MaskVCT sets a precedent for multi-condition masked VC architectures, opening avenues for highly controllable, zero-shot, and adaptable voice conversion across numerous application verticals (Lee et al., 21 Sep 2025).