Self Voice Conversion Overview

• Self voice conversion is a technique that leverages voice conversion pipelines with self-supervised features to remap speech while preserving linguistic content and speaker identity.
• It enables rigorous evaluation of conversion fidelity and privacy by altering prosodic and micro-acoustic details while maintaining intelligibility.
• Applications include watermark removal, speaker anonymization, and examining the disentanglement of content, speaker, and prosody using encoder–decoder architectures.

Self voice conversion is a special case of voice conversion (VC) in which the input (source) and output (target) speaker identities are intentionally set to be identical. The principal goal is to remap a speech signal through a VC pipeline such that linguistic content and speaker identity are preserved, while acoustic characteristics (such as prosody, fine-timing, and low-level spectral details) are modulated or reconstructed. Self voice conversion is widely studied both as a means of assessing conversion system fidelity and as a vector for privacy/anonymization or adversarial attacks—most notably, attacks on neural audio watermarking. Recent advances leverage self-supervised learning (SSL) for extracting content and speaker representations, enabling zero-shot, any-to-any conversion with minimal supervision or prior knowledge.

1. Self Voice Conversion: Definition, Motivations, and Core Objectives

Self voice conversion is defined as the application of a VC system to a speech utterance ( x ) such that both the source and target speakers are the same (( A \to A )), i.e., the model is explicitly conditioned on the original signal’s speaker characteristics. The intended outputs ( \hat{x} ) must match ( x ) in linguistic content, speaker identity, and perceived quality—but need not be a faithful, frame-by-frame replica. The principal motivations for this task are:

• Fidelity assessment: Evaluating whether a conversion model preserves speaker and content information or introduces artifacts/identity leakage.
• Adversarial obfuscation: Destroying low-level acoustic details (e.g., digital watermarks) while keeping audible properties unaltered for downstream speech and speaker recognition.
• Privacy/anonymization: By altering prosodic or micro-acoustic features, self voice conversion allows for anonymization while maintaining intelligibility [2601.20432].
• Analytical probe: Testing the representational clarity and disentanglement capacity of VC models—i.e., whether SSL or factorized features are truly speaker-independent or linearly separable.

Self voice conversion has become a standard attack model and evaluation tool in the context of neural audio watermarking and privacy-critical speech systems [2601.20432].

2. Model Architectures and Training Objectives

Self voice conversion pipelines universally adhere to the encoder–decoder paradigm, often instantiated with self-supervised representation backbones. The typical modules and their mathematical formulations include:

• Content Encoder ( E_c ): Extracts phonetic/linguistic features from ( x ), ideally discarding speaker/affective cues. Examples include Conformer-SSL [2302.08137], HuBERT/WavLM [2505.08278], Wav2Vec 2.0, or discrete VQ-VAE tokenizers [2502.04519].
• Speaker Encoder ( E_s ): Extracts a fixed-dimensional embedding ( s = E_s(x) ), capturing identity-related properties [2601.20432].
• Prosody/Pitch Extractors: Many models explicitly extract and normalize pitch contours ( p ) (e.g., via CREPE or PYin), sometimes learning prosodic representations [2110.14422].
• Decoder ( D ): Generates a mel-spectrogram or waveform from (potentially disentangled) ( c, s, p ). Decoder may be a FastPitch-style feedforward/transducer, a Transformer LM [2502.04519], or a non-autoregressive diffusion-based network [2505.16691].
• Neural Vocoder ( V ): HiFi-GAN, BigVGAN, MelGAN, or PWGAN invert the acoustic features to time-domain audio [2302.08137, 2505.08278, 2505.16691].

The general forward process in self-VC can be summarized:
1. ( c = E_c(x) )
2. ( s = E_s(x) ) (for self-VC, the same utterance)
3. ( p = F(x) )
4. ( \hat{y} = D(c, s, p) )
5. ( \hat{x} = V(\hat{y}) )

Training losses are reconstructions of spectral or waveform features (( | \hat{y} - y |_1 )), regularization or adversarial terms to promote disentanglement (e.g., contrastive/siamese losses on pitch-shifted audio), and sometimes cycle or identity losses [2302.08137, 2202.10976, 2112.04424]. The specific architecture and loss weighting are informed by the need to simultaneously preserve intelligibility, speaker similarity, and prosodic fidelity.

3. Disentanglement of Content, Speaker, and Prosody

Modern self-VC frameworks focus extensively on the explicit or implicit disentanglement of content from speaker and prosodic cues.

• Explicit Disentanglement: ACE-VC, for instance, adopts a multi-task model with a content classification (CTC) head and a speaker verification (SV) head, with the content dimension further regularized by a cosine similarity loss applied to original and pitch-shifted versions (( L_{\text{disentangle}} = 1 - \cos(z_c, z_c') )) [2302.08137].
• Adversarial and Cycle Constraints: Cycle reconstruction and "same" losses, as used in DRVC (( L_{\text{cycle}} ), ( L_{\text{same}} )), force invariance of content codes and style transferability in content/timbre spaces [2202.10976]. Speaker-domain adversarial losses encourage the speaker code to contain only identity.
• Prosody Factorization: Systems like [2110.14422] employ self-supervised prosody encoders to extract orthogonal pitch and volume representations, learned by pairwise ranking and discouraged from leaking information across factors.
• SSL Feature Geometry: LinearVC demonstrates that content and speaker characteristics are embedded in largely orthogonal linear subspaces within SSL feature space—simple linear or even rotational transformations suffice for VC, and SVD factorization isolates a low-rank "content" subspace [2506.01510].

Disentanglement is key to robust self-VC, especially in scenarios demanding fine-grained control over output prosody, cross-lingual transfer, or resistance to adversarial transfer.

4. Evaluation Benchmarks and Quantitative Outcomes

Objective and subjective evaluations of self-VC prioritize:

• Speaker Similarity: Measured by verification Equal Error Rate (SV-EER), cosine/Resemblyzer similarity, or human-annotated MOS scores [2302.08137, 2601.20432].
• Intelligibility: Character or word error rate (CER/WER) using high-performance ASR backends (QuartzNet, Whisper-L) [2505.08278, 2310.09653].
• Naturalness: Mean Opinion Score (MOS/Sim-MOS/NMOS) as rated by humans or predicted by models (UTMOS, MOSNet) [2302.08137, 2505.16691].
• Prosody Matching: F0 correlation or pitch/volume KL divergence when evaluating prosody transfer [2505.08278, 2110.14422].
• Watermark attack efficacy: In watermarking attack scenarios, bitwise-extraction accuracy is the primary metric, with degradation to random guess (0.5) indicating attack success [2601.20432].

Representative quantitative results include:
- ACE-VC achieves SV-EER of 5.5% (seen speakers), 8.4% (unseen), and MOS 3.62–3.75 [2302.08137].
- SelfVC attains SV-EER of 3.4% (vs. 6–7% baselines) and human MOS 4.06 (vs. 3.49–3.77 for prior systems) [2310.09653].
- LinearVC achieves WER 4.9%, CER 2.6%, and EER 33.6% (speaker similarity) with a simple linear map [2506.01510].
- In watermark attacks, self-VC reduces extraction accuracy from nearly perfect to chance-level for all major watermarking schemes, while maintaining speaker similarity (0.857/0.748, kNN-VC/RVC) and low WER (0.115/0.120) [2601.20432].

5. Applications: Privacy, Security, and Analytical Probes

Self voice conversion plays a critical role in several high-stakes applications:

• Watermark Removal Attacks: Self-VC universally defeats contemporary neural watermarking systems by discarding micro-structure not aligned with phonetic and speaker latents [2601.20432]. Watermarks relying on imperceptible high-frequency or phase perturbations are not preserved by latent-based resynthesis.
• Speaker Anonymization: GenVC demonstrates that autoregressive variation in prosody and timing enables privacy gains (EER ≈ 29%) while preserving content intelligibility (WER 6.7%) [2502.04519].
• Quality Benchmarks: Self-VC is the gold-standard for upper-bound voice conversion fidelity, as it exposes any loss of information or low-level artifacts in the conversion pipeline [2302.08137, 2112.04424].
• Geometry of SSL Feature Spaces: The ability to map self-voice conversion via global linear or near-orthogonal transformations reveals the algebraic structure of SSL spaces and provides a minimally invasive model for representation probing [2506.01510].

6. Limitations and Open Challenges

While the current state-of-the-art in self voice conversion using SSL features is robust, several limitations persist:

• Prosody and Expressivity: Fine-grained control over expressive features (emotion, emphasis) and robust prosody disentanglement remain challenging [2302.08137, 2505.08278].
• Cross-lingual Generalization: Most VC systems are trained/benchmarked in English; extending self-VC to non-parallel, cross-lingual or code-switched data is an open research direction [2412.08312, 2505.08278, 2505.16691].
• Encoder Dependency: Model generalization is sensitive to the SSL encoder’s language and domain coverage; monolingual or limited encoders degrade performance when facing unseen accents/languages [2505.16691].
• Inference Cost and Complexity: Architectures employing deep diffusion transform, large transformers, or massive SSL backbones (e.g., WavLM-Large) are computationally intensive [2505.16691].
• Adversarial weight tuning: Over-regularization in adversarial losses can either let in speaker leakage or degrade content/prosody [2505.08278].
• Watermarking countermeasures: There is no known, fully effective counter against latent-based self-VC; hybrid watermark detection, latent-aware watermark embedding, or joint-adversarial training remain research frontiers [2601.20432].

7. Summary Table: Core Self Voice Conversion Models

Model	Disentanglement	Main Loss Functions	Notable Metric/Results
ACE-VC	Multi-task + siamese	CTC, SV, cosine sim	SV-EER 5.5%, MOS 3.7
DRVC	Cycle + same + domain adv	Cycle, same, domain, GAN	MOS 3.32, MCD 7.39
GenVC	Self-supervised, no ext. supervision	VQ-VAE, token loglik, adversarial	SV-sim 0.88, Privacy EER 28%
LinearVC	Global linear or SVD	OLS/F-norm regression	EER 33.6%, WER 4.9%
EZ-VC	Flow-matching diffusion	CFM regression	SSim 0.71, NMOS 3.91
SelfVC	Iterative self-syn.	L2 synth. reconstr.	SV-EER 3.4%, MOS 4.06
[2601.20432]	kNN-VC, RVC	Variational, cosine sim	Speaker sim. 0.75–0.86, WER ~0.12

References

• ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations [2302.08137]
• DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning [2202.10976]
• SelfVC: Voice Conversion With Iterative Refinement using Self Transformations [2310.09653]
• GenVC: Self-Supervised Zero-Shot Voice Conversion [2502.04519]
• LinearVC: Linear transformations of self-supervised features through the lens of voice conversion [2506.01510]
• EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion [2505.16691]
• Self Voice Conversion as an Attack against Neural Audio Watermarking [2601.20432]
• Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features [2112.04424]
• Investigating self-supervised features for expressive, multilingual voice conversion [2505.08278]
• Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning [2110.14422]
• Self-Supervised Representations for Singing Voice Conversion [2303.12197]
• A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction [2412.08312]