- The paper introduces a novel SA-based feature perturbation technique that minimizes timbre leakage while preserving linguistic and prosodic features in streaming VC.
- The methodology employs a strictly causal encoder and a HiFi-GAN-inspired decoder with adversarial training to achieve a latency of 20 ms per frame.
- Experimental results show improved SS-s and SS-r metrics and a Real-Time Factor of 0.063, setting new benchmarks for real-time, privacy-preserving voice conversion.
Zero-VC: Strictly Causal, Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Zero-shot voice conversion (VC) aims to convert speech from a source speaker to a previously unseen target speaker using limited reference material. Real-time, streaming zero-shot VC accentuates the trade-off between speaker timbre leakage and utility preservation (prosody, intelligibility), made challenging by the latency constraints of live applications. Previous streaming VC methods generally fall into two paradigms: information bottleneck (IB) mechanisms and speaker perturbation. IB-based approaches excel at removing timbre but degrade prosody, often necessitating explicit prosodic feature injection (e.g., f0) and architectural lookahead, inflating latency. Perturbation-based schemes avoid lossy bottlenecks but have not explicitly optimized the leakage-utility trade-off, resulting in sub-optimal speaker adaptation and conversion fidelity.
The paper identifies this overlooked leakage-utility trade-off as critical for minimizing latency and maximizing conversion quality. It recognizes that the objectives of speaker anonymization (SA)—concealing speaker identity while strictly preserving linguistic and prosodic utility—align with the requirements for robust VC feature representations. This insight motivates the integration of SA as a high-fidelity perturbation mechanism, thus obviating the need for future context and enabling a strictly causal, zero-lookahead architecture.
Methodology
SA-Based Feature Disentanglement
Zero-VC leverages an off-the-shelf SA module to perturb the source audio, mapping speech to a pseudo-speaker space while retaining temporal alignment, prosodic contours, and phonetic integrity. This yields intermediate representations that are highly robust to timbre leakage and remain prosodically expressive. These anonymized features are fed to a streaming encoder operating at 20 ms frame shifts with no future context requirements.
Timbre Conditioning
For target timbre encoding, Zero-VC employs WavLM-large, extracting frame-level hidden states from reference speech and aggregating them via an attention-based learnable pooling mechanism. This strategy generates a global speaker embedding, which conditions the generation process in a manner reminiscent of state-of-the-art VC architectures.
Causal Generation Architecture
The synthesis decoder is adapted from HiFi-GAN, with all convolutions replaced by strictly causal convolutions. The global timbre embedding is injected via convolutional conditioning layers. The training procedure employs adversarial objectives via multi-scale and multi-period discriminators alongside Mel spectrogram and feature-matching losses, ensuring high perceptual quality and target adaptation. During inference, only past frames are cached for convolutional layers, guaranteeing constant per-frame computational complexity and minimal algorithmic latency.
Experimental Analysis
Ablation and Objective Study
A rigorous ablation demonstrates that SA as a perturbation module achieves the lowest source similarity (SS-S 0.119), outperforming both LSCodec-Perturb (0.704) and Seed-VC-Perturb (0.411). It also maintains robust prosody preservation (FPC 0.718). Training identical VC models with each perturbation further illustrates that SA yields a model closest to the ideal region in the speaker similarity space, achieving both minimal source leakage and maximal reference similarity. While LSCodec-Perturb preserves prosody strongly, its conversion often reconstructs the source without effective timbre alteration, invalidating utility metrics.
Lookahead Dependency and Latency Benchmarking
Zero-VC models trained with SA saturate performance with virtually zero lookahead, evidenced by negligible (≤3%) improvement when context is extended up to 80 ms, unlike baseline models that require 40–60 ms for stabilization. This architectural advantage is reflected in algorithmic latency, where Zero-VC operates at a theoretical minimum of 20 ms per frame, outperforming DualVC3 (40 ms), StreamVC (60 ms), and RT-VC (60 ms).
Comprehensive Evaluation
Zero-VC achieves an SS-S of 0.171 and SS-R of 0.521, setting new benchmarks for minimal timbre leakage and maximal target similarity among both streaming and non-streaming systems. Subjective metrics show SMOS of 3.88±0.05 and NMOS of 3.81±0.07. Notably, the system achieves a Real-Time Factor (RTF) of 0.063 on commodity hardware, comfortably exceeding real-time requirements.
Implications and Future Directions
The implications for both theory and practice are significant. Integrating an SA module resolves the major trade-off in feature disentanglement, allowing strict causality with no algorithmic lookahead—a performance floor critical for real-time applications. The methodology opens avenues for joint, end-to-end training that incorporates SA directly, potentially reducing preprocessing overhead and further reducing total system latency. Additionally, future work can extend this paradigm toward multilingual or cross-lingual zero-shot VC.
From a broader perspective, optimizing the balance between utility preservation and identity leakage in streaming VC can enable new privacy-preserving telephony, real-time translation, and cross-modal generative speech systems with minimal latency constraints. The adoption of SA for speaker perturbation advances the state of the art by combining low latency and high conversion fidelity in strictly causal frameworks.
Conclusion
Zero-VC establishes a new milestone in streaming zero-shot voice conversion by introducing SA as a robust speaker perturbation mechanism. The strictly causal, zero-lookahead architecture eliminates future context dependency, thus minimizing algorithmic latency to 20 ms while achieving superior conversion quality and target speaker adaptation. Objective and subjective evaluations confirm that SA dramatically outperforms prior perturbation techniques in leakage suppression and prosody preservation. The approach has broad implications for high-fidelity, privacy-aware, real-time voice manipulation, and future research can further refine end-to-end integration and generalize to multilingual settings.
