NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization

Abstract: Advanced neural technologies in speech synthesis and voice conversion (VC) have introduced severe risks to personal privacy, necessitating robust Speaker Anonymization Systems (SAS). Existing SAS approaches modify voice characteristics in the hand-crafted feature space or speaker embedding space, often struggling to provide sufficient identity variance across generated voices. In this paper, we propose NouveauVoice, a novel pseudo-speaker generation framework based on a Hierarchical Deep Variational Autoencoder (NVAE). Integrated as a standalone plug-in module on top of state-of-the-art architectures (FACodec and CosyVoice2), our approach leverages tractable sampling and the Evidence Lower Bound (ELBO) objective to synthesize highly expressive pseudo-speaker embeddings with significantly enhanced speaker diversity. Evaluating our framework under a protocol similar to the VoicePrivacy Challenge alongside Maximum Mean Discrepancy (MMD) analysis, we demonstrate that NouveauVoice achieves strong identity concealment, yielding an Equal Error Rate (EER) exceeding 38% against an automatic speaker verification attacker model. Our system shows a reasonable trade-off between strict anonymity, rich pseudo-speaker diversity, and downstream speech utility, such as intelligibility and emotional expressiveness.

• The paper introduces a hierarchical NVAE approach that generates pseudo-speaker embeddings for robust voice anonymization, enhancing privacy against ASV attacks.
• It integrates with state-of-the-art voice conversion systems, enabling fine-grained control over privacy by selectively randomizing NVAE latent groups.
• Experimental results show improved privacy (EER beyond 38%) and enhanced speaker diversity compared to GMM baselines, with minimal impact on speech intelligibility.

NouveauVoice: A Hierarchical NVAE Framework for Robust Voice Anonymization

Introduction

The proliferation of advanced neural speech synthesis and voice conversion (VC) technologies has escalated concerns regarding speaker privacy, necessitating sophisticated Speaker Anonymization Systems (SAS). Traditionally, SAS methods manipulate features or embeddings using signal processing or deep neural strategies, but they often yield insufficient pseudo-speaker diversity, limiting their efficacy in defending against automatic speaker verification (ASV) attacks. "NouveauVoice: Generating Novel Pseudo Speakers for Voice Anonymization" (2607.03985) introduces a framework leveraging a hierarchical Deep Variational Autoencoder (NVAE) for pseudo-speaker embedding generation, optimizing for both privacy and natural diversity in anonymized speech.

Methodology

Hierarchical NVAE Architecture

NouveauVoice applies the NVAE, originally devised for high-quality image synthesis, to the speaker embedding space. The model's encoder maps input speaker embeddings into a multi-level latent structure, where each layer represents progressively finer speaker identity features. Training utilizes the ELBO objective, jointly maximizing speaker embedding reconstruction fidelity and minimizing divergence between model posterior and prior at every hierarchy level, thus enforcing distributional alignment with the original population.

Modifications for speech include replacing 2D with 1D convolutions and employing quantile normalization, free-bits regularization, and KL warmup to stabilize training and prevent posterior collapse.

Figure 1: NouveauVoice generator overview. Left side shows the encoder structure, while right side depicts the decoder structure. Dashed lines between encoder and decoder blocks denote residual connections.

Integration with SOTA VC Systems

NouveauVoice is designed as an independent, plug-in module, sampled pseudo-speaker embeddings can be injected into existing VC architectures:

• FACodec: Factorized neural speech codec performing prosody, content, and timbre disentanglement, with a dedicated speaker encoder.
• CosyVoice2: Zero-shot text-to-speech/voice conversion pipeline utilizing a CAM++ speaker embedding; NouveauVoice replaces this embedding with synthesized pseudo-speaker vectors.

Progressive Privacy Control

Privacy strength is made tunable by varying the number of replaced NVAE latent groups in the embedding. Randomizing more coarse-to-fine groups yields greater dissimilarity from the original identity and higher anonymization, allowing fine-grained balance between intelligibility, emotional expressiveness, and privacy.

Experimental Setup and Evaluation

Evaluation Protocol

Adopting VoicePrivacy Challenge guidelines, NouveauVoice is evaluated using:

• Privacy: Equal Error Rate (EER, higher is better), estimated by ASV attacks using ECAPA-TDNN trained on VoxCeleb.
• Utility: Word Error Rate (WER, lower is better) on ASR via wav2vec 2.0, and Unweighted Average Recall (UAR, higher is better) on Speech Emotion Recognition (SER).
• Diversity: Top-5 cosine similarity (lower is more diverse) and Maximum Mean Discrepancy (MMD, a measure of embedding distribution divergence).

Comparisons are made against a GMM-based pseudo-speaker generator.

Results

Privacy-Utility Tradeoffs

NouveauVoice, when integrated with either FACodec or CosyVoice2, achieves EER values beyond 38%, indicating strong resistance to ASV-based re-identification. For CosyVoice2, the framework matches GMM baselines in privacy and intelligibility, while slightly outperforming in emotion preservation. For FACodec, NouveauVoice achieves a superior privacy-utility balance compared to GMMs, offering reduced intelligibility degradation and improved emotional expressiveness at a minor cost in EER.

Fine-Grained Anonymization

Progressively replacing NVAE latent groups allows continuous control of anonymization strength. With CosyVoice2, EER increases monotonically up to 36.4% as more groups are randomized, while WER and UAR remain relatively stable, indicating negligible utility loss. FACodec-NV shows that privacy gains saturate early (by replacing two latent groups), but further randomization sharply impacts intelligibility, highlighting model-specific privacy-utility frontiers.

Figure 2: Effect of the number ($N$) of randomized groups of the NVAE latents on privacy (EER$\uparrow$) and utility (WER$\downarrow$ {content} UAR$\uparrow$).

Speaker Embedding Diversity

NouveauVoice-generated pseudo-speaker embeddings show substantially lower top-5 cosine similarity than GMM baselines, especially in deeper CosyVoice2-NV configurations (0.40 vs. 0.75), reflecting enhanced speaker diversity. However, excess diversity can increase MMD, indicating a tradeoff between spanning the embedding space and maintaining close distributional alignment with natural speakers.

Implications and Perspectives

Theoretical Significance

This hierarchical NVAE application to speaker anonymization demonstrates the efficacy of deep generative priors in modeling complex, multi-modal identity distributions, going beyond the capacity of GMMs or shallow models to ensure both privacy and diversity. The explicit decoupling of hierarchical levels enables nuanced ablation analysis and privacy-utility tradeoff control, advancing research in explainable, controllable anonymization pipelines.

Practical Applicability

NouveauVoice functions as a generic module compatible with state-of-the-art VC frameworks, requiring minimal integration overhead. Experimental evidence suggests that when optimally tuned (e.g., selecting a minimal number of latent group replacements), it upholds strong anonymization standards while largely preserving linguistic and paralinguistic fidelity, including emotion—an emerging requirement in SAS (Tomashenko et al., 17 Jan 2026).

Directions for Future Research

Potential enhancements include attribute-guided pseudo-speaker synthesis (e.g., conditioning on age, accent, or gender), dynamic adaptation to attacker models, and exploration of textless or cross-lingual anonymization. Further, integrating GAN or diffusion models with hierarchical VAEs may increase realism or support novel application scenarios in privacy-preserving speech synthesis and data sharing.

Conclusion

NouveauVoice establishes a robust, flexible framework for voice anonymization via hierarchical NVAE-generated pseudo-speaker embeddings. The method sets a new standard for tradeoff-adjustable, privacy-preserving speaker anonymization, substantially enhancing speaker diversity relative to conventional methods while maintaining speech utility and naturalness. Its architecture and analytic flexibility render it highly relevant for both academic research and practical privacy-focused deployments (2607.03985).