Voice Anonymization System

• Voice anonymization systems are signal processing pipelines that transform speech to conceal speaker identity while retaining key linguistic and paralinguistic features for downstream tasks.
• They typically use a modular architecture involving feature extraction, speaker embedding anonymization through methods like pseudo-speaker pooling, GAN-based generation, and waveform synthesis using neural acoustic models.
• Evaluation balances privacy (measured by EER), utility (assessed by ASR WER), and emotion preservation, while addressing challenges such as real-time performance and bias mitigation.

A voice anonymization system is a signal processing or neural transformation pipeline that modifies speech waveforms to conceal speaker identity while preserving the linguistic and paralinguistic content necessary for downstream tasks such as automatic speech recognition (ASR) and emotion recognition. The core objective is to render the speech “unlinkable” to its original source according to the privacy definition used in the VoicePrivacy Challenge, preventing any practical attacker—including those retraining speaker verification systems on anonymized data (semi-informed attackers)—from reliably associating an anonymized utterance with a specific speaker, while maintaining the utility of the transformed audio for ASR or other desired functions (Tomashenko et al., 2020, Tomashenko et al., 17 Jan 2026, Tomashenko et al., 2024, Kuzmin et al., 20 Jan 2026).

1. System Architectures and Canonical Pipelines

Contemporary voice anonymization systems share a modular, three-stage pipeline: (1) feature extraction, (2) privacy-preserving transformation of speaker-dependent features, and (3) waveform synthesis. In practical terms:

• Feature Extraction: Given a raw waveform $x(t)$, extract a high-dimensional speaker embedding (e.g., x-vector, ECAPA-TDNN, or neural codec embedding), linguistic/semantic content features (ASR bottleneck, phoneme sequence, semantic tokens), and prosodic features (F0, energy, duration) (Nespoli et al., 2023, Tomashenko et al., 2024, Kuzmin et al., 20 Jan 2026).
• Anonymization Transformation: Apply deterministic or stochastic replacements of speaker-identity representation with a pseudo-speaker embedding, random vector, or GAN-sampled token. Methods include x-vector averaging from pools of distant speakers, convex mixing with random noise, GAN-sampled style embeddings, or neural codec disentanglement (Tomashenko et al., 2020, Tomashenko et al., 17 Jan 2026, Yao et al., 2024, Deng et al., 2022).
• Synthesis: Use a neural acoustic model (e.g., FastSpeech2, neural source-filter networks, HiFi-GAN, EnCodec, Firefly-GAN) to reconstruct time-domain speech from the anonymized identity token together with preserved content and prosody (Yao et al., 2024, Kuzmin et al., 20 Jan 2026, Yao et al., 2022).

Canonical systems, such as the VPC B1 baseline, follow this pattern: (a) extract ASR bottleneck features and x-vectors, (b) anonymize x-vectors by averaging random subsets of distant pool vectors, and (c) synthesize speech with an NSF vocoder (Tomashenko et al., 2020, Tomashenko et al., 2021). Advanced pipelines combine multiple layers of disentanglement, as in the NPU-NTU VPC 2024 system, using serial VQ-codecs to separate speaker identity, linguistic content, and emotional prosody, then replace the speaker factors via weighted pooling and Gaussian mixing (Yao et al., 2024, Tomashenko et al., 17 Jan 2026).

2. Anonymization Strategies and Algorithms

Multiple strategies have been established for speaker de-identification:

• Pseudo-Speaker Pooling and Averaging: Given an original embedding $s$, select $N$ farthest embeddings from a large public pool and average a random subset to obtain $s_\textrm{anon}$ (Tomashenko et al., 2020, Tomashenko et al., 2024).
• Convex Combination with Random Noise: Mix the average pool embedding with a draw from a standard Gaussian: $$s_\textrm{anon} = \alpha \bar{s} + (1-\alpha)\hat{s}$$, controlling the privacy/utility trade-off via $\alpha$ (Yao et al., 2024, Tomashenko et al., 17 Jan 2026, Kuzmin et al., 20 Jan 2026).
• GAN-based Embedding Generation: Employ a GAN to generate embeddings restricted to have low cosine similarity with the original. Accepted only if $d_\cos(s, s_\textrm{fake}) > \epsilon$ (Zhu et al., 2023, Tomashenko et al., 2024).
• Codebook or Quantization-based Anonymization: Leverage neural codecs (e.g., NaturalSpeech 3 FACodec) to disentangle speaker, content, and prosodic information, and replace only the speaker tokens (Yao et al., 2024, Kuzmin et al., 20 Jan 2026).
• Signal Processing Approaches: Direct transformations like McAdams formant warping or VTLN/pitch shift, which alter the spectral envelope/pitch to obscure timbre (Yao et al., 2022, Zhu et al., 2023, Tomashenko et al., 2024).
• Text-based Re-synthesis: Recognize and transcribe content to phones/words, then re-synthesize with a TTS system driven by a private or random speaker embedding (Turner et al., 2022, Meyer et al., 2022).

Innovations such as neural codec disentanglement, serial VQ layering, and explicit multi-level distillation for emotion and content have enabled robust anonymization with high utility in recent VPCs (Yao et al., 2024, Tomashenko et al., 17 Jan 2026).

3. Evaluation Methodologies and Metrics

Evaluation is defined by the VoicePrivacy protocol (Tomashenko et al., 2024, Tomashenko et al., 17 Jan 2026):

• Privacy: Measured by Equal Error Rate (EER) of an ASV system under lazy-informed and (critically) semi-informed attacks, i.e., after retraining ASV models on anonymized data. Higher EER indicates stronger anonymization.
• Utility: Assessed by ASR word error rate (WER) and, since VPC 2024, emotion preservation via unweighted average recall (UAR) from pretrained emotion recognizers.
• Bias/Fairness: Performance stratified by sex and dialect subgroup (Bias$_g$ = EER$_g$ / EER$_\textrm{all}$), to reveal systematic disparities (Leschanowsky et al., 2023).
• Subjective Evaluation: Naturalness, intelligibility, and perceived speaker similarity rated by human listeners (Tomashenko et al., 2021).
• Trade-off Frameworks: Privacy-utility surface (PU$_\textrm{tr}$ curves) characterize operational regimes (Nespoli et al., 2023).

The most robust systems (e.g., NPU-NTU, T10-2) achieve EER $>$ 40% under semi-informed attack, with WER $<$ 3.5% and UAR $>$ 60% (Yao et al., 2024, Tomashenko et al., 17 Jan 2026).

4. Limitations, Attacks, and Open Vulnerabilities

Despite strong scores in standard metrics, voice anonymization systems face structural limitations:

• Linear Invertibility: Attacks using Procrustes or Wasserstein-Procrustes alignment can recover up to $\sim 60\%$ of speaker identities in embedding space if the anonymization is a global orthogonal transform (Champion et al., 2021).
• Paralinguistic Leakage: Prosody (F0), residuals in bottleneck or PPG features, and emotion-related cues may insufficiently disentangle from speaker identity, leaking PPI (personally predictable information) (Gaznepoglu et al., 2022, Nespoli et al., 2023, Zhu et al., 2023).
• Text-Linguistic Stylometry: Advanced multimodal attacks (e.g. VoxATtack) exploit textual and semantic content, fusing BERT-encoded text with ECAPA-TDNN embeddings to achieve EERs well below 30% on strong anonymizers (Aloradi et al., 16 Jul 2025).
• Fairness: Sex- and dialect-based bias persists, with subgroup fairness degrading under stronger attacker models (Leschanowsky et al., 2023, Tomashenko et al., 17 Jan 2026).
• Non-invertibility: Many pipelines do not achieve formal non-invertibility, leaving recovery attacks possible, especially if the adversary can accumulate public side-channel examples or access the anonymization algorithm (Champion et al., 2021, Turner et al., 2022).
• Utility-Emotion Trade-off: Naive ASR-TTS cascades obliterate emotion (UAR $\sim$ 30%), while codec-based systems maintain content and (partially) paralinguistics (Tomashenko et al., 17 Jan 2026).

5. Real-Time Systems, Implementation, and Practical Considerations

The requirement for real-time, streaming-capable anonymization is being addressed in designs such as Stream-Voice-Anon (Kuzmin et al., 20 Jan 2026):

• Low Latency: Architectures achieve $\sim$180 ms one-way latency with chunked codec processing and dual-stage AR transformers.
• Disentanglement: Quantized content codes with low $I(c; s)$ ensure speaker-independence.
• Embedding Mixing: Dynamic prompt-based sampling effectively increases unpredictability while balancing naturalness.
• Resource Constraints: Full real-time performance typically requires GPUs, due to $\sim$200M parameter ARVC models; CPU-only feasibility is an open issue (Kuzmin et al., 20 Jan 2026, Deng et al., 2022).
• Dynamic–Fixed Delay Configuration: Latency–privacy–utility trade-offs are governed by frame lookahead and prompt sampling, but EER is largely constant for L $\gtrsim$130 ms (Kuzmin et al., 20 Jan 2026).
• Edge Integration: SOTA pipelines (e.g., HiFiGAN and EnCodec) can operate in chunked, streaming mode for on-device deployment (Kuzmin et al., 20 Jan 2026).

6. Applications, Extensions, and Future Directions

Voice anonymization is critical for:

• Data Sharing, Cloud Services, and Compliance: Ensuring user privacy in voice assistants, digital healthcare, and cloud transcription (Tomashenko et al., 2020).
• Speech Diagnostics: For affective computing and disease detection (e.g., COVID-19 diagnostics), anonymizers should preserve paralinguistic features; GAN-based embedding replacement greatly degrades diagnostic utility (Zhu et al., 2023).
• User-side Privacy Middleware: Systems like AltVoice convert speech to text, then re-synthesize it with secret-based embeddings, enabling revocable/unlinkable privacy for user-chosen identities (Turner et al., 2022).
• Personalized Re-Identification/Forensics: Future frameworks may require reversible anonymization (e.g., SpeechGuard) for lawful interception, or support for multi-attribute anonymization (paralinguistics, background sound, etc.) (Tomashenko et al., 17 Jan 2026).
• Fairness and Bias Mitigation: Emerging benchmarks and subgroup analysis required to ensure equitable privacy across sex, ethnicity, and dialect (Leschanowsky et al., 2023, Tomashenko et al., 17 Jan 2026).
• Adaptive/Prompted Anonymization: On-demand parameterization, user-provided prompts, and dynamic privacy-utility scheduling.

Research continues on non-linear, adversarially robust transformations, emotion/attribute disentanglement, CPU-efficient models, and privacy metrics aligned with regulatory requirements (e.g., GDPR's “singling-out” risk) (Kuzmin et al., 20 Jan 2026, Yao et al., 2024, Tomashenko et al., 2024).

7. Summary Table: Principal Techniques and Performance

System/Method	Privacy (EER)	Utility (WER)	Emotion (UAR)	Key Strategy
VPC B1 Baseline (Tomashenko et al., 2024)	9–16%	2.9–3.1%	42.7%	x-vector averaging
NPU-NTU VPC 2024 (Yao et al., 2024)	40–42%	2.5–3.5%	61–66%	Codec, serial disentanglement
Stream-Voice-Anon (Kuzmin et al., 20 Jan 2026)	46–48% (lazy)	4.7–6.6%	40–44%	Codec+prompt mixing
V-Cloak (Deng et al., 2022)	42.6–46.1%	7.65%	N/A	Real-time, waveform perturb.
Two-Stage (B1a+ZS-VC) (Nespoli et al., 2023)	43%	12.7%	N/A	Cascaded anonymization + zero-shot VC
GAN-Based (Ling-GAN) (Zhu et al., 2023)	$\sim$>40%	$>$10%	N/A	WGAN embedding
AltVoice (Turner et al., 2022)	$\sim$0.1 (re-id)	22–29%	N/A	Text-based synth., secret embeddings

*Privacy: equal error rate under (semi-)informed attack; Utility: ASR word error rate; Emotion: unweighted average recall. All values as reported in the cited works.

---

References:

• (Tomashenko et al., 2020) Introducing the VoicePrivacy Initiative
• (Tomashenko et al., 2024) The VoicePrivacy 2024 Challenge Evaluation Plan
• (Tomashenko et al., 17 Jan 2026) The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization
• (Kuzmin et al., 20 Jan 2026) Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and LLMs
• (Yao et al., 2024) NPU-NTU System for Voice Privacy 2024 Challenge
• (Leschanowsky et al., 2023) Voice Anonymization for All -- Bias Evaluation of the Voice Privacy Challenge Baseline System
• (Champion et al., 2021) On the invertibility of a voice privacy system using embedding alignment
• (Aloradi et al., 16 Jul 2025) VoxATtack: A Multimodal Attack on Voice Anonymization Systems
• (Zhu et al., 2023) On the Impact of Voice Anonymization on Speech Diagnostic Applications: a Case Study on COVID-19 Detection
• (Deng et al., 2022) V-Cloak: Intelligibility-, Naturalness- & Timbre-Preserving Real-Time Voice Anonymization
• (Meyer et al., 2022) Speaker Anonymization with Phonetic Intermediate Representations
• (Turner et al., 2022) I'm Hearing (Different) Voices: Anonymous Voices to Protect User Privacy
• (Gaznepoglu et al., 2022) VoicePrivacy 2022 System Description: Speaker Anonymization with Feature-matched F0 Trajectories
• (Nespoli et al., 2023) Two-Stage Voice Anonymization for Enhanced Privacy
• (Yao et al., 2022) NWPU-ASLP System for the VoicePrivacy 2022 Challenge