VoicePrivacy 2024 Protocol

• VoicePrivacy 2024 Protocol is a standardized framework for anonymizing speech by concealing speaker identity while preserving linguistic content and emotional state.
• It decomposes the process into feature extraction, speaker embedding anonymization, and synthesis, with evaluation based on EER, WER, and UAR metrics over established corpora.
• The protocol exposes the tension between privacy and utility, guiding challenge ranking through practical benchmarks under a semi-informed attack scenario.

The VoicePrivacy 2024 Protocol is the evaluation framework for the VoicePrivacy 2024 challenge, formulated as a semi-informed anonymization task in which an anonymization system must conceal the speaker’s identity in speech recordings while preserving linguistic content and emotional state (Tomashenko et al., 2024, Cai et al., 2024). In the challenge formulation, privacy is assessed with automatic speaker verification, linguistic utility with word error rate, and emotional utility with unweighted average recall on four emotion classes, yielding a three-way privacy–utility benchmark rather than a single optimization target (Cai et al., 2024). Subsequent analyses describe the same framework as the third VoicePrivacy Challenge and emphasize that it provides a reproducible protocol for benchmarking voice anonymization under a well-defined semi-informed threat model (Tomashenko et al., 17 Jan 2026).

1. Task definition and formalization

The protocol defines an anonymization problem at the utterance level. Its stated objectives are to conceal the speaker’s identity in speech recordings, preserve linguistic content, and preserve emotional state (Cai et al., 2024). In the official evaluation plan, participants develop a voice anonymization system for speech data, run the provided evaluation scripts, and submit evaluation results together with anonymized speech data (Tomashenko et al., 2024).

A later formal restatement writes the task as follows: given an input utterance $x$, produce an anonymized utterance $y$ such that $y=\mathcal{A}(x;r)$, where $\mathcal{A}(\cdot; r)$ is a randomized anonymization function and $r$ is a per-utterance random seed (Tomashenko et al., 17 Jan 2026). The same restatement decomposes the mapping into feature extraction, anonymization of speaker embedding, and synthesis:

$$y=\mathcal{A}(x;r)=D\circ (g\times E_s, E_c, E_{f0})(x;r).$$

This formulation is not a separate protocol; it is a compact mathematical rendering of the challenge design used in later analysis (Tomashenko et al., 17 Jan 2026).

The protocol is explicitly utility-preserving rather than purely privacy-maximizing. The challenge targets not only de-identification but also content preservation and emotion preservation, making it distinct from settings in which anonymization quality is judged only by speaker verification failure (Tomashenko et al., 2024, Cai et al., 2024). A recurrent result in later challenge analyses is that privacy, linguistic utility, and emotional expressiveness remain in tension, and that the protocol is built to expose that tension rather than hide it (Cai et al., 2024, Tomashenko et al., 17 Jan 2026).

2. Corpora, partitions, and evaluation sets

The protocol combines read speech for privacy and ASR evaluation with acted emotional speech for SER evaluation, and it also specifies auxiliary corpora for training anonymization systems and studying information leakage (Tomashenko et al., 2024, Cai et al., 2024).

Corpus	Protocol role	Key details
LibriSpeech	ASV and ASR development/evaluation	read English speech, 16 kHz
IEMOCAP	SER evaluation	English dyadic emotional conversations, 16 kHz, 10 actors
LibriTTS	VC training and target/prompt pool	585 h, 24 kHz, 2 456 speakers
VoxCeleb1	study of speaker information in emotion embeddings	1211 train speakers, 40 test speakers, ~153 k utterances

For LibriSpeech, the evaluation plan specifies train-clean-100, train-clean-360, and train-other-500 as training data for the ASV and ASR evaluation systems (Tomashenko et al., 2024). The development split uses dev-clean and dev-clean enrollment, with 29 speakers (15 F, 14 M), 343 enrollment utterances, and 40 trial speakers (20 F, 20 M) forming 1 978 trials (Tomashenko et al., 2024). The evaluation split uses test-clean and test-clean enrollment, with 29 speakers (16 F, 13 M), 438 enrollment utterances, and 40 trial speakers (20 F, 20 M) forming 1 496 trials (Tomashenko et al., 2024). In later protocol summaries, these are also described as fixed speaker sets from the standard LibriSpeech releases (Tomashenko et al., 17 Jan 2026).

For IEMOCAP, the evaluation plan specifies a 5-fold leave-one-conversation-out protocol: in each fold, 4 conversations (8 speakers) train the SER-eval model, and the remaining conversation provides one speaker for development and one for evaluation (Tomashenko et al., 2024). The corpus is described as English dyadic emotional conversations, 16 kHz, with 10 actors and emotions neutral, sadness, anger, happiness; later summaries describe 10 actors (5 F, 5 M), 12 h total, and four emotion classes neutral, sadness, anger, happiness/excitement (Tomashenko et al., 2024, Tomashenko et al., 17 Jan 2026). Cai et al. refer to IEMOCAP-dev and IEMOCAP-test as the SER evaluation sets and describe the emotion utility target as UAR over four classes: Happy, Neutral, Sad, Angry (Cai et al., 2024). This difference is terminological rather than a different benchmark.

LibriTTS serves two distinct roles in protocol usage. Cai et al. state that LibriTTS, with 585 h at 24 kHz from 2 456 speakers, is used to train anonymization VC models, as a pool of target speakers for kNN-VC, and as a pool of prompt utterances in ASR-TTS (Cai et al., 2024). VoxCeleb1, with 1211 train speakers, 40 test speakers, and approximately 153 k utterances, is used in that study to examine speaker information in emotion embeddings (Cai et al., 2024).

3. Threat model and privacy assumptions

The core challenge is organized around a semi-informed attacker. In the formulation summarized by Cai et al., the user applies a black-box anonymization system to raw speech, while the attacker receives only anonymized speech and original speaker labels and trains an ASV system to recover identity (Cai et al., 2024). The same source states that the evaluation server automatically anonymizes enrollment and trial utterances and runs ASV, ASR, and SER pipelines (Cai et al., 2024).

The evaluation plan provides the operational detail behind that abstraction. The speaker-verification attacker is retrained on LibriSpeech-train-clean-360 anonymized with the tested system, using an ECAPA-TDNN ASV system denoted ASV_eval$^{anon}$ (Tomashenko et al., 2024). Later summaries make the semi-informed assumption explicit: the adversary knows the anonymization function $\mathcal{A}$, has full white-box access, can apply $\mathcal{A}$ to enrollment utterances, and can retrain or fine-tune an ASV system on anonymized data produced from LibriSpeech-train-clean-360 (Tomashenko et al., 17 Jan 2026).

The protocol’s privacy criterion is therefore not resistance to an uninformed or static verifier. It is resistance to a verifier adapted to the anonymizer. Later streaming work further distinguishes naïve attacker, lazy-informed attacker, and semi-informed attacker, but presents the semi-informed case as the direct continuation of the official VoicePrivacy 2024 setup (Kuzmin et al., 20 Jan 2026). A plausible implication is that the protocol was designed to test anonymization under a stronger and more realistic re-identification regime than a fixed off-the-shelf ASV backend.

4. Reference baselines and admissible system components

The evaluation plan provides six reference systems, B1 through B6, spanning several families of anonymization methods (Tomashenko et al., 2024). B1 combines x-vector extraction with a neural source-filter synthesizer: it extracts F0, 256 d bottleneck features, and a 512 d x-vector, selects $N=200$ furthest x-vectors from a LibriTTS-other-500 pool, randomly averages $y$0 of them into a pseudo-speaker x-vector, and synthesizes with a neural source-filter system conditioned on anonymized x-vector, original F0, and bottleneck features (Tomashenko et al., 2024). B2 performs LPC source-filter analysis and applies a random McAdams coefficient $y$1 per utterance, shifting complex-pole phases before resynthesis (Tomashenko et al., 2024). B3 extracts phonetic transcript, F0, energy, durations, and a GST speaker embedding, replaces the embedding with a Wasserstein-GAN sample dissimilar to the original, randomly rescales per-phone F0 and energy in $y$2, and synthesizes with FastSpeech2 plus HiFi-GAN (Tomashenko et al., 2024).

B4 is a neural audio codec language-model baseline: HuBERT Base provides one semantic token per frame, EnCodec provides 8 acoustic tokens per frame, and Bark’s GPT-like transformer generates new acoustic tokens conditioned on semantic tokens and a random acoustic prompt from a pseudo-speaker pool (Tomashenko et al., 2024). B5 and B6 use ASR bottleneck features plus vector quantization and HiFi-GAN synthesis, differing in the ASR frontend: wav2vec 2.0 plus 3 TDNN-F layers for B5, and a 12-layer TDNN-F for B6 (Tomashenko et al., 2024).

The evaluation plan also constrains training and development resources. It states that all of the following are allowed and that no other data or models may be used: WavLM Base/Large, HuBERT, Whisper, XLS-R, wav2vec 2.0, ContentVec, w2v-BERT; ECAPA2, ECAPA-TDNN (SpeechBrain), Resemblyzer; HiFi-GAN, EnCodec, Bark, VITS, PIPER, RVC, DISSC; LibriSpeech, LibriTTS, Libri-light, VoxCeleb 1&2, VCTK, LJSpeech, ESD, CREMA-D, RAVDESS, SAVEE, EMO-DB, CMU-MOSEI, VGAF, MSP-Podcast, MUSAN, RIR; and the Kaldi, SpeechBrain, and PyTorch toolkits (Tomashenko et al., 2024).

Cai et al. illustrate how the protocol accommodates newer architectures without changing the scoring rules. Their systems include VC-based methods such as ConVec2Mel-VC and kNN-VC, as well as cascaded ASR→TTS methods using Whisper “medium-en” and XTTS, with variants including ConVec2Mel-VC→XTTS, kNN-VC→XTTS, XTTS, Emo$y$3-XTTS, and Emo$y$4-XTTS (Cai et al., 2024). This suggests that the protocol is architecture-agnostic: it standardizes evaluation while allowing broad methodological diversity.

5. Metrics, evaluation workflow, and ranking

The protocol uses three primary objective metrics. Privacy is measured by Equal Error Rate, defined as the point where False Acceptance Rate equals False Rejection Rate (Tomashenko et al., 2024, Cai et al., 2024). In the formulation used by Cai et al.,

$y$5

where $y$6 solves $y$7 (Cai et al., 2024). Higher EER indicates better anonymization (Cai et al., 2024).

Linguistic utility is measured by Word Error Rate:

$y$8

It is computed on libri-dev-asr and libri-test-asr by comparing ASR transcripts on anonymized audio to ground truth, and lower WER indicates better content preservation (Cai et al., 2024). The official evaluation plan specifies a fixed ASR_eval based on wav2vec2-large-960h fine-tuned on LibriSpeech-960 (Tomashenko et al., 2024).

Emotional utility is measured by Unweighted Average Recall:

$y$9

with $y=\mathcal{A}(x;r)$0 emotion classes (Cai et al., 2024). It is computed on IEMOCAP development and test data by comparing the SER prediction on anonymized audio to the original emotion label, and higher UAR indicates better emotion preservation (Cai et al., 2024). The evaluation plan specifies SER_eval as a wav2vec2-based model trained per IEMOCAP fold (Tomashenko et al., 2024).

The procedural workflow is fixed. Cai et al. summarize it in five steps: the anonymization module transforms each source utterance to an anonymized waveform; the ASV attacker trains on the anonymized train set with original speaker labels; each enrollment–test pair from the trial lists is anonymized and scored to obtain similarity scores and EER; anonymized ASR sets are decoded by a fixed ASR system to compute WER; anonymized IEMOCAP sets are scored by a fixed SER model to compute UAR (Cai et al., 2024). The official evaluation plan similarly requires participants to apply their system to LibriSpeech-train-clean-360, dev-clean, and test-clean, retrain ASV_eval$y=\mathcal{A}(x;r)$1 on the anonymized train-clean-360 set, and run the provided Voice-Privacy-Challenge-2024 GitHub evaluation scripts (Tomashenko et al., 2024).

Ranking is not based on a single scalar. The evaluation plan defines four target privacy conditions, minimum EER values of 10 %, 20 %, 30 %, and 40 %; for each target EER$y=\mathcal{A}(x;r)$2, submissions with EER $y=\mathcal{A}(x;r)$3 EER$y=\mathcal{A}(x;r)$4 are ranked separately by ascending WER and descending UAR (Tomashenko et al., 2024). Later summaries state the final reporting more compactly as ranking by $y=\mathcal{A}(x;r)$5 (Cai et al., 2024). The official submission rules also require unmodified evaluation recipes, utterance-level anonymization, an archive containing the exp/ directory and anonymized waveforms, and a detailed system description (Tomashenko et al., 2024).

6. Interpretive issues, empirical findings, and later extensions

A central empirical finding obtained under this protocol is that privacy and emotion preservation are difficult to optimize simultaneously. Cai et al. report that their anonymization pipelines either excel at anonymization or preserving emotion state, but not both simultaneously, and argue that achieving both would require an in-domain emotion recognizer (Cai et al., 2024). The same study also trains a speaker verification system on emotion embeddings alone and reports EERs of 19.3 % on lib-dev-f and less than 10 % on other subsets, demonstrating that emotion embeddings leak speaker identity (Cai et al., 2024). Within the logic of the protocol, this result is significant because emotion preservation is itself a potential identity leakage channel.

Later challenge analysis identifies several evaluation limitations. It states that EER may overestimate privacy under a single attack model, that WER and UAR do not capture perceptual naturalness or prosody nuance, and that the current datasets are clean read English and do not cover noise, accents, multi-speaker settings, or low-resource languages (Tomashenko et al., 17 Jan 2026). The same analysis recommends stronger and more diverse attacks, subjective tests or new objective metrics, fairness-aware anonymization, and real-time low-latency algorithms (Tomashenko et al., 17 Jan 2026). These are not changes to the 2024 protocol itself, but they delimit what the protocol measures well and what it leaves for future work.

The protocol also became the basis for subsequent extensions. The First VoicePrivacy Attacker Challenge evaluates attacker systems against anonymization systems submitted to VoicePrivacy 2024 and describes the resulting setup as a “best-attacker vs best-anonymizer” benchmark (Tomashenko et al., 2024). Later work on streaming anonymization applies the VoicePrivacy 2024 protocol under lazy-informed and semi-informed attackers to real-time systems (Kuzmin et al., 20 Jan 2026), and full-duplex dialogue work adapts the protocol to hidden-state leakage analysis in always-on speech LLMs (Kuzmin et al., 9 Mar 2026). This suggests that VoicePrivacy 2024 has evolved from a challenge specification into a reference evaluation framework for privacy-preserving speech transformation more broadly.

One common misconception is that the protocol establishes a formal differential privacy guarantee. It does not: no differential privacy guarantee is formally defined in the challenge or the paper (Cai et al., 2024). The protocol instead operationalizes privacy through attack-based speaker re-identification failure, with EER as the principal privacy metric (Tomashenko et al., 2024, Cai et al., 2024).