---
title: VoicePrivacy 2024 Protocol
url: https://www.emergentmind.com/topics/voiceprivacy-2024-protocol
type: topic
---

# VoicePrivacy 2024 Protocol

The VoicePrivacy 2024 Protocol is the evaluation framework for the VoicePrivacy 2024 challenge, formulated as a semi-informed anonymization task in which an anonymization system must conceal the speaker’s identity in speech recordings while preserving linguistic content and emotional state [2404.02677][2409.03655]. In the challenge formulation, privacy is assessed with automatic speaker verification, linguistic utility with word error rate, and emotional utility with unweighted average recall on four emotion classes, yielding a three-way privacy–utility benchmark rather than a single optimization target [2409.03655]. Subsequent analyses describe the same framework as the third VoicePrivacy Challenge and emphasize that it provides a reproducible protocol for benchmarking voice anonymization under a well-defined semi-informed threat model [2601.11846].

## 1. Task definition and formalization

The protocol defines an anonymization problem at the utterance level. Its stated objectives are to conceal the speaker’s identity in speech recordings, preserve linguistic content, and preserve emotional state [2409.03655]. In the official evaluation plan, participants develop a voice anonymization system for speech data, run the provided evaluation scripts, and submit evaluation results together with anonymized speech data [2404.02677].

A later formal restatement writes the task as follows: given an input utterance $x$, produce an anonymized utterance $y$ such that $y=\mathcal{A}(x;r)$, where $\mathcal{A}(\cdot; r)$ is a randomized anonymization function and $r$ is a per-utterance random seed [2601.11846]. The same restatement decomposes the mapping into feature extraction, anonymization of speaker embedding, and synthesis:
$$
y=\mathcal{A}(x;r)=D\circ (g\times E_s, E_c, E_{f0})(x;r).
$$
This formulation is not a separate protocol; it is a compact mathematical rendering of the challenge design used in later analysis [2601.11846].

The protocol is explicitly utility-preserving rather than purely privacy-maximizing. The challenge targets not only de-identification but also content preservation and emotion preservation, making it distinct from settings in which anonymization quality is judged only by speaker verification failure [2404.02677][2409.03655]. A recurrent result in later challenge analyses is that privacy, linguistic utility, and emotional expressiveness remain in tension, and that the protocol is built to expose that tension rather than hide it [2409.03655][2601.11846].

## 2. Corpora, partitions, and evaluation sets

The protocol combines read speech for privacy and ASR evaluation with acted emotional speech for SER evaluation, and it also specifies auxiliary corpora for training anonymization systems and studying information leakage [2404.02677][2409.03655].

| Corpus | Protocol role | Key details |
|---|---|---|
| LibriSpeech | ASV and ASR development/evaluation | read English speech, 16 kHz |
| IEMOCAP | SER evaluation | English dyadic emotional conversations, 16 kHz, 10 actors |
| LibriTTS | VC training and target/prompt pool | 585 h, 24 kHz, 2 456 speakers |
| VoxCeleb1 | study of speaker information in emotion embeddings | 1211 train speakers, 40 test speakers, ~153 k utterances |

For LibriSpeech, the evaluation plan specifies train-clean-100, train-clean-360, and train-other-500 as training data for the ASV and ASR evaluation systems [2404.02677]. The development split uses dev-clean and dev-clean enrollment, with 29 speakers (15 F, 14 M), 343 enrollment utterances, and 40 trial speakers (20 F, 20 M) forming 1 978 trials [2404.02677]. The evaluation split uses test-clean and test-clean enrollment, with 29 speakers (16 F, 13 M), 438 enrollment utterances, and 40 trial speakers (20 F, 20 M) forming 1 496 trials [2404.02677]. In later protocol summaries, these are also described as fixed speaker sets from the standard LibriSpeech releases [2601.11846].

For IEMOCAP, the evaluation plan specifies a 5-fold leave-one-conversation-out protocol: in each fold, 4 conversations (8 speakers) train the SER-eval model, and the remaining conversation provides one speaker for development and one for evaluation [2404.02677]. The corpus is described as English dyadic emotional conversations, 16 kHz, with 10 actors and emotions neutral, sadness, anger, happiness; later summaries describe 10 actors (5 F, 5 M), 12 h total, and four emotion classes neutral, sadness, anger, happiness/excitement [2404.02677][2601.11846]. Cai et al. refer to IEMOCAP-dev and IEMOCAP-test as the SER evaluation sets and describe the emotion utility target as UAR over four classes: Happy, Neutral, Sad, Angry [2409.03655]. This difference is terminological rather than a different benchmark.

LibriTTS serves two distinct roles in protocol usage. Cai et al. state that LibriTTS, with 585 h at 24 kHz from 2 456 speakers, is used to train anonymization VC models, as a pool of target speakers for kNN-VC, and as a pool of prompt utterances in ASR-TTS [2409.03655]. VoxCeleb1, with 1211 train speakers, 40 test speakers, and approximately 153 k utterances, is used in that study to examine speaker information in emotion embeddings [2409.03655].

## 3. Threat model and privacy assumptions

The core challenge is organized around a semi-informed attacker. In the formulation summarized by Cai et al., the user applies a black-box anonymization system to raw speech, while the attacker receives only anonymized speech and original speaker labels and trains an ASV system to recover identity [2409.03655]. The same source states that the evaluation server automatically anonymizes enrollment and trial utterances and runs ASV, ASR, and SER pipelines [2409.03655].

The evaluation plan provides the operational detail behind that abstraction. The speaker-verification attacker is retrained on LibriSpeech-train-clean-360 anonymized with the tested system, using an ECAPA-TDNN ASV system denoted ASV\_eval$^{anon}$ [2404.02677]. Later summaries make the semi-informed assumption explicit: the adversary knows the anonymization function $\mathcal{A}$, has full white-box access, can apply $\mathcal{A}$ to enrollment utterances, and can retrain or fine-tune an ASV system on anonymized data produced from LibriSpeech-train-clean-360 [2601.11846].

The protocol’s privacy criterion is therefore not resistance to an uninformed or static verifier. It is resistance to a verifier adapted to the anonymizer. Later streaming work further distinguishes naïve attacker, lazy-informed attacker, and semi-informed attacker, but presents the semi-informed case as the direct continuation of the official VoicePrivacy 2024 setup [2601.13948]. A plausible implication is that the protocol was designed to test anonymization under a stronger and more realistic re-identification regime than a fixed off-the-shelf ASV backend.

## 4. Reference baselines and admissible system components

The evaluation plan provides six reference systems, B1 through B6, spanning several families of anonymization methods [2404.02677]. B1 combines x-vector extraction with a neural source-filter synthesizer: it extracts F0, 256 d bottleneck features, and a 512 d x-vector, selects $N=200$ furthest x-vectors from a LibriTTS-other-500 pool, randomly averages $N^*=100$ of them into a pseudo-speaker x-vector, and synthesizes with a neural source-filter system conditioned on anonymized x-vector, original F0, and bottleneck features [2404.02677]. B2 performs LPC source-filter analysis and applies a random McAdams coefficient $\alpha \sim U(0.5,0.9)$ per utterance, shifting complex-pole phases before resynthesis [2404.02677]. B3 extracts phonetic transcript, F0, energy, durations, and a GST speaker embedding, replaces the embedding with a Wasserstein-GAN sample dissimilar to the original, randomly rescales per-phone F0 and energy in $[0.6,1.4)$, and synthesizes with FastSpeech2 plus HiFi-GAN [2404.02677].

B4 is a neural audio codec language-model baseline: HuBERT Base provides one semantic token per frame, EnCodec provides 8 acoustic tokens per frame, and Bark’s GPT-like transformer generates new acoustic tokens conditioned on semantic tokens and a random acoustic prompt from a pseudo-speaker pool [2404.02677]. B5 and B6 use ASR bottleneck features plus vector quantization and HiFi-GAN synthesis, differing in the ASR frontend: wav2vec 2.0 plus 3 TDNN-F layers for B5, and a 12-layer TDNN-F for B6 [2404.02677].

The evaluation plan also constrains training and development resources. It states that all of the following are allowed and that no other data or models may be used: WavLM Base/Large, HuBERT, Whisper, XLS-R, wav2vec 2.0, ContentVec, w2v-BERT; ECAPA2, ECAPA-TDNN (SpeechBrain), Resemblyzer; HiFi-GAN, EnCodec, Bark, VITS, PIPER, RVC, DISSC; LibriSpeech, LibriTTS, Libri-light, VoxCeleb 1&2, VCTK, LJSpeech, ESD, CREMA-D, RAVDESS, SAVEE, EMO-DB, CMU-MOSEI, VGAF, MSP-Podcast, MUSAN, RIR; and the Kaldi, SpeechBrain, and PyTorch toolkits [2404.02677].

Cai et al. illustrate how the protocol accommodates newer architectures without changing the scoring rules. Their systems include VC-based methods such as ConVec2Mel-VC and kNN-VC, as well as cascaded ASR→TTS methods using Whisper “medium-en” and XTTS, with variants including ConVec2Mel-VC→XTTS, kNN-VC→XTTS, XTTS, Emo$_{(MSP)}$-XTTS, and Emo$_{(IEMOCAP)}$-XTTS [2409.03655]. This suggests that the protocol is architecture-agnostic: it standardizes evaluation while allowing broad methodological diversity.

## 5. Metrics, evaluation workflow, and ranking

The protocol uses three primary objective metrics. Privacy is measured by Equal Error Rate, defined as the point where False Acceptance Rate equals False Rejection Rate [2404.02677][2409.03655]. In the formulation used by Cai et al.,
$$
\mathrm{EER}=\mathrm{FRR}(\tau^*)=\mathrm{FAR}(\tau^*)
$$
where $\tau^*$ solves $\mathrm{FRR}(\tau)=\mathrm{FAR}(\tau)$ [2409.03655]. Higher EER indicates better anonymization [2409.03655].

Linguistic utility is measured by Word Error Rate:
$$
\mathrm{WER}=\frac{\#\mathrm{Substitutions}+\#\mathrm{Deletions}+\#\mathrm{Insertions}}{\#\mathrm{Reference\ Words}}.
$$
It is computed on libri-dev-asr and libri-test-asr by comparing ASR transcripts on anonymized audio to ground truth, and lower WER indicates better content preservation [2409.03655]. The official evaluation plan specifies a fixed ASR\_eval based on wav2vec2-large-960h fine-tuned on LibriSpeech-960 [2404.02677].

Emotional utility is measured by Unweighted Average Recall:
$$
\mathrm{UAR}=\frac{1}{K}\sum_k \frac{\mathrm{TP}_k}{\mathrm{TP}_k+\mathrm{FN}_k},
$$
with $K=4$ emotion classes [2409.03655]. It is computed on IEMOCAP development and test data by comparing the SER prediction on anonymized audio to the original emotion label, and higher UAR indicates better emotion preservation [2409.03655]. The evaluation plan specifies SER\_eval as a wav2vec2-based model trained per IEMOCAP fold [2404.02677].

The procedural workflow is fixed. Cai et al. summarize it in five steps: the anonymization module transforms each source utterance to an anonymized waveform; the ASV attacker trains on the anonymized train set with original speaker labels; each enrollment–test pair from the trial lists is anonymized and scored to obtain similarity scores and EER; anonymized ASR sets are decoded by a fixed ASR system to compute WER; anonymized IEMOCAP sets are scored by a fixed SER model to compute UAR [2409.03655]. The official evaluation plan similarly requires participants to apply their system to LibriSpeech-train-clean-360, dev-clean, and test-clean, retrain ASV\_eval$^{anon}$ on the anonymized train-clean-360 set, and run the provided Voice-Privacy-Challenge-2024 GitHub evaluation scripts [2404.02677].

Ranking is not based on a single scalar. The evaluation plan defines four target privacy conditions, minimum EER values of 10 %, 20 %, 30 %, and 40 %; for each target EER$_i$, submissions with EER $\ge$ EER$_i$ are ranked separately by ascending WER and descending UAR [2404.02677]. Later summaries state the final reporting more compactly as ranking by $(\mathrm{EER}\uparrow,\mathrm{WER}\downarrow,\mathrm{UAR}\uparrow)$ [2409.03655]. The official submission rules also require unmodified evaluation recipes, utterance-level anonymization, an archive containing the exp/ directory and anonymized waveforms, and a detailed system description [2404.02677].

## 6. Interpretive issues, empirical findings, and later extensions

A central empirical finding obtained under this protocol is that privacy and emotion preservation are difficult to optimize simultaneously. Cai et al. report that their anonymization pipelines either excel at anonymization or preserving emotion state, but not both simultaneously, and argue that achieving both would require an in-domain emotion recognizer [2409.03655]. The same study also trains a speaker verification system on emotion embeddings alone and reports EERs of 19.3 % on lib-dev-f and less than 10 % on other subsets, demonstrating that emotion embeddings leak speaker identity [2409.03655]. Within the logic of the protocol, this result is significant because emotion preservation is itself a potential identity leakage channel.

Later challenge analysis identifies several evaluation limitations. It states that EER may overestimate privacy under a single attack model, that WER and UAR do not capture perceptual naturalness or prosody nuance, and that the current datasets are clean read English and do not cover noise, accents, multi-speaker settings, or low-resource languages [2601.11846]. The same analysis recommends stronger and more diverse attacks, subjective tests or new objective metrics, fairness-aware anonymization, and real-time low-latency algorithms [2601.11846]. These are not changes to the 2024 protocol itself, but they delimit what the protocol measures well and what it leaves for future work.

The protocol also became the basis for subsequent extensions. The First VoicePrivacy Attacker Challenge evaluates attacker systems against anonymization systems submitted to VoicePrivacy 2024 and describes the resulting setup as a “best-attacker vs best-anonymizer” benchmark [2410.07428]. Later work on streaming anonymization applies the VoicePrivacy 2024 protocol under lazy-informed and semi-informed attackers to real-time systems [2601.13948], and full-duplex dialogue work adapts the protocol to hidden-state leakage analysis in always-on speech LLMs [2603.08179]. This suggests that VoicePrivacy 2024 has evolved from a challenge specification into a reference evaluation framework for privacy-preserving speech transformation more broadly.

One common misconception is that the protocol establishes a formal differential privacy guarantee. It does not: no differential privacy guarantee is formally defined in the challenge or the paper [2409.03655]. The protocol instead operationalizes privacy through attack-based speaker re-identification failure, with EER as the principal privacy metric [2404.02677][2409.03655].

Source: https://www.emergentmind.com/topics/voiceprivacy-2024-protocol