DiffAnon: Diffusion Voice Anonymization

• DiffAnon is a diffusion-based voice anonymization framework that treats prosody as a controllable inference-time variable.
• It replaces source speaker embeddings with pseudo-speaker embeddings and uses semantic representations from an RVQ codec to preserve content.
• Empirical evaluations reveal a structured privacy–utility trade-off, with metrics like EER, WER, UAR, and F0-correlation reflecting performance shifts.

DiffAnon is a diffusion-based voice anonymization framework that treats prosody preservation as an explicit inference-time control variable rather than a fixed design choice. It anonymizes speaker identity by replacing the source speaker condition with a pseudo-speaker embedding, preserves linguistic content through a semantic representation derived from an RVQ codec, and modulates source prosody with classifier-free guidance (CFG). The method is formulated in the latent embedding space of SpeechTokenizer, uses a conditional diffusion backbone to reconstruct full codec embeddings, and evaluates the privacy–utility trade-off with equal error rate (EER), word error rate (WER), unweighted average recall (UAR), and rank-based $F_0$ correlation (Ulgen et al., 29 Apr 2026).

1. Problem setting and conceptual scope

DiffAnon addresses a specific tension in voice anonymization: speech must remain useful after anonymization, but many of the cues that preserve usefulness also reveal identity. In the VoicePrivacy framing used by the paper, utility includes at least linguistic intelligibility and para-linguistic fidelity, while privacy is resistance to speaker verification. Prosody is central to this tension because it carries meaning, emphasis, intent, and emotion, yet is also correlated with speaker identity through pitch movement, rhythm, and phrasing (Ulgen et al., 29 Apr 2026).

The method is proposed against two limitations attributed to existing voice anonymization systems. First, some methods discard prosody to improve privacy, thereby reducing expressive fidelity. Second, other methods preserve prosody without a principled mechanism for controlling the utility–privacy trade-off, operating only at fixed design points. DiffAnon’s main claim is that this trade-off should be adjustable continuously at inference time, within one trained model, by controlling the degree of source prosody preservation with CFG (Ulgen et al., 29 Apr 2026).

This positions DiffAnon as a conditional generative anonymization system rather than a differential privacy mechanism or a diffusion-based anomaly detector. Its output is anonymized speech that preserves content and modulates prosodic fidelity, not a privacy guarantee in the $(\varepsilon,\delta)$-DP sense and not an anomaly score (Ulgen et al., 29 Apr 2026).

2. Representation design and conditioning structure

DiffAnon operates in the latent embedding space of SpeechTokenizer, an RVQ-based neural codec with eight quantization stages $Q^{1:8}$. The clean diffusion target is the full codec embedding

$$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$

while the semantic condition is the first RVQ level

$c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$

The paper states that $Q^1$ is semantically distilled and primarily captures linguistic content with relatively little speaker information, whereas higher levels $Q^{2:8}$ add finer acoustic detail (Ulgen et al., 29 Apr 2026).

Three conditioning streams are used to control generation:

Stream	Representation	Role
Semantic/content	$Q^1$	Preserves linguistic content
Prosody	$z'_{mpm} \in \mathbb{R}^{256 \times T}$	Carries source prosodic structure
Speaker	$z'_{spk} \in \mathbb{R}^{256 \times T}$	Controls speaker identity via pseudo-speaker substitution

The prosody condition comes from frame-level masked prosody model features, temporally interpolated to match the codec frame rate. The speaker condition comes from a pre-trained FreeVC speaker encoder; it is utterance-level, 256-dimensional, and repeated over frames. At inference, the source speaker embedding is replaced by a pseudo-speaker $(\varepsilon,\delta)$0, sampled from a pool built from LibriTTS training speakers (Ulgen et al., 29 Apr 2026).

The diffusion backbone is inspired by NaturalSpeech2 and contains 40 WaveNet-style residual blocks, 1D non-dilated convolutions, kernel size 5, and 1024 channels throughout. Timestep embeddings are produced via an MLP. Prosody and speaker conditions are injected through separate non-dilated 1D convolutional projections, whereas the semantic condition $(\varepsilon,\delta)$1 is added directly in each residual block. This design makes the semantic scaffold persistent while speaker identity and prosody act as controllable acoustic refinements (Ulgen et al., 29 Apr 2026).

3. Diffusion formulation and CFG-based prosody control

DiffAnon uses a DDPM-style latent diffusion process with $(\varepsilon,\delta)$2-prediction. The forward noising equation is

$(\varepsilon,\delta)$3

with

$(\varepsilon,\delta)$4

Rather than predicting noise, the model predicts the clean codec embedding directly through the loss

$(\varepsilon,\delta)$5

This is the only explicit training loss given for the anonymization model (Ulgen et al., 29 Apr 2026).

The paper’s most distinctive contribution is the use of classifier-free guidance to create a continuous knob over source prosody preservation. The main prosody-adjusted guidance rule is

$(\varepsilon,\delta)$6

Equivalently,

$(\varepsilon,\delta)$7

The reported operating points use $(\varepsilon,\delta)$8, where $(\varepsilon,\delta)$9 corresponds to full source prosody conditioning and $Q^{1:8}$0 to null prosody (Ulgen et al., 29 Apr 2026).

The paper also evaluates pseudo-speaker guidance, used when prosody is null: $Q^{1:8}$1 with reported results at $Q^{1:8}$2. In the implementation, a null condition is realized by adding zero-valued conditioning (Ulgen et al., 29 Apr 2026).

This formulation makes the trade-off structured. Content remains anchored by $Q^{1:8}$3, pseudo-speaker conditioning keeps generation away from the original speaker, and varying $Q^{1:8}$4 interpolates between stronger prosodic fidelity and stronger anonymization. A plausible implication is that DiffAnon treats prosody not as a nuisance variable to suppress wholesale, but as the principal axis along which the privacy–utility operating point is navigated.

4. Training regime and anonymization procedure

Training is reconstructive rather than pairwise anonymization-specific. The model is trained on LibriTTS training subsets for about 400k steps with learning rate $Q^{1:8}$5, batch size 8, and a single NVIDIA H100 GPU. Inference uses DDIM sampling with 100 denoising steps (Ulgen et al., 29 Apr 2026).

To enable CFG, DiffAnon uses random condition dropping during training with the following mixture:

• 50% full conditioning:

$Q^{1:8}$6

• 30% prosody dropped:

$Q^{1:8}$7

• 20% both prosody and speaker dropped:

$Q^{1:8}$8

The paper explicitly avoids speaker-only dropping. Its reported reason is that, in preliminary experiments, if speaker was dropped but prosody retained, the model learned to exploit speaker information leaking through prosodic features, which harmed anonymization (Ulgen et al., 29 Apr 2026).

At inference, anonymization proceeds by extracting source $Q^{1:8}$9, extracting source prosody features from the masked prosody model, replacing the source speaker embedding with a pseudo-speaker embedding $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$0, applying CFG with the chosen $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$1 or $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$2, and decoding the generated codec embeddings to waveform through the SpeechTokenizer decoder. Pseudo-speakers are constructed by averaging speaker embeddings across each LibriTTS training speaker’s utterances, yielding a stable embedding per pseudo-speaker identity (Ulgen et al., 29 Apr 2026).

This means anonymization is achieved by condition substitution plus guidance, not by a separate anonymization loss or anonymization-pair supervision. The model is trained to reconstruct codec embeddings under varying dropped-condition configurations, and anonymization emerges when source speaker identity is replaced while semantic conditioning is retained. A plausible implication is that DiffAnon’s controllability depends heavily on the partial disentanglement assumed by $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$3, the masked prosody model, and the utterance-level speaker embedding.

5. Evaluation protocol and controllable operating points

DiffAnon is evaluated under the VoicePrivacy Challenge 2024 protocol. Training uses LibriTTS subsets; privacy and ASR utility are evaluated on Librispeech-based dev/test sets; emotion preservation uses IEMOCAP for UAR; and VoicePrivacy 2022-style evaluation on libri-dev and libri-test is used for rank-based $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$4 correlation (Ulgen et al., 29 Apr 2026).

The reported metrics are:

• Privacy: EER under lazy-informed and semi-informed attacker settings; higher is better.
• Utility/content: WER; lower is better.
• Utility/emotion: UAR from speech emotion recognition; higher is better.
• Utility/prosody: rank-based $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$5 correlation; higher is better (Ulgen et al., 29 Apr 2026).

The central empirical result is the monotonic behavior across the prosody sweep. The reported test-set operating points are:

Setting	Privacy	Utility
$$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$6	lazy EER 33.09, semi EER 14.53	WER 4.62, UAR 50.80, $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$7-corr 75.58
$$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$8	lazy EER 34.40	WER 4.75, UAR 50.38, $$\mathbf{x}_0 = Q^{1:8} \in \mathbb{R}^{1024 \times T},$$9-corr 73.82
$c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$0	lazy EER 36.41, semi EER 17.15	WER 5.02, UAR 48.93, $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$1-corr 68.23
$c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$2	lazy EER 41.82, semi EER 18.87	WER 5.38, UAR 46.46, $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$3-corr 63.76
$c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$4	lazy EER 42.43, semi EER 20.66	WER 5.61, UAR 45.23, $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$5-corr 62.45
$c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$6, null prosody	lazy EER 48.16, semi EER 22.78	WER 6.22, UAR 43.39, $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$7-corr 56.06

The paper summarizes this as a smooth interpolation: as $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$8 decreases, EER rises, while WER rises moderately and both UAR and $c_{sem} = Q^1 \in \mathbb{R}^{1024 \times T}.$9-correlation decrease. One explicit comparison given in the paper is that increasing test lazy EER from $Q^1$0 to $Q^1$1 is accompanied by a drop in test $Q^1$2-correlation from $Q^1$3 to $Q^1$4 and UAR from $Q^1$5 to $Q^1$6 (Ulgen et al., 29 Apr 2026).

DiffAnon does not dominate every baseline on every metric. The paper compares against official VoicePrivacy baselines B1–B6 and top systems T8, T9, and T10. It states that privacy-heavy baselines such as B4/B5 reach similar privacy regimes, and that T9/T10 obtain higher UAR. DiffAnon’s main empirical distinction is therefore not uniform state-of-the-art performance at one point, but the fact that one trained model traces a structured privacy–utility curve through inference-time control (Ulgen et al., 29 Apr 2026).

6. Nomenclature, adjacent work, and limitations

The name “DiffAnon” refers specifically to voice anonymization with CFG-based prosody control (Ulgen et al., 29 Apr 2026). It is distinct from several adjacent diffusion literatures that can cause terminological confusion. “Anomaly detection using Diffusion-based methods” studies DDPM- and DiT-based reconstruction for anomaly detection rather than anonymization (Bhosale et al., 2024). “Dynamic Addition of Noise in a Diffusion Model for Anomaly Detection” likewise concerns unsupervised industrial anomaly localization (Tebbe et al., 2024). “DiffSLVA” addresses sign language video anonymization with Stable Diffusion, ControlNet, and a facial-expression module, but not voice anonymization (Xia et al., 2023). “DIFF2” is a differential privacy optimization framework for nonconvex distributed learning, not an anonymization model (Murata et al., 2023).

Several limitations are explicit or strongly implied in the DiffAnon paper. Privacy under semi-informed attack remains materially lower than under lazy-informed attack, including in the stronger anonymization settings. Full source prosody preservation yields substantially worse privacy than lower-$Q^1$7 settings, showing that prosody still leaks identity. Systems T9 and T10 achieve higher UAR, so DiffAnon is not optimized specifically for emotion preservation. Stronger privacy settings also reduce utility measurably: WER increases, UAR falls, and $Q^1$8-correlation declines sharply under null prosody and pseudo-speaker guidance (Ulgen et al., 29 Apr 2026).

The paper also notes that the current control mechanism targets prosody broadly through masked prosody model features rather than decomposing specific prosodic dimensions. Duration is identified as future work. A further limitation is more implicit: the method depends on the assumption that $Q^1$9 is sufficiently speaker-lean and that the masked prosody model sufficiently suppresses speaker information. The deliberate exclusion of speaker-only dropout because of identity leakage through prosody indicates that this disentanglement is not perfect (Ulgen et al., 29 Apr 2026).

Taken together, DiffAnon’s significance lies in showing that diffusion-based voice anonymization need not commit to a single prosody policy. It reformulates anonymization as conditional diffusion over codec embeddings, with semantic anchoring through $Q^{2:8}$0, pseudo-speaker substitution through FreeVC embeddings, and a continuous CFG knob over prosodic fidelity. The method’s principal contribution is therefore architectural and operational: it makes prosody an explicit, interpolatable inference-time control axis for navigating the utility–privacy trade-off in speech anonymization (Ulgen et al., 29 Apr 2026).