DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

Abstract: To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points.

• The paper introduces a diffusion-based framework that provides explicit, continuous control over prosody in voice anonymization through a single scalar parameter.
• It refines codec-based speech embeddings by disentangling content, prosody, and speaker features to navigate the privacy–utility trade-off with classifier-free guidance.
• Experimental results on LibriTTS demonstrate competitive EER, F0-correlation, and emotion recognition metrics, confirming robust performance across various privacy settings.

DiffAnon: Diffusion-Based Prosody Control for Voice Anonymization

Introduction

Voice anonymization faces a fundamental privacy–utility trade-off due to the entanglement among linguistic content, paralinguistic information, and speaker identity. Prosody is pivotal in this trade-off: preserving it supports expressiveness and meaning, but prosodic features are strong correlates of speaker identity, thus heightening re-identification risk. Existing anonymization systems either suppress prosody for privacy, heuristically perturb prosodic features without guarantees, or operate at fixed, non-interpolatable privacy-utility points. The paper "DiffAnon: Diffusion-based Prosody Control for Voice Anonymization" (2604.26281) introduces a diffusion-based framework that, for the first time, provides explicit, continuous inference-time control over prosodic preservation in a unified model via classifier-free guidance (CFG).

Methodology: Diffusion-Based Controllable Voice Anonymization

DiffAnon leverages denoising diffusion probabilistic models (DDPMs) for conditional generation in speaker anonymization, refining codec-based speech embeddings conditioned on disentangled content, prosody, and speaker embeddings. At training, the model reconstructs codec embeddings from the SpeechTokenizer RVQ encoder, receiving as conditional inputs: (1) first-level semantic tokens capturing linguistic content, (2) frame-level prosody features extracted by a masked prosody model (MPM), and (3) utterance-level speaker representations from a FreeVC encoder.

At inference, DiffAnon anonymizes an utterance by combining the source's content and controllable contributions of its prosody with a randomly sampled pseudo-speaker embedding. This is achieved via CFG, whereby the model’s denoising process interpolates between unconditional and fully prosody-conditioned generations. The system thus enables explicit navigation across the privacy–utility curve by adjusting a single scalar parameter at generation time.

Figure 1: a) Conditional diffusion training to construct codec embeddings; b) anonymization inference via adjusted prosody with CFG and pseudo-speaker condition sampled from a pseudo-speaker pool.

Technically, prosody and speaker embeddings are projected via conv1d layers, aligned to the frame rate and channel dimension of the codec frame sequence, and added as conditional information at each denoising step. The semantic content embedding is always provided, anchoring content preservation independently of other conditions.

Classifier-Free Guidance for Prosody and Privacy Control

CFG in DiffAnon operates as a convex combination of conditional (i.e., with prosody) and unconditional (i.e., null prosody) denoising predictions. By modulating the weight $w_{\mathrm{pro}}$ at inference, one can explicitly interpolate between strict anonymization (minimal source prosody retained for maximal privacy) to high-fidelity reconstruction (maximal prosody preserved for maximal utility).

Additionally, privacy can be further strengthened by augmenting the CFG weighting on the pseudo-speaker condition. Multiple architectural variants (prosody-only control, dual control on prosody and pseudo-speaker, etc.) demonstrate the flexibility of the framework.

Experimental Evaluation

Experiments are primarily conducted on LibriTTS, with evaluation using the VoicePrivacy Challenge 2024 protocol, measuring privacy (using equal error rate, EER), utility via ASR-based word error rate (WER), and prosodic/affective utility via emotion recognition (UAR) and F0-correlation.

Across all settings, DiffAnon remains competitive with or surpasses strong baselines. With $w_{\mathrm{pro}}=1.0$, DiffAnon achieves 76.67% F0-correlation and 52.32% UAR on the dev set, confirming robust prosodic and emotional preservation. As the prosody control parameter decreases, privacy (EER) increases monotonically, with a corresponding smooth degradation in F0-correlation and UAR. For instance, EER increases from 33.09% to 42.43% (libri-test "lazy" scenario) as prosody is gradually suppressed, with F0-correlation dropping from 75.58% to 62.45%. WER increases only moderately, showing that content is robustly preserved by the conditional codec embedding.

Figure 2: DiffAnon's privacy–utility trade-off curve as a function of prosody guidance; the model traverses intermediate operating points via a single parameter sweep.

A strong result is that under pseudo-speaker dominant CFG ($w_{spk}=3$), DiffAnon reaches up to 48.16% EER while WER remains below 6.3%, matching the best reported systems in privacy-focused scenarios, but with superior interpretability and continuous trade-off control.

Theoretical and Practical Implications

DiffAnon empirically demonstrates that prosody is the primary axis for controlling the privacy–utility trade-off in voice anonymization. Unlike prior approaches that fix the operating point at training or rely on crude, often unpredictable perturbation, DiffAnon’s generative framework decouples privacy-versus-utility selection from model training, enabling dynamic, structured adjustment per utterance at deployment time. This supports both regulatory-compliant privacy settings and application-specific requirements for expressiveness and affective communication, e.g., in call centers, clinical interviews, or public datasets for research.

Mechanistically, the model’s successful factorization of content, prosody, and speaker within the diffusion and codec embedding space suggests new directions for explainable and controllable generative speech models beyond anonymization, such as universal translator frameworks, emotion transfer, and unsupervised paralinguistic attribute modification.

Future Directions

Potential extensions include: (1) fine-grained control over additional prosodic and paralinguistic features (e.g., duration, rhythm, style), (2) multi-language or code-switched anonymization, (3) adversarial robustness analysis and defenses for advanced anonymization attackers, and (4) adaptation to end-to-end self-supervised or large speech foundation models. Moreover, DiffAnon establishes a methodological baseline for structured privacy–utility control in generative models that could be generalized to other domains containing entangled personal information attributes.

Conclusion

DiffAnon demonstrates that diffusion-based conditional generation, when combined with classifier-free guidance on disentangled prosody and speaker embeddings, enables explicit, continuous control over the privacy–utility trade-off in voice anonymization. The framework attains state-of-the-art or competitive results on both privacy and utility, provides structured navigation of operating points via a single hyperparameter, and constitutes a step change in interpretability and flexibility for privacy-preserving speech technologies. The approach substantiates the centrality of prosody for privacy–utility balancing, and furnishes a well-founded basis for future development of controllable generative anonymization methods.

(2604.26281)