DDPO-VC: Speaker De-Identification via Diffusion Denoising Policy Optimization

Abstract: A key challenge of speaker de-identification is the balance between privacy and utility. Many utility variables, such as the cognitive health status of the speaker, are correlated with the privacy variable, such as the speaker identity, violating the independence assumption held by the disentanglement-based approaches, causing leakage of private information and the loss of useful information for downstream tasks. To tackle this challenge, we propose a general framework, DDPO-VC, for speaker de-identification through reinforcement learning-based post-training with diffusion models. Learning from reward signals combining knowledge from privacy-focused and utility-focused teachers, our method outperforms various strong \deid/ methods in both privacy preservation and cognitive utility on two commonly used dementia speech benchmarks. Please check out our code\footnote{\href{https://github.com/cactuswiththoughts/DDPO-VC}{https://github.com/cactuswiththoughts/DDPO-VC}} and demo\footnote{\href{https://cactuswiththoughts.github.io/SpeakerDeID-Demo/}{https://cactuswiththoughts.github.io/SpeakerDeID-Demo/}}.

• The paper introduces DDPO-VC, which reframes speaker de-identification as reinforcement learning fine-tuning over diffusion models to balance privacy and utility.
• The study leverages teacher-based reward functions from a speaker verifier and dementia classifier to drive denoising policy optimization, achieving improved EER and AUC scores.
• Experimental results on ADReSS and FHS-Gold datasets validate the approach, showing up to 19% relative improvement in utility AUC and high naturalness via UTMOS.

DDPO-VC: Speaker De-Identification via Diffusion Denoising Policy Optimization

Motivation and Problem Statement

Speaker de-identification (DeID) is central to privacy-preserving speech processing, particularly in healthcare, where both privacy (speaker identity) and utility (cognitive health status) must be balanced. The challenge is exacerbated in domains like dementia speech, as utility variables (e.g., cognitive impairment indicators) are highly correlated with the private variable (speaker identity), violating the independence assumption often imposed by disentanglement-driven methods. Empirically, these correlations lead to incomplete privacy protection and degrade utility in downstream tasks. The paper introduces DDPO-VC, which reframes DeID as reinforcement learning (RL) fine-tuning over diffusion models (DMs), leveraging teacher-based reward functions to surpass the privacy-utility tradeoff of prior approaches.

Methodology

Model Architecture

The model comprises two main stages: (1) base conditional diffusion model (CDM) pretraining and (2) RL-based post-training via denoising diffusion policy optimization (DDPO). The CDM generates speech conditioned on a semantic encoder, aiming to suppress speaker identity while retaining paralinguistic information.

Figure 1: Model architecture of DDPO-VC, detailing the separation of utility extraction and speaker de-identification using RL post-training on a diffusion backbone.

Semantic Encoder: A frozen large speech encoder (e.g., WavLM) extracts continuous representations, preserving utility but inevitably retaining partial speaker cues.

Conditional Diffusion Model: Trained to reconstruct speech from semantic encodings, the CDM is first optimized via conditional score matching, minimizing the $\ell_2$ norm between predicted and actual noise as in standard DDPM.

Reinforcement Learning Post-training: To optimize the utility–privacy tradeoff, the base CDM is fine-tuned using DDPO. Teacher models—an ECAPA-TDNN speaker verifier and task-specific dementia classifier—deliver reward signals for privacy and utility, respectively. Speaker independence is incentivized by maximizing embedding distance between original and generated utterances, while dementia classifier confidence acts as a utility reward. The total reward is a weighted sum of these components.

Optimization Details

The RL objective is entropy-regularized, balancing reward maximization against KL-divergence from the pretrained CDM to mitigate reward hacking. The loss is expressed as:

$$\mathcal{L}_{DDPO}(\theta) = \mathbb{E}[r(x')\lVert \epsilon_\theta(x_t, \tilde{c}, t) - \varepsilon \rVert^2] + \beta \mathbb{E}[\lVert \epsilon_\theta(x_t, \tilde{c}, t) - \epsilon_{\bar{\theta}}(x_t, \tilde{c}, t) \rVert^2]$$

Normalized reward clipping and trust region strategies are employed to stabilize learning. Additionally, the utility teacher can be updated in an actor-critic loop for improved reward alignment as the distribution of generated samples shifts during post-training.

Experimental Evaluation

Datasets and Baselines

Experiments are primarily conducted on dementia speech datasets—ADReSS and FHS-Gold—using domain-specific splits and preprocessing to isolate patient speech. Baselines include non-diffusion VC models (KNN-VC, Linear-VC, TriAAN-VC, VEVO, FACodec), TTS models (VALL-E), and direct preference optimization (DPO) post-training for diffusion models.

Metrics

DeID quality is jointly evaluated using:

• Equal Error Rate (EER): Speaker verification failure rate (higher $\rightarrow$ greater privacy).
• Dementia Classification AUC: Area under curve, both zero-shot (zs) and finetuned (ft), reflecting downstream utility.
• UTMOS: Objective perceptual quality metric.

Results

Training Convergence

Figure 2: Training loss curves for DDPO-VC on ADReSS, illustrating convergence dynamics across RL objective variants.

Main Results

On ADReSS, DDPO-VC (trainable reward) achieves a zero-shot AUC of 0.77 and finetuned AUC of 0.87, outperforming all non-TTS VC baselines. The EER attains 0.43—significantly above Linear-VC (0.28)—highlighting superior privacy. VALL-E achieves the highest EER (0.46) but at the expense of utility in health-related tasks due to its reliance on lossy text input.

Notably, generated speech maintains high naturalness, with UTMOS approaching original speech levels. Furthermore, RL post-training delivers a 10–19% relative improvement in utility AUC over the base model without privacy regression, with an additional boost when using a trainable utility teacher, confirming the advantage of adaptive reward models as the generation distribution migrates.

On FHS-Gold, results are consistent: DDPO-VC matches or surpasses most baselines in AUC and maintains competitive EER, especially after RL post-training.

Ablations

Ablation studies reveal:

• Utility Teacher Type: Whisper-based reward teachers yield higher AUCs than EfficientNet-based ones, likely due to richer linguistic feature extraction.
• Privacy-Utility Reward Weighting: Increasing speaker reward weight ($\lambda_{speaker}$) beyond zero degrades overall performance, likely due to either the semantic encoder's efficiency at removing speaker cues or risk of reward hacking under fixed (frozen) teachers.
• Post-Training Objective: Standard DPO under-performs DDPO, particularly for utility preservation, confirming that DDPO is more effective for nuanced, correlated privacy-utility scenarios.

Implications

The deployment of RL post-training on diffusion models for DeID establishes a scalable, model-free framework that addresses correlated privacy/utility variables without restrictive disentanglement assumptions. The capacity to synthesize natural speech while balancing competing constraints is validated in high-stake healthcare domains, demonstrating that such systems can serve in privacy-sensitive annotation and analysis pipelines.

These findings suggest DeID performance cannot be fully captured by traditional metrics such as naturalness or speaker verification alone. Comprehensive evaluation—including task-specific utility—is essential for high-consequence applications. The observed reward hacking and distributional shift effects during RL highlight the critical importance of adaptive, domain-tailored reward modeling.

Future research can explore adversarial robustness, improved privacy reward estimators via trainable or ensemble teachers, and extension to broader utility variables and more diverse domains.

Conclusion

DDPO-VC advances DeID methodology through RL-finetuned diffusion modeling, optimizing the privacy-utility tradeoff via composite teacher-driven rewards. Empirical analysis shows that this framework achieves state-of-the-art privacy and utility preservation in healthcare speech, supporting the generalization of RL-based approaches for privacy-centric audio generation and underscoring the necessity for multidimensional evaluation and robust reward architectures.