Speaker-Zero Retrain Forgetting (spk-ZRF)

Updated 31 July 2025

spk-ZRF is a metric that quantifies the erasure of speaker-specific attributes using Jensen–Shannon divergence to gauge randomness in output embeddings.
Techniques such as Teacher-Guided Unlearning and Negative Gradient unlearning enable controlled erasure while maintaining overall model performance.
Empirical results indicate that high spk-ZRF scores correlate with effective privacy protection and minimal impact on speech quality in TTS and ASR models.

Speaker-Zero Retrain Forgetting (spk-ZRF) describes the phenomenon and measurement of how speech models, particularly those for text-to-speech (TTS) and automatic speech recognition (ASR), fail to preserve prior speaker-related information—or, in the special case of unlearning, how well they irreversibly “forget” the acoustic attributes of specific speakers—when adapting to new data, domains, or privacy requests. This concept has come to the forefront in continual learning, speaker adaptation, and machine unlearning research as a quantitative, reproducible criterion for evaluating privacy, robustness, and generalization in modern speech technology (Kim et al., 27 Jul 2025).

1. Definition and Motivation

Speaker-Zero Retrain Forgetting (spk-ZRF) is devised as an explicit metric and broader framework to assess a model’s capacity to forget the identity of particular speakers following retraining or machine unlearning interventions. For generative models (e.g., Zero-Shot TTS), the aim is to ensure that, upon removal (“zero retraining”) of a given speaker, the model’s ability to reproduce that speaker's timbre, prosody, and style is neutralized—specifically, that its outputs for these prompts become random rather than simply degenerate or muted (Kim et al., 27 Jul 2025). This metric is distinct from general catastrophic forgetting, as it measures not only performance drop but the extent to which model outputs for deleted speakers approach a distribution indistinguishable from synthetic, unconditioned outputs.

spk-ZRF is vital both for privacy (“right to be forgotten”)—ensuring retraining leaves no trace of a speaker’s acoustic identity—and for continual learning scenarios, where adaptation to new speakers or domains should not cause total erasure or unintentional re-synthesis of previously learned speakers (Cheng et al., 1 Jun 2025, Koudounas et al., 21 May 2025).

2. Core Mechanisms and Formal Metric Definition

spk-ZRF as a metric operates by evaluating the randomness or unpredictability in the generated speaker identities for prompts associated with forgotten speakers. Formally, spk-ZRF is defined via the Jensen–Shannon divergence (JSD) between the output speaker embedding distributions for two inference modes:

θ⁻(xᶠ, y): the unlearned model’s output when given a “forget” speaker's acoustic prompt xᶠ and the text prompt y,
θ(y): the same model or a teacher model given only the text prompt (serving as a random, unconditioned proxy).

Given n forget-test pairs {(xᵢ, yᵢ)}, the spk-ZRF is calculated as:

$\text{spk-ZRF} = 1 - \frac{1}{n} \sum_{i=1}^n \mathrm{JSD}\left( \mathrm{Softmax}(s_{\theta^-}(x_i, y_i)),\, \mathrm{Softmax}(s_\theta(y_i)) \right)$

where s(·) denotes the output speaker embedding, and the JSD measures divergence between the retaining (θ⁻) and randomized (θ) speaker identity distributions.

A spk-ZRF close to 1 indicates that the model generates outputs for forgotten speakers that are as random as those generated with no voice prompt, signifying effective erasure; spk-ZRF near 0 implies deterministic, traceable or non-randomized outputs (i.e., incomplete forgetting) (Kim et al., 27 Jul 2025).

3. Methodologies for Achieving and Measuring spk-ZRF

A spectrum of methods has been developed and benchmarked to achieve high spk-ZRF scores, spanning both generative and discriminative models.

A. Teacher-Guided Unlearning (TGU) and Randomness Induction

The Teacher-Guided Unlearning (TGU) framework specifically targets the zero-shot TTS setting. For each forget prompt (xᶠ, y), the unlearned model θ⁻ is trained to generate outputs not matching the original xᶠ, but instead to mimic a “guidance” signal θ(y) that is random with respect to speaker identity:

The loss for forget set is L₍CFM-forget₎ = 𝔼[ || m ⊙ uₜ(x|𝑥̄) – vₜ(wᶠ, y, x_{ctx}^f; θ⁻) ||² ], with 𝑥̄ = θ(y) as the random guidance sample.
This enforces the model to avoid convergence to any fixed or latent representation for the forgotten speaker, instead generating outputs whose speaker embedding matches the distribution of unprompted generations.

spk-ZRF is then computed as described above to certify the effectiveness of this randomness induction (Kim et al., 27 Jul 2025).

B. Machine Unlearning Techniques in Discriminative Models

In closed-set speaker recognition and keyword spotting tasks, machine unlearning typically relies on:

Negative Gradient (NG): Imposing anti-learning steps to push the model's inference away from the forgotten data, shown to give high unlearning efficacy and computational efficiency (Koudounas et al., 21 May 2025).
Distillation-based (SCRUB/Bad Teaching): Training a student model to minimize similarity on the forget set with respect to an incompetent teacher, while matching predictions on the retain set (Cheng et al., 1 Jun 2025).
Experience Replay or Saliency-Guided Updates: Selectively updating parameters specifically important to forgotten categories.

For each, task utility, efficacy (as measured by membership inference attack lift), and efficiency (runtime speedup) may be combined into a comprehensive measure (GUM), which—along with per-speaker accuracy and downstream privacy audits—constitutes part of the overall spk-ZRF evaluation framework.

4. Experimental Findings and Implications

Empirical studies on state-of-the-art TTS and SLU models indicate:

Methods such as TGU in ZS-TTS yield high spk-ZRF (0.871), indicating effective randomization and privacy for forget speakers, with minimal loss in word error rate (WER) or speaker similarity for remain speakers (Kim et al., 27 Jul 2025).
Negative Gradient unlearning achieves the best trade-off (efficacy, utility, efficiency) on multilingual and multi-speaker benchmarks, outperforming iterative retraining and teacher–student methods in both unlearning speed and ability to mitigate speaker information retrievability (as measured by membership inference attacks) (Koudounas et al., 21 May 2025).
For discriminative models, partial unlearning approaches (e.g., last-layer fine-tuning, or random labeling only on salient parameters) provide controlled forgetting but often suffer decreased global utility compared to full retraining or more sophisticated anti-distillation, especially on high-dimensional or sequence-oriented speech models (Cheng et al., 1 Jun 2025).
Aggressive unlearning (high λ in the unified loss) can decrease overall performance (as speaker and linguistic features are difficult to disentangle cleanly in speech); thus, tuning the forgetting-retention trade-off is necessary for optimal spk-ZRF outcomes (Cheng et al., 1 Jun 2025).

5. Technical Challenges

Several unique challenges arise in achieving robust speaker-zero retrain forgetting:

High-dimensional, time-continuous speech features with overlapping phonetic and speaker information, making disentanglement and selective forgetting harder than in vision or text (Cheng et al., 1 Jun 2025).
Data distributional shifts, accent variability, and entanglement of identity with prosody and channel factors, complicating both erasure and measurement of speaker information (Kim et al., 27 Jul 2025).
Ensuring that the unlearned model’s outputs for forgotten speakers do not collapse to a degenerate or overfit mode, but maintain sufficient diversity and unpredictability (as verified by spk-ZRF or similar randomness measures).

To address these, studies highlight the importance of adversarial discriminators, memory-efficient adapters, hierarchical or multi-level modeling, and modular adaptation strategies (Wang et al., 28 Apr 2024, Wang et al., 2023).

6. Broader Application Context and Future Directions

spk-ZRF has foundational implications for the privacy, safety, and regulatory compliance of voice AI systems:

It validates claims of privacy protection under “right to be forgotten” schemes, offering a quantitative guarantee that a user’s vocal traits can be irretrievably erased from production TTS systems.
In the continual learning paradigm, spk-ZRF offers a rigorous measure of how well models maintain generalization across evolving speaker populations, guiding architecture and algorithm design in multi-speaker, multi-domain deployments (Hemati et al., 2021, Eeckt et al., 2022).
Future enhancements could involve more refined audit metrics, adversarial robustness checks (for resistance to voice re-identification attacks), efficient large-scale spk-ZRF audits, and extension to multi-modal and cross-lingual settings (Cheng et al., 1 Jun 2025, Kim et al., 27 Jul 2025).

A plausible implication is that as voice synthesis and recognition systems scale, combined frameworks measuring spk-ZRF, task utility, and privacy leakage will become standard for deployment certification and ongoing system audit.

Summary Table: spk-ZRF-Related Methods and Metrics

Method/Metric	Principle	Application Domain
spk-ZRF (JSD-based)	Randomness in output voice embeddings for forget speakers	Zero-shot TTS (speaker unlearning)
TGU	Teacher-guided output randomization	ZS-TTS, voice privacy
Negative Gradient (NG)	Gradient reversal on forget set	SLU, speaker/intent unlearning
GUM (Global Unlearning Metric)	Utility × Efficacy × Efficiency	Multi-lingual SLU, speaker unlearning

All of the approaches referenced emphasize efficient, high-fidelity speaker unlearning with robust privacy guarantees, empirically validated via the spk-ZRF metric and complementary privacy/utility criteria.