Continual Speaker Identity Unlearning with Minimal Interference

Abstract: Machine unlearning removes designated concepts or knowledge from pre-trained models. Recent work has extended this paradigm to speaker identity unlearning in zero-shot text-to-speech (ZS-TTS), the task of selectively erasing a model's ability to replicate a speaker's voice. Existing methods, however, quietly assume all unlearning requests arrive at once; an unrealistic assumption, since privacy-motivated removals arrive sequentially over time. We show this assumption breaks state-of-the-art methods: unlearning each new speaker fully revives previously unlearned speakers, reintroducing the very privacy risk unlearning was meant to eliminate. We present Cumulative ORThogonal Identity Suppression (CORTIS), the first framework for continual speaker identity unlearning in ZS-TTS that requires no access to previously-unlearned speaker data. CORTIS combines Fisher-information-based parameter masking, which localizes updates to speaker-relevant weights, with orthogonal projection against subspaces spanned by prior unlearning updates. With VoiceBox, CORTIS unlearns each requested speaker while keeping previously unlearned speakers forgotten across long request sequences, substantially outperforming sequential application of prior methods. The demo is available at https://cumulativeortis.github.io/ .

• The paper introduces CORTIS, a method combining Fisher-informed parameter localization with orthogonal subspace projection to prevent catastrophic re-learning in ZS-TTS.
• It details experimental evaluation on VoiceBox, demonstrating durable speaker identity suppression (SIM < 0.18) over sequential unlearning requests with constant per-request computational cost.
• The framework meets regulatory RTBF compliance by avoiding retention of removed speaker data while maintaining synthesis quality for remaining speakers.

Continual Speaker Identity Unlearning in Zero-Shot Text-to-Speech: Summary and Analysis

Problem Setting and Motivation

The paper "Continual Speaker Identity Unlearning with Minimal Interference" (2605.25962) rigorously formalizes the problem of continual speaker identity unlearning in zero-shot text-to-speech (ZS-TTS) models under real-world deployment constraints, motivated by privacy regulations such as GDPR and CCPA. Unlike joint unlearning—where the entire set of removal requests is processed simultaneously—continual unlearning requires sequential handling of identity removal requests, with the additional constraint that forget-speaker data must not be retained after each request. Under this regime, prior speaker unlearning methods (such as Teacher-Guided Unlearning and Sample-Guided Unlearning) suffer catastrophic re-learning: new unlearning steps inadvertently restore previously erased speakers, revoking guaranteed privacy.

Figure 1: Naïve sequential application of prior unlearning methods reverts previously removed speaker identities.

The central challenge is to preserve suppression of all previously unlearned speakers over a sequence of removal requests, without revisiting their data or incurring quadratic training costs. Furthermore, generic continual learning methods, which typically regularize parameter update norms or restrict fine-tuning to salient parameters, fail to protect against identity reversion due to the highly entangled representations in modern ZS-TTS architectures.

Methodology: CORTIS Framework

The CORTIS (Cumulative ORThogonal Identity Suppression) framework introduces a dual-mechanism approach for continual identity unlearning that is compliant with deployment constraints:

1. Contrastive Parameter Localization: CORTIS computes the diagonal Fisher Information Matrix over the forget set and constructs a saliency map that assigns parameter-level importance contrasting current forget speakers against previously forgotten speakers and the remain set. By masking updates to the top-$k\%$ saliency-ranked parameters, CORTIS ensures unlearning is localized, avoiding interference with parameters important for retention or prior unlearning.
2. Orthogonal Subspace Projection: Inspired by gradient memory methods in continual learning (OGD, GPM), CORTIS accumulates per-speaker gradient subspaces via truncated SVD, merging them into a fixed-rank orthonormal basis. Each new unlearning request projects optimizer updates onto the orthogonal complement of the cumulative forget subspace, ensuring that updates cannot drift in directions previously used to suppress earlier speakers. This subspace protection is applied only within the localized parameter region.

   Figure 2: CORTIS framework: (a) contrastive Fisher saliency mask localizes trainable parameters for current forget request; (b) weight updates are projected away from cumulative prior forget directions.

This approach uniquely enables durable suppression of all previously erased identities, even as new requests arrive, without retention of prior forget data or checkpoints.

Experimental Evaluation

CORTIS is evaluated on VoiceBox—a state-of-the-art flow-matching ZS-TTS backbone—using sequential speaker removal requests sampled from LibriHeavy and validated on LibriSpeech test-clean. Performance is measured via Word Error Rate (WER) for synthesis fidelity and Speaker Similarity (SIM) for identity leakage.

Figure 3: Empirical speaker similarity bounds: same-speaker lower bound (0.46), different-speaker upper bound (0.32). These guide interpretation of retention and forgetting success.

Figure 4: Forget speakers are well-separated in embedding space—minimal pairwise cosine similarity between mean speaker embeddings.

Figure 5: Per-speaker saliency masks from CORTIS exhibit low pairwise Jaccard overlap, confirming parameter-level localization and independence across requests.

Results demonstrate that CORTIS is the only method that maintains the suppression of all previously forgotten speakers (SIM consistently $<0.18$ for each erased speaker after five sequential requests), with minimal impact on synthesis quality for retain speakers and at constant per-request computational cost. Baselines either suffer catastrophic re-learning or sacrifice model utility.

Ablation studies confirm the orthogonal projection step is necessary to prevent direction-level reversion, while the Fisher saliency mask is essential for parameter-level isolation. The method is robust to mask fraction and projection rank.

Practical and Theoretical Implications

The findings have several implications:

• Privacy Durability: CORTIS provides a mathematically and empirically substantiated mechanism for durable identity erasure in deployed ZS-TTS systems, directly supporting regulatory requirements for RTBF compliance without data retention paradox.
• Deployment Scalability: The method achieves constant computational overhead per request, unlike naive cumulative retraining or model merging baselines that linearly or quadratically scale in resource cost and are non-compliant with RTBF.
• Architectural Generality: While evaluated on VoiceBox, the core mechanisms—parameter saliency and cumulative subspace projection—are architecture-agnostic, suggesting potential adaptation to diffusion-based or codec-based generative speech models as next steps.
• Continual Unlearning Theory: The diagnosis of catastrophic re-learning in ZS-TTS calls for novel continual unlearning principles that go beyond simple regularization; the contrastive masking and subspace projection applied here may generalize to other identity-centric generative domains (image, language, biometrics).
• Limitations and Future Directions: Adversarial robustness, prompt engineering attacks, and cross-architecture validation are natural extensions. Additionally, integrating continual unlearning with generative replay or adapter-based isolation strategies may further enhance durability.

Conclusion

This paper establishes continual speaker identity unlearning as a distinct and practically significant challenge in ZS-TTS deployment. By theoretically diagnosing catastrophic re-learning and empirically validating CORTIS—a method combining Fisher-informed parameter masking and cumulative subspace projection—the research delivers a framework for minimally interfering, durable, and scalable identity suppression. The practical deployment of ZS-TTS in a privacy-regulated environment hinges on the adoption of such techniques, and further exploration into adversarial robustness and broader generative domains is warranted.