Teacher-Guided Unlearning (TGU)

• Teacher-Guided Unlearning (TGU) is a set of methodologies that uses teacher-student frameworks to selectively erase targeted data while maintaining model utility.
• It employs dual-teacher architectures, including competent and stochastic teachers, to strategically guide the student model's unlearning process.
• TGU methods offer theoretical guarantees, practical metrics, and applications across modalities like vision, text, and speech, supporting privacy regulations.

Teacher-Guided Unlearning (TGU) refers to a class of machine unlearning methodologies in which the knowledge transfer between ‘teacher’ and ‘student’ models is strategically leveraged to selectively erase information associated with a chosen subset of training data or behaviors, while preserving generalization and utility elsewhere. These approaches are characterized by direct manipulation of the post-training fine-tuning or distillation stage, often without resorting to full retraining. Across modalities—vision, language, federated, and audio—the guiding principle is that a teacher (competent, incompetent, or neutral/stochastic) provides signals that steer the student to ‘forget’ targeted data by imitating non-informative or randomized outputs, while reinforcing retention of desirable knowledge via conventional teacher guidance. TGU systems are increasingly deployed to meet privacy regulations, such as the right to be forgotten, and to mitigate membership inference or copyright risks, with metrics and theoretical guarantees tailored to post-hoc unlearning assessment.

1. Foundational Architectures: Dual-Teacher Frameworks and Knowledge Transfer

The formative method of TGU, introduced in "Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher" (Chundawat et al., 2022), formalizes a dual-teacher framework in which two teachers guide the student during a fine-tuning stage:

• Competent Teacher $T_s$: The original, fully trained model providing informative, correct predictions on the retained data $D_r$.
• Incompetent Teacher $T_d$: A randomly initialized or minimally trained model, generating essentially random outputs on the forget set $D_f$.

The TGU loss for a sample $x$ with an unlearning label $l_u$ (0 for retain, 1 for forget) is:

$$L(x, l_u) = (1 - l_u) \cdot \text{KL}(T_s(x) || S(x)) + l_u \cdot \text{KL}(T_d(x) || S(x))$$

This formulation enforces selective knowledge retention and targeted erasure within the student model $S$. The guiding intuition is that output distributions encode learned information; aligning the student’s response on $D_f$ to randomized teacher outputs effectively disrupts memorized associations, while mimicking $T_s$ on $D_r$ preserves utility.

Teacher-guided knowledge transfer is also adapted in federated contexts ("Class-wise Federated Unlearning" (Li et al., 2023)) via teacher-generated, debiased “knowledge-free” label distributions, allowing for class-wise or sample-wise unlearning without compromising global performance.

2. Methodological Adaptations: Stochastic, Debiased, and Vocabulary-Agnostic Teachers

Several variants and generalizations of the foundational TGU paradigm have emerged:

• Stochastic Teacher Networks: In "Machine Unlearning Methodology based on Stochastic Teacher Network" (Zhang et al., 2023), a randomly initialized (unbiased) teacher is employed to provide a neutral target distribution for the student, ensuring the output over $D_f$ approaches an uninformed baseline via KL divergence minimization. This is effectuated in a two-stage procedure: (A) knowledge erasure (matching $D_f$ outputs to random teacher), and (B) model reconstruction (distillation from the original teacher for $D_r$).
• Debiased Teacher-Student Memory Generation: In federated unlearning, new “memories” for the forget set are synthesized by averaging outputs of untrained teachers and applying a debiasing vector to nullify any residual class information before retraining the student (Li et al., 2023).
• Vocabulary-Agnostic Alignment: For text models with mismatched teacher/student vocabularies, “Vocabulary-agnostic Teacher Guided LLMing” (Shin et al., 24 Mar 2025) introduces fine-grained mapping between disjoint tokenizations based on character offsets, and an adaptive loss-reweighting scheme using the teacher’s token-level loss as a guidance signal.

A plausible implication is that TGU is widely extensible across architectures, including CNNs, Transformers, LSTMs, and LLMs—requiring only that an appropriate teacher output distribution (randomized, debiased, or expert) can be synthesized.

3. Formal Evaluation Measures and Theoretical Guarantees

The evaluation of TGU departs from older unlearning metrics by proposing retraining-free, distributional alignment objectives; for instance, the Zero Retrain Forgetting (ZRF) score (Chundawat et al., 2022) defined as:

$$ZRF = 1 - \frac{1}{n_f} \sum_{i=1}^{n_f} \text{JS}(M(x_i), T_d(x_i))$$

with Jensen–Shannon divergence quantifying the similarity between the unlearned model $M$ and the random teacher’s output distributions. A similar principle underlies the speaker-ZRF (spk-ZRF) in ZS-TTS for assessing the indistinguishability of speaker videos after unlearning (Kim et al., 27 Jul 2025).

Theoretical analysis in GUARD (Ma et al., 12 Jun 2025) demonstrates that adaptive reweighting of forget samples according to their proxy attribution (gradient alignment to retained data) provably lowers the utility loss (sacrifice rate) on $D_r$ without diminishing forgetting effectiveness on $D_f$:

$$\rho^{GA} - \rho^{GUARD} = \frac{\kappa^2 + \sigma_\kappa^2}{\tau \|\bar{g}_f\|_2^2} + O(\delta^2)$$

where $\omega_i^{GUARD}$ are forget weights assigned by a temperature-controlled softmax over alignment scores.

These advances position TGU as both empirically and theoretically robust, offering guarantees on unlearning completeness (as measured by the spk-ZRF, ZRF, or MUSE-based metrics) and utility preservation.

4. Modality-Specific Implementations and Impact

Teacher-guided unlearning strategies have been systematized for diverse application domains:

Modality	Teacher Role	Illustrative Work
Vision	Randomized or debiased predictions for $D_f$	(Chundawat et al., 2022, Zhang et al., 2023)
Federated	Synthesized “knowledge-free” memory generation	(Li et al., 2023)
Text (LLM)	Supervision via negative sampling, mean teacher	(Yao et al., 2023, Klochkov, 18 Apr 2025)
Tokenization	Token-level lexical alignment and loss mapping	(Shin et al., 24 Mar 2025)
Speech (TTS)	Randomized speech synthesis for forgotten IDs	(Kim et al., 27 Jul 2025)

In Zero-Shot Text-to-Speech (Kim et al., 27 Jul 2025), TGU is realized by having the unlearned model approximate teacher outputs that are conditioned only on text, ensuring that prompts for forgotten speakers yield random, non-identifiable voices. The corresponding spk-ZRF metric quantitatively evaluates the randomness of speaker distributions in outputs, confirming effective erasure of the target speaker’s identity.

For LLMs, gradient ascent on undesirable exemplars combined with random mismatch and KL losses against the original model aligns the model output away from harmful responses, reducing the harmful rate with minimal utility degradation (Yao et al., 2023).

In federated or online learning scenarios, TGU operates via public teacher signals—such as identification of unlearning times or selection of data to unlearn—allowing both passive (noise-injection) and active (descent-to-delete) learner-unlearner protocols with formal Rényi divergence guarantees and regret bounds (Hu et al., 13 May 2025).

5. Limitations, Data-Level Factors, and Enhancements

Although TGU frameworks are general, several empirical findings highlight sensitivity to data-level and architectural factors. "Learning-Time Encoding Shapes Unlearning in LLMs" (Wu et al., 18 Jun 2025) demonstrates that factual knowledge encoded via diverse paraphrases is more amenable to selective unlearning; in contrast, entanglement of target information within mixed text chunks impedes precise forgetting, even under guided gradient approaches.

Limitations of current implementations include:

• Decreased scalability to complex architectures in some stochastic/distillation-based interfaces (Zhang et al., 2023).
• Potential collateral loss of “neighboring” or similar information, especially when attribution or disentanglement is imperfect (Ma et al., 12 Jun 2025, Kim et al., 27 Jul 2025).
• Sensitivity to hyperparameter settings (e.g., temperature in teacher loss aggregations, $\lambda$ for loss combination).

A plausible implication is that improved disentanglement during training, adaptive weight assignment, and reinforcement of data boundaries at learning time will likely enhance future TGU systems’ precision and selectivity.

6. Privacy, Regulatory, and Real-World Contexts

Teacher-guided unlearning is motivated by an array of regulatory, safety, and ethical concerns:

• GDPR, CCPA, and Privacy: TGU enables post-hoc compliance without retraining or utility loss, by facilitating erasure of personally identifiable or sensitive data upon request (Chundawat et al., 2022, Ma et al., 12 Jun 2025).
• Copyright and Misuse Prevention: In LLMs and TTS systems, TGU allows for targeted forgetting of outputs, sentences, or voices to enforce opt-out or prevent replication of unauthorized content (Yao et al., 2023, Kim et al., 27 Jul 2025).
• Attack Mitigation: By removing the influence of data known to be poisoned or attacked, TGU increases the resilience of deployed models against backdoor or membership inference exploits (Li et al., 2023, Zhang et al., 2023).

Because the teacher (competent or incompetent) serves as an explicit or implicit regulator, TGU is naturally extensible to scenarios where unlearning guidance must be externally validated or where auditability of what has been “forgotten” is essential.

7. Prospects and Emerging Directions

Future inquiry into TGU is focusing on:

• Generalizing teacher-guided unlearning to continual, federated, and streaming settings, with strong post-hoc guarantees (Hu et al., 13 May 2025).
• Developing attribution and structure-aware training regimes to optimize later unlearning selectivity (Wu et al., 18 Jun 2025, Ma et al., 12 Jun 2025).
• Advancing evaluation metrics to capture fine-grained unlearning-utility trade-offs, particularly for large generative models and entangled multimodal representations (Kim et al., 27 Jul 2025, Klochkov, 18 Apr 2025).
• Investigating improvements in scaling and efficiency for complex, large-scale architectures, and extending vocabulary-/structure-agnostic TGU to broader language and cross-modal tasks (Shin et al., 24 Mar 2025).

In summary, TGU has become a central paradigm for post-hoc machine unlearning, unifying selective memory erasure and knowledge retention via teacher-mediated guidance. It is supported by rigorous theoretical analysis, extensive empirical validation, and practical alignment with ethical, privacy, and regulatory requirements, across a range of data modalities and learning architectures.