Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation

Abstract: With the increasing importance of data privacy and security, federated unlearning has emerged as a novel research field dedicated to ensuring that federated learning models no longer retain or leak relevant information once specific data has been deleted. In this paper, to the best of our knowledge, we propose the first complete pipeline for federated unlearning, which includes a federated unlearning approach and an evaluation framework. Our proposed federated unlearning approach ensures high efficiency and model accuracy without the need to store historical data.It effectively leverages the knowledge distillation model alongside various optimization mechanisms. Moreover, we propose a framework named Skyeye to visualize the forgetting capacity of federated unlearning models. It utilizes the federated unlearning model as the classifier integrated into a Generative Adversarial Network (GAN). Afterward, both the classifier and discriminator guide the generator in generating samples. Throughout this process, the generator learns from the classifier's knowledge. The generator then visualizes this knowledge through sample generation. Finally, the model's forgetting capability is evaluated based on the relevance between the deleted data and the generated samples. Comprehensive experiments are conducted to illustrate the effectiveness of the proposed federated unlearning approach and the corresponding evaluation framework.

• The paper proposes a federated unlearning strategy leveraging an incompetent teacher model, attention map alignment, and optimization techniques.
• It achieves near-zero backdoor and membership inference attack success rates while preserving model performance on non-deleted data.
• The Skyeye framework provides a visual audit through a GAN setup, enabling empirical evaluation of the unlearning effectiveness.

Efficient Federated Unlearning and Visible Evaluation via Skyeye

Introduction

"Forgetting to Witness: Efficient Federated Unlearning and Its Visible Evaluation" (2604.04800) addresses the challenge of complying with user data deletion requests in Federated Learning (FL) environments—a requirement imposed by regulations like GDPR and CCPA. The paper proposes an efficient federated unlearning approach that does not require costly retraining or storage of historical data. In addition, the authors introduce the Skyeye framework for visible evaluation of unlearning effectiveness, leveraging generative adversarial neural architectures to directly visualize the scope and completeness of data removal at the model level.

Federated Unlearning Methodology

The proposed federated unlearning approach combines knowledge distillation using an incompetent teacher model, attention map alignment for internal representation control, and optimization mechanisms targeting practical viability in real-world FL systems.

When a user requests erasure, the model deletion execution unfolds as follows:

• Initialization: The current global model is assigned as the student. The teacher model is an incompetent instance lacking knowledge on the data for deletion.
• Distillation: Only deleted data are passed through both models. The student is trained to match the incompetent teacher's (essentially uninformative) predictions on these samples, enforcing an unlearning effect.
• Attention Map Alignment: Internal attention maps between the two models are aligned to prevent subtle information leakage and ensure robust removal from intermediate feature representations.
• Optimization Enhancements: Bounded loss and learning rate decay prevent loss explosion during unlearning, while regularization terms mitigate catastrophic forgetting. A collaborative post-unlearning fine-tuning protocol ensures model utility on retained data.

  Figure 1: Diagrammatic overview of the federated unlearning execution, including model initialization, loss computation, and efficiency mechanisms.

This approach explicitly does not require historical data or gradients, in contrast to many prior unlearning protocols, and maintains high model utility post-unlearning by targeting both output and intermediate representations.

Visible Evaluation: The Skyeye Framework

One persistent limitation in machine unlearning research is the absence of direct, interpretable evaluation methodologies. Common audit strategies—such as measuring output similarity to retrained models or analyzing attack success rates—are indirect and often subject to statistical uncertainty.

Skyeye provides an explicit, visual method for evaluation. The framework integrates the unlearned model as the classifier in a conditional GAN setup. The generator is trained with guidance from both the classifier and the discriminator, attempting to produce synthetic data that satisfies the classifier's criteria for each target class label.

Figure 2: Illustration of Skyeye's evaluation flow, including data deletion requests, unlearning execution, GAN integration, and decision-making via generation analysis.

The key intuition is that if unlearning has succeeded, the generator should be unable to synthesize realistic samples for forgotten categories. By inspecting the generator's outputs, practitioners and auditors can visually and empirically assess whether any residual data dependence remains.

Experimental Evaluation and Key Findings

Comprehensive empirical evaluations validate the proposed approach across MNIST, CIFAR-10, CIFAR-100, and AT&T datasets using LeNet and ResNet architectures. The paper benchmarks against retraining-from-scratch and other teacher-based unlearning baselines.

Sample Unlearning

On all datasets, the method achieves a backdoor attack success rate and membership inference attack success rate comparable to retraining from scratch (e.g., near 0% attack success), while incurring negligible utility degradation (Fig. 3–5). The approach consistently outperforms state-of-the-art teacher-based and attention-based baselines in both utility and resilience metrics.

Figure 3: Unlearning effectiveness on MNIST measured by accuracy, backdoor attack, and membership inference attack rates versus several baselines.

Category Unlearning

Experimental results demonstrate that single-category and 20%-category forgetting can be enforced with post-unlearning accuracy on non-forgotten data matching or closely approaching retrained models, while accuracy for forgotten categories drops to zero.

Distributional Similarity

Skyeye's generator outputs further demonstrate that after unlearning, the generative model cannot synthesize convincing samples for forgotten classes, but easily generates for retained data (Figs. 9–12). Quantitative L2 and JSD metrics between predictions of the unlearned and retrained model also converge (Figs. 6–8).

Internal Representation

Attention map discrepancy is minimized with the proposed attention-alignment mechanism, further reducing the risk of reconstruction or memorization attacks from internal features.

Ablations and Mechanism Analysis

Ablation studies on hyperparameters for the learning rate, loss bounding, and regularization confirm the necessity of these components for ensuring that unlearning is both effective and does not cause catastrophic performance collapse or over-unlearning.

Figure 4: Generator ablation: outputs with only discriminator loss (left) and only classifier loss (right) on MNIST. Hybrid loss is necessary for meaningful unlearning evaluation.

Implications and Future Directions

Practically, these results indicate that it is possible to deploy auditable, efficient, and accurate federated unlearning at scale without the storage of past data or full retraining. The Skyeye paradigm generalizes to any context where unlearning must be certified or inspected, providing means for user-side verification and regulatory compliance.

Theoretically, explicit attention alignment addresses subtler forms of data retention previously overlooked in unlearning literature. The generative evaluation method also exposes latent model dependencies not visible through standard metrics.

Future research directions include extending this methodology to more complex modalities (text, speech), integrating cryptographic proofs for audit trails, and adapting Skyeye for online or continual unlearning use cases.

Figure 5: Visual comparison of generated samples for forgotten MNIST digits before and after unlearning, verifying comprehensive erasure.

Conclusion

The paper provides a robust solution for federated unlearning, combining efficient knowledge distillation with internal representation control and proposing the first direct, generative evaluation framework for unlearning effectiveness. The methodology achieves strong empirical performance, preserves model utility, and provides clear, interpretable auditability. The Skyeye framework establishes a blueprint for future research and deployment of accountable, user-aligned unlearning in privacy-sensitive systems.