Asynchronous Federated Unlearning with Invariance Calibration for Medical Imaging

Abstract: Federated Unlearning (FU) is an emerging paradigm in Federated Learning (FL) that enables participating clients to fully remove their contributions from a trained global model, driven by data protection regulations that mandate the right to be forgotten. However, existing FU methods mostly rely on synchronous coordination. This requirement forces the entire federation to halt and wait for stragglers to complete erasure, creating significant delays due to device heterogeneity. Furthermore, these methods often face the problem that the influence of erased data is merely suppressed temporarily and resurfaces during subsequent training, rather than being genuinely removed. To overcome these limitations, this paper proposes Asynchronous Federated Unlearning with Invariance Calibration (AFU-IC), a novel framework for medical imaging that decouples the erasure process from the global training workflow. This enables the target client to perform unlearning asynchronously without interrupting global training. Meanwhile, a server-side invariance calibration mechanism prevents the model from relearning the erased data. Extensive experiments on three medical benchmarks demonstrate that AFU-IC achieves unlearning efficacy and model fidelity comparable to gold-standard retraining while significantly reducing wall-clock latency compared to synchronous baselines. AFU-IC ensures efficient, compliant and reliable FL in cross-silo medical environments.

• The paper presents the AFU-IC framework, which decouples client-side erasure from global training using asynchronous local updates and server-side invariance calibration.
• It demonstrates a 4x speedup over synchronous methods while maintaining high clean accuracy and effective backdoor removal across multiple medical imaging benchmarks.
• The experimental results confirm that AFU-IC efficiently complies with data deletion requests without interrupting ongoing federated training, ensuring robust knowledge retention.

Asynchronous Federated Unlearning with Invariance Calibration for Medical Imaging

Motivation and Problem Context

Federated learning (FL) is widely adopted for collaborative medical image analysis, enabling multiple institutions to train deep models without sharing raw data. However, data protection regulations such as GDPR and CCPA mandate the "right to be forgotten," requiring the capability to retroactively erase specific clients’ contributions from a trained global model. Standard federated unlearning (FU) techniques commonly adopt synchronous protocols, introducing straggler bottlenecks and high latency due to device heterogeneity. More critically, these methods often fail to achieve genuine erasure—erased influences are only temporarily suppressed and can re-emerge during subsequent training rounds. This paper proposes the Asynchronous Federated Unlearning with Invariance Calibration (AFU-IC) framework, addressing both efficiency and robustness in unlearning for cross-silo medical FL scenarios (2604.26809).

AFU-IC Framework and Methodology

AFU-IC decouples the client-side erasure process from the global training pipeline. The architecture features a two-phase protocol: asynchronous local erasure, followed by server-side invariance calibration. This design enables the target client to perform unlearning independently, without imposing system-wide synchronization constraints or interrupting training for retained clients.

Figure 1: Overview of the AFU-IC framework.

In Phase I, the target client forms a reference model by subtracting its previous contribution from the latest global model, anchoring subsequent parameter updates. Local projected gradient ascent (PGA) maximizes loss on the target data within a constrained ball around the reference, preventing catastrophic forgetting of retained knowledge.

Phase II addresses the insufficiency of standard gradient-based erasure, which performs masking rather than structural elimination. A server-side invariance calibration procedure minimizes the KL divergence between predictions on clean examples and their augmented variants, ensuring the model’s output invariance to non-robust features associated with the unlearned client. This regularization mathematically decouples decision logic from erased information and robustly prevents model reverting when training continues with the retained clients.

Empirical Evaluation

The evaluation spans three medical imaging benchmarks (OASIS, PathMNIST, OrganAMNIST), using both behavioral (backdoor accuracy, BA) and structural ($L_2$ distance to retrain oracle) efficacy metrics, as well as fidelity (clean accuracy, CA) and wall-clock efficiency. The experimental protocol injects backdoors as a traceable fingerprint for unlearning completeness and simulates cross-silo non-IID heterogeneity.

AFU-IC achieves erasure efficacy and clean accuracy highly comparable to full retraining, while significantly reducing unlearning latency. For OASIS, AFU-IC completes the process in 295 seconds—a substantial improvement over both retraining (5,420 seconds) and synchronous FU baselines. BA after unlearning with AFU-IC approaches the retrain baseline, while $L_2$ distances post-calibration confirm genuine movement toward the retrained oracle.

Figure 2: Clean accuracy and backdoor accuracy of the AFU-IC and fully retrained model with respect to the number of FL post-learning rounds in each dataset for $N=5$ clients.

AFU-IC preserves main-task utility during and after unlearning. The evolution of CA shows that any drop induced by PGA quickly recovers, with final CA exceeding the retrained baseline—demonstrating knowledge retention on retained data with cost-effective computation.

Figure 3: Evolution of CA throughout the federated training process.

Ablation studies confirm several key properties:

• Necessity of Invariance Calibration: Removing the calibration regularizer leads to incomplete backdoor removal, revealing masking rather than genuine erasure.
• Asynchrony Advantage: Asynchronous execution provides a 4x speedup over synchronous equivalents without loss in unlearning efficacy or fidelity, effectively eliminating the straggler bottleneck.
• Stability and Scalability: AFU-IC is robust to extreme non-IID settings and maintains performance across a range of federation sizes.

Theoretical and Practical Implications

The introduction of server-side invariance calibration establishes a rigorous mechanism for feature-level unlearning. By aligning outputs on original and augmented inputs, the model is regularized not to rely on client-specific, non-robust feature correlations. This approach closes a substantive gap in the unlearning literature regarding model reverting and demonstrates that federated unlearning can be both robust and non-blocking.

Practically, AFU-IC operationalizes regulatory mandates in heterogeneous medical FL environments. Its decoupled protocol guarantees compliance with deletion requests without service downtime—critical for high-availability clinical systems.

Theoretically, the KL-based calibration objective and asynchronous infrastructure suggest new research directions for “structural unlearning” in non-stationary, federated, and foundation model settings.

Outlook and Future Directions

AFU-IC sets a precedent for asynchronous, structure-aware FU in realistic cross-silo deployments. Extensions can include privacy-preserving calibration (e.g., differential privacy in server-side regularization), adaptation to large foundation models with heterogeneous client computation, and application beyond vision to multimodal federated unlearning. The framework may also inspire dynamic unlearning for continual learning systems and robustness against adversarial erasure requests. Considering the rise of complex medical AI consortia, AFU-IC’s scalable and robust methodology constitutes an important contribution to privacy guarantees in collaborative healthcare AI.

Conclusion

AFU-IC provides an efficient, robust, and scalable solution for federated unlearning in medical imaging contexts, combining asynchronous protocol execution and invariance-based regularization to ensure both genuine erasure and system efficiency. Experimental and theoretical analysis demonstrate its superiority to prior synchronous and masking-driven techniques. This framework advances both the state of the art in FL compliance and the theory of robust machine unlearning (2604.26809).