EASE: Federated Multimodal Unlearning via Entanglement-Aware Anchor Closure

Abstract: Federated Multimodal Learning (FML) trains multimodal models across decentralized clients while keeping their image-text pairs private. However, joint embedding training entangles forgotten knowledge across both modalities and client gradient subspaces, hindering federated unlearning. Previous federated unlearning approaches neither sever the cross-modal reconstruction channel mediated by bilinear coupling nor separate forget-exclusive update directions from those shared with retained clients. We identify an Anchor Principle for federated multimodal contrastive unlearning: forgotten alignments persist through three residual anchors arising from bilinear cross-modal coupling, principal-angle subspace entanglement, and continued federated updates. At the modality level, we show that bilateral displacement of both visual and language branches closes the cross-modal reconstruction channel. Correspondingly, our method addresses subspace entanglement through Cosine--Sine decomposition of client-update subspaces, isolating forget-exclusive directions from retain support. Moreover, we propose a direction-selective Forget Lock that bounds residual drift across rounds. Combining these strategies, we present EASE, an Entanglement-Aware Subspace Excision framework that closes all three anchor channels under a unified design. EASE demonstrates consistent superiority across multiple datasets and unlearning scenarios, for instance, matching the retrain reference to within 0.2 and 4.2 R@1 points on the forget and retain sides under client unlearning on Flickr30K with CLIP-B/32.

• The paper introduces EASE, a framework that leverages bilateral excision to erase cross-modal alignments in federated models.
• It employs gradient subspace decomposition and a forget lock to isolate and remove forget-specific content while preserving retained knowledge.
• Experimental results show significant reduction in the forget–retain gap and privacy leakage, achieving near-retraining performance on diverse datasets.

Entanglement-Aware Federated Multimodal Unlearning: The EASE Framework

Introduction

This paper introduces EASE, a framework for federated unlearning in multimodal joint-embedding models under privacy constraints, with particular focus on removing image-text associations from decentralized models without reconstructing or centralizing raw data. The work addresses the underexplored problem of federated multimodal unlearning, where the cross-modal entanglement of learned representations and client-update subspaces hinders straightforward removal of information pertaining to a targeted concept, sample, or client. EASE analyzes failures of prior work and formalizes why multimodal contrastive settings require more nuanced anchor closure strategies. The method operationalizes these insights by jointly exploiting subspace geometry and modular parameter excision to excise forget-specific content while preserving all retain-support directions.

Figure 1: Overview of EASE. BKE displaces both visual and language branches simultaneously to close the cross-modal reconstruction channel. GSD separates forget-exclusive directions from retain support via principal-angle decomposition. PFL projects the server displacement off the unique subspace and locks each client's drift along it during continued training.

Problem Setting: Multimodal Federated Unlearning and the Anchor Principle

The system considered comprises a central server coordinating $K$ clients, each with private image-text pairs, for federated multimodal learning (FML) using frozen pretrained backbones with trainable LoRA adapters and projectors. The model is trained with a symmetric InfoNCE contrastive alignment loss on both modalities. Unlearning requests can target clients, classes, or samples, corresponding to a partition of the global data into forget and retain sets. The desiderata are retrieval-level unlearning completeness on the forget set, retention integrity on the retain set, and computational efficiency relative to retrain.

The paper diagnoses fundamental failure modes in prior federated unlearning methods. Standard approaches that project out the forget-update subspace or subtract the forget task vector exhibit a persistent forget–retain tradeoff. The authors introduce the Anchor Principle, formalizing three residual mechanisms (anchors) that impede complete unlearning in federated multimodal contrastive models:

• Modality Anchor: Cross-modal bilinear coupling means that erasing only one modality leaves the other as a persistent channel for reconstructing forgotten alignments.
• Unique-Subspace Anchor: Forget-client gradient directions are entangled with those of retained clients; naive projection erases both removable (forget-exclusive) and irremovable (shared) content.
• Temporal Re-anchoring: Continued post-unlearning federated training allows gradual rewriting of previously excised directions.

Extensive experiments demonstrate these anchors are not merely theoretical: current baselines either leak forget information or destructively impact retention on realistic tasks and datasets.

Figure 2: Per-metric comparison on CLIP-B/32 / Flickr30K under three unlearning scenarios. Each sector represents one metric; bars extend outward proportional to the metric value. Our method achieves the lowest Forget R@k while maintaining competitive Retain R@k across all scenarios.

Methodology

Bilateral Knowledge Excision (BKE)

To close the Modality Anchor, EASE applies bilateral displacement to both visual and language parameter branches, as one-sided excision is provably insufficient due to InfoNCE’s bilinear form. This bilateral excision removes the reconstructive influence of the untouched branch, eliminating the primary gradient drive toward the forgotten alignment. The paper provides analytic expressions of the gradient field, formalizing the unique impact of simultaneous projection in contrastive dual-encoder settings and introducing a quantitative alignment-residual metric $\rho$ to measure anchor closure.

Gradient Subspace Decomposition (GSD)

For the Unique-Subspace Anchor, EASE constructs per-client update histories and applies SVD to extract principal directions for both the forget and retain subspaces. The critical innovation is the use of Cosine–Sine (CS) decomposition between forget and retain subspaces: each canonical direction in the forget subspace is classified by its principal angle as either removable (orthogonal to the retain subspace, zero entanglement) or must-preserve (high entanglement, significant overlap). This enables excision of only those directions that are minimal support for retain behavior, systematically controlling the risk of collateral damage and removing only the unique anchor energy.

Projection with Forget Lock (PFL)

Temporal Re-anchoring is addressed by pairing server-side projection with a direction-selective Forget Lock regularizer on each client. At each round, the server projects updated parameters to guarantee continued excision along removable anchors, while clients add a quadratic penalty confining drift specifically in that subspace during local SGD. Analytical results bound residual re-anchoring inversely with lock strength $\alpha$, ensuring stable anchor closure across arbitrary numbers of post-unlearning rounds.

Experimental Results

EASE is empirically validated on Flickr30K, MS COCO, and TextCaps using CLIP (B/32, L/14) and SigLIP multimodal backbones. Evaluations include retrieval recall (R@k), alignment residuals, membership inference attacks (MIA/LiRA), and per-pair embedding similarities.

EASE consistently achieves near-retrain recall on both forget and retain sets across client, class, and sample unlearning, reducing the forget–retain gap by factors of $3\times$–$11\times$ against the closest federated baselines. On Flickr30K with CLIP-B/32, for example, EASE matches retrain within $0.2$ and $4.2$ R@1 on forget and retain, where the closest baseline gap exceeds $60$ points on forget and $5$ points on retain. Ablations confirm necessity: removal of bilateral excision or entanglement-aware subspace isolation reopens the Modality Anchor or severely hurts retained knowledge.

Figure 3: Per-pair image--text cosine similarity under the original, our method, and retrain. Top: forget pairs, where our method shifts toward the retrain reference. Bottom: retain pairs, where the three distributions overlap.

Privacy metrics show that EASE reduces both shadow-model MIA and low-FPR LiRA success rates on the forget set down to retrain levels, indicating that anchor closure is realized not just at the embedding level but under realistic attacker models. Communication cost analyses demonstrate practical efficiency: EASE achieves unlearning quality comparable to retrain while using $20$–$\rho$0 of the communication rounds.

(Figures 3–5)

Figure 4: Sensitivity of EASE to the entanglement threshold $\rho$1, demonstrating robust recall performance across a range of anchor–support boundaries.

Figure 5: Sensitivity of EASE to the Forget Lock strength $\rho$2, showing that effective re-anchoring suppression does not depend on brittle hyperparameter tuning.

Figure 6: Additional robustness results for EASE across federation size $\rho$3 and heterogeneity $\rho$4.

Theoretical and Practical Implications

The strongest claims are:

• EASE is the first to provide coordinated closure of all three anchor channels for federated multimodal models, substantially reducing the previously irreducible forget–retain tradeoff.
• The formal Anchor Principle articulates failure modes and their geometric loci in federated unlearning with cross-modal contrastive models. Disabling any channel admits measurable leakage.
• Bilateral subspace projection, supported by joint singular vector and principal angle analysis, generalizes one-shot single-modality projection to the entangled, high-dimensional, and ongoing-training regimes characteristic of modern FML.

These methods provide a geometric framework for future extensions to generative or late-fusion multimodal architectures and indicate that subspace-aware, branch-symmetric excision is operative for preservation–deletion tradeoffs in privacy-centric distributed ML systems.

Conclusion

EASE introduces an entanglement- and anchor-aware paradigm for federated multimodal unlearning, delivering order-of-magnitude advances in recall and privacy leakage benchmarks relative to all tested baselines. The methods and analyses clarify structural challenges posed by bilinear coupling and subspace entanglement and show that direction-selective, bilateral projection is necessary for robust privacy guarantees under continued distributed deployment. Extension to more complex model families (generative or MLLM) and more elaborate privacy threat models is a compelling direction for future work (2605.00733).