Fed-FBD: Federated Functional Block Diversification for Isolation, Privacy, and Surgical Unlearning

Abstract: Federated learning (FL) enables collaborative model training without sharing raw patient data, but standard approaches such as FedAvg treat each client as a black box and provide no mechanism for isolating an adversarial contributor, auditing per-client influence, or honoring a departed participant's right to be forgotten. We present Fed-FBD (Federated Functional Block Diversification), a modular federated architecture that decomposes a ResNet backbone into six functional blocks (the stem, four residual groups, and the classification head) and maintains a warehouse of N color variants, each assembled from independently tracked and contributor-stamped blocks. Fed-FBD provides three capabilities absent in FedAvg: (i) architecturally guaranteed block-level isolation, so that an adversarial or mislabelled client cannot contaminate the clean colous; (ii) privacy-by-design, where membership inference advantage is already indistinguishable from chance before any privacy mechanism is applied; and (iii) surgical machine unlearning of a departed participant's contribution at sub-second cost and without retraining. Experiments on six MedMNIST-2D datasets, PathMNIST at 224x224, and CIFAR-10 show that Fed-FBD trades a modest 0.3%-3.1% IID accuracy gap on the adequately sized datasets for these guarantees, remains within 0.8%-4.0% of FedAvg at Dirichlet alpha=1.0 on three of four datasets, and confines all six adversarial attacks we study to the poisoned client's own blocks with at most +/-0.01 AUC drift on the clean colors.

• The paper introduces a novel block-diversification approach that re-parameterizes neural networks into independently updated functional blocks to ensure isolation, privacy, and surgical unlearning.
• It employs a strict shipping/request plan that limits client block updates, preventing cross-client interference and enabling constant-time, precise unlearning with minimal accuracy loss.
• Empirical results on medical imaging benchmarks show near FedAvg performance while robustly defending against adversarial attacks in both IID and non-IID settings.

Fed-FBD: Federated Functional Block Diversification for Isolation, Privacy, and Surgical Unlearning

Introduction and Motivation

Federated learning (FL) has matured as a framework for decentralized training of neural networks on sensitive or siloed medical data, allowing collaborative model construction without centralized data pooling. However, canonical FL methods—FedAvg, FedProx, and FedNova—suffer from an absence of architectural mechanisms for robust contributor isolation, principled client-level privacy, or efficient machine unlearning. Specifically, the classical parameter sharing approach entangles updates from all clients, leaving the global model vulnerable to adversarial contamination, unable to audit or sever the influence of particular clients, and forced to resort to expensive retraining operations to honor withdrawal requests.

Fed-FBD introduces a radical re-parameterization: it decomposes a ResNet into discrete "functional blocks" and tracks a warehouse of $N$ complete model "colours," each colour comprising independently trained and contributor-stamped blocks. Ownership is tightly controlled at the block level such that isolation, privacy, and unlearning become direct algebraic operations rather than costly approximations or external bolt-ons.

Figure 1: Fed-FBD system overview: each client ships and requests specific block weights to/from a warehouse organized as a $B{\times}N$ grid of block positions (in, L1–L4, out) by "colours". The shipping plan restricts client block writing to specific colours, guaranteeing architectural isolation.

Methodological Framework

Functional Block Diversification

Fed-FBD partitions the ResNet-18 architecture into six granular functional blocks: input stem, four residual groups, and the output head. The server maintains a warehouse $\mathcal{W}$ of $N$ colour variants, each a composite of $B=6$ independently assigned blocks. Each block is not only versioned but stamped with a contributor trace, enabling precise downstream accountability and operations such as block-level unlearning.

FL rounds adhere to a shipping/request plan that strictly governs which colours (model variants) and blocks a client can update. A central methodological innovation is the use of direct block replacement (as opposed to aggregation or averaging) upon update return, guaranteeing non-interference across disjoint client assignments.

Training Objective

Clients perform local updates using a per-batch objective—jointly optimizing cross-entropy loss and a KL-divergence-based consistency regularizer between an update colour and a randomly selected reference colour. This regularization encourages functional diversity without collapse, with hyperparameter $\lambda$ controlling the regularization strength.

Privacy and Unlearning

By ensuring that no single block encounters enough data to meaningfully memorize client-specific content, Fed-FBD achieves privacy-by-design: precluding membership inference attacks even before the imposition of external privacy mechanisms. Surgical unlearning follows as a constant-time operation—the server identifies and replaces all blocks touched by a departing client with the aggregate of untampered peers, avoiding retraining and sharply limiting residual utility loss (typically under 0.25% AUC).

Empirical Evaluation and Results

Performance Under IID and Non-IID Partitioning

On standardized medical imaging benchmarks (BloodMNIST, DermaMNIST, PneumoniaMNIST, PathMNIST, CIFAR-10), Fed-FBD exhibits an IID AUC deficit of just $0.3\%$–$3.1\%$ compared to FedAvg in adequately sampled regimes (e.g., $>2000$ samples per client). Under moderate heterogeneity (Dirichlet $\alpha=1.0$), the performance gap is similarly constrained (losses remain within $B{\times}N$0–$B{\times}N$1 on the majority of datasets), but grows with increased non-IIDness or severe data scarcity per client. This reveals a sharply delineated phase transition: above roughly $B{\times}N$2K samples per client, Fed-FBD remains competitive even at strong heterogeneity ($B{\times}N$3); below this threshold, especially in pathological label drift, collapsed ensemble behavior emerges.

Figure 2: The data-per-client and heterogeneity phase transition. Ensemble AUC gap to FedAvg (at $B{\times}N$4) closes as per-client data increases, except for BloodMNIST, which is highlighted as an outlier.

Adversarial Isolation

Fed-FBD's structural guarantees yield perfect block-level isolation in all six adversarial attack settings evaluated—both label-flip and noise-injection, spanning three datasets. When a single client is adversarial, only the set of colours to which it has write access show pathology (AUC degradation up to $B{\times}N$5 for the poisoned group), while all other colours remain invariant to the attack—with maximum absolute drift below $B{\times}N$6 AUC.

Figure 3: Block-level isolation: under a label-flip attack by client 0, only the "poisoned" group experiences AUC collapse; the "clean" group and an unpoisoned ensemble retain their original performance, confirming architectural immunity of isolated blocks.

Privacy and Machine Unlearning

Membership inference attack (MIA) AUCs before unlearning operations are empirically indistinguishable from chance ($B{\times}N$7). Unlearning via aggregate block replacement incurs average AUC utility loss below $B{\times}N$8 and maximum observed loss below $B{\times}N$9, with wall-clock cost per operation well under one second. This approach neither requires approximate retraining nor suffers adversarial information leakage.

Inference-Time Routing and Ablations

While naive ensemble averaging is usually optimal once the ensemble is healthy, in regimes where specific colours are poisoned or degraded, routing strategies—selecting the most confident, top-k or learned combination of colours—can recapture substantial portions of the non-IID gap. Ablation studies demonstrate the stability of Fed-FBD to colour count $\mathcal{W}$0 (beyond $\mathcal{W}$1), to the regularization parameter $\mathcal{W}$2 (ideal in $\mathcal{W}$3), and to normalization scheme (BatchNorm preferred).

Implications and Theoretical Impact

Fed-FBD fundamentally redefines isolation as a primitive architectural property in FL, rather than as a probabilistic consequence of aggregation or the presence of bolted-on privacy mechanisms. The explicit, structural mapping between client contributors and model parameters enables both forensic and remedial operations: auditing, exclusion, and deletion of client influence become direct, explainable mechanisms subject to cryptographic audit.

Practically, this approach aligns with the strict requirements of hospital networks and other sensitive, multi-institutional deployments confronted with adversarial contamination, privacy audits, or deletion requests. The limitation to constant-time unlearning and robust privacy arises from the architectural imposition that each parameter must be owner-accountable and discretely replaceable.

The trade-off is an accuracy gap against classical FL methods in highly non-IID or small data-per-client regimes, a zone requiring further innovation—potentially via adaptive shipping plans, partial block sharing, or integration with personalized FL advances.

Conclusion

Fed-FBD establishes a new paradigm for architected isolation, privacy, and deletability in federated neural networks. It offers near-FedAvg accuracy on practical sample sizes while immunizing the FL system to adversarial contamination, limiting memorization of client-specific data, and enabling efficient, surgical machine unlearning operations—all without resorting to costly retraining or complex aggregators. The principle of block-level parameter ownership can be further generalized beyond ResNets and applied to future FL architectures, motivating new research directions in adaptable isolation strategies and fine-tuned personalization within federated regimes.