pFedDSH: Personalized Fed Data-Free Sub-Hypernetwork

• The paper introduces pFedDSH, a framework that uses a central hypernetwork with client embeddings to generate personalized models in federated learning.
• It employs batch-specific binary masks to carve sub-hypernetworks, ensuring that existing clients maintain performance without further local retraining.
• The method integrates a server-side data-free replay mechanism inspired by DeepInversion, yielding positive retro-active improvement across client batches.

Searching arXiv for pFedDSH and closely related work to ground the article. {"query":"(Nguyen et al., 7 Aug 2025) pFedDSH Personalized Federated Data-Free Sub-Hypernetwork", "max_results": 5} {"query":"Personalized Federated Data-Free Sub-Hypernetwork hypernetwork federated learning progressive client onboarding", "max_results": 10} pFedDSH is a Personalized Federated Learning framework for Progressive Client Onboarding in Federated Learning (PCO-FL), a setting in which clients arrive incrementally in batches while the underlying prediction task and label space remain fixed. The method combines a central hypernetwork that generates a full personalized model from a small client embedding, batch-specific binary masks that carve the hypernetwork into sub-hypernetworks specialized for different onboarding batches, and a server-side data-free replay mechanism motivated by DeepInversion for backward knowledge transfer without raw data sharing. It is designed to satisfy three requirements simultaneously: preserving the performance of existing clients without further local retraining, improving the initialization and adaptation of new clients, and enabling backward transfer to earlier clients under privacy constraints (Nguyen et al., 7 Aug 2025).

1. Progressive client onboarding formulation

pFedDSH is defined for a federated scenario in which clients do not all exist at the beginning of training. Instead, clients arrive in batches $\{\mathcal{B}_t\}_{t=0}^{T}$ over timesteps $t$, and each client $c \in \mathcal{B}_t$ possesses a local dataset $\mathcal{D}_c$ drawn from a client-specific distribution $P_c(x,y)$. The learning task remains unchanged across time; examples given for this regime are fixed-task image classification problems such as CIFAR-10 classification. This setting is more restrictive than ordinary static-client pFL and differs from task-incremental or class-incremental Federated Continual Learning because the label space does not change while the client population does (Nguyen et al., 7 Aug 2025).

The formulation introduces three evaluation requirements. Stability of existing clients requires that once a client has been onboarded and trained, its accuracy should not drop when later batches join, even though that client does not retrain locally. Pro-active Adaptation (PA) measures forward transfer to new clients by comparing their federated personalized models against purely local training from scratch:

$$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$

Retro-active Improvement (RI) measures backward transfer to existing clients:

$$\text{RI} = \mathbb{E}_{c \in \mathcal{B}_{<t}}\big[ Acc^{(t)}_{\text{Trained}(c)} - Acc^{(t-1)}_{\text{Trained}(c)} \big].$$

The explicit requirement that old clients retain or improve accuracy without local retraining or full re-sync is central to the method’s design.

A common misconception is to treat this regime as ordinary pFL with intermittent participation. In pFedDSH, the issue is not merely partial participation; it is the structural introduction of new client batches into an already trained personalized system. Another misconception is to equate it with classical FCL. The paper explicitly distinguishes PCO-FL from FCL by noting that the task and label space are fixed, whereas many FCL methods are designed for changing tasks or classes (Nguyen et al., 7 Aug 2025).

2. Hypernetwork-based personalization

The core model is a central hypernetwork $H(\cdot;\phi)$ maintained on the server. For client $c$, the server stores or receives a low-dimensional embedding $e_c \in \mathbb{R}^d$ and generates that client’s full model parameters by

$t$0

where $t$1 denotes global hypernetwork parameters and $t$2 denotes the weights of the personalized base network. In the reported implementation, $t$3 is a 2-layer MLP that outputs all parameters of a ResNet-18 (~11.2M weights) from a 32-dimensional embedding. The paper also states that a small CNN is used to generate $t$4, although conceptually the embedding can be regarded as a learnable client-specific vector (Nguyen et al., 7 Aug 2025).

This design yields two structural properties. First, the full personalized model for any client is reconstructible on the server from its embedding, which is what makes server-side data-free replay feasible. Second, personalization is represented through jointly optimized differentiable quantities, namely the shared hypernetwork parameters $t$5 and the client embeddings $t$6. These properties connect pFedDSH to hypernetwork-based pFL, especially pFedHN, which also uses a central hypernetwork to generate client-specific models from client embeddings (Shamsian et al., 2021).

Component	Definition	Role
Central hypernetwork	$t$7	Generates full personalized model parameters
Client embedding	$t$8	Encodes client identity and data distribution
Sub-hypernetwork mechanism	Batch-specific mask applied to generated weights	Preserves older knowledge and allocates capacity

The term sub-hypernetwork in pFedDSH refers not to separate independent generators, but to a masked operational slice of a single global hypernetwork. Different onboarding batches therefore occupy different masked regions of the same parameter-generating system.

3. Batch-specific masks and sub-hypernetworks

For a client $t$9 in batch $c \in \mathcal{B}_t$0, pFedDSH first generates $c \in \mathcal{B}_t$1 and then applies a batch-specific binary mask:

$c \in \mathcal{B}_t$2

where $c \in \mathcal{B}_t$3 and $c \in \mathcal{B}_t$4 denotes element-wise multiplication. The mask acts at parameter or neuron granularity, so that only coordinates with mask value $c \in \mathcal{B}_t$5 are active. The combination of the hypernetwork and this mask defines the effective sub-hypernetwork used by that onboarding batch (Nguyen et al., 7 Aug 2025).

Mask learning is formulated as a sparsity-constrained optimization problem:

$c \in \mathcal{B}_t$6

Here $c \in \mathcal{B}_t$7 is the task loss and $c \in \mathcal{B}_t$8 is an $c \in \mathcal{B}_t$9 regularizer encouraging sparse activation patterns. The paper’s interpretation is direct: sparse masks minimize the subset of neurons used for a new batch, preserve capacity for future batches, and reduce interference with parameters already important for earlier batches.

Knowledge preservation depends on mask immutability. Once a batch has completed training, its learned mask is frozen for future rounds. The paper formalizes this in the theory section as the assumption that for all later rounds, a client’s mask remains equal to the mask at its freeze round. This means that earlier clients retain a fixed pattern of active neurons even while the shared hypernetwork parameters $\mathcal{D}_c$0 continue to evolve. The intended effect is a stability–plasticity trade-off: older subnetworks remain protected from direct overwriting, while newer batches can be assigned new or partially overlapping neural resources (Nguyen et al., 7 Aug 2025).

Empirically, this masking mechanism is tied to the method’s stability claims. The paper reports that without masking, later updates overwrite parameters and induce large negative RI, including RI $\mathcal{D}_c$1 on CIFAR-10. With masking, existing clients remain stable, and when replay is added, earlier clients can improve rather than merely avoid degradation.

4. Data-free replay and optimization procedure

Masks protect existing subnetworks, but pFedDSH does not rely on masking alone. To obtain positive backward transfer, it introduces a server-side data-free replay mechanism inspired by DeepInversion. The goal is to synthesize pseudo-data $\mathcal{D}_c$2 whose internal batch-normalization statistics match those observed during real training on new clients. The synthesis objective includes a BN alignment loss, total variation regularization, $\mathcal{D}_c$3 regularization, and a feature distribution regularizer. The server collects BN statistics from client training states and optimizes synthetic images from noise without accessing raw client data (Nguyen et al., 7 Aug 2025).

Once a new batch $\mathcal{D}_c$4 has been integrated, the server constructs a Synthetic Data Pool $\mathcal{D}_c$5 from clients in $\mathcal{D}_c$6. For each earlier batch $\mathcal{D}_c$7, the server retrieves its masked subnetwork and fine-tunes it on the synthetic data:

$\mathcal{D}_c$8

Because the masks for earlier batches are frozen, this replay step injects information from the new client distribution into older subnetworks without any new client-side training. The paper attributes positive RI specifically to this replay stage; removing replay preserves much of the stability but eliminates positive backward transfer.

The federated training loop is organized batch by batch. The server maintains the global hypernetwork $\mathcal{D}_c$9 and all previously learned masks. For a new batch, clients initialize embeddings and transmit them to the server. The server generates masked personalized models,

$P_c(x,y)$0

which are trained locally for $P_c(x,y)$1 epochs with SGD. Clients send gradients with respect to $P_c(x,y)$2 and the batch mask back to the server, and the server updates both the hypernetwork and the current mask. After batch completion, the mask becomes effectively fixed, and the server performs data-free replay if $P_c(x,y)$3 (Nguyen et al., 7 Aug 2025).

The theoretical analysis assumes that the global objective

$P_c(x,y)$4

is $P_c(x,y)$5-smooth, gradients are unbiased with bounded variance, masks are immutable after their batch, and server learning rates satisfy Robbins–Monro conditions. Under these assumptions, the paper proves convergence to a stationary point:

$P_c(x,y)$6

A frequent misunderstanding is that “data-free replay” implies a public dataset or a learned generator. In pFedDSH, replay is server-side and DeepInversion-style, using BN statistics rather than raw data; this differs from knowledge-transfer methods such as FedHKD, which aggregate class-wise mean representations and soft predictions as “hyper-knowledge” and do not rely on DeepInversion (Chen et al., 2023).

5. Experimental protocol and empirical findings

The reported experiments use CIFAR-10, CIFAR-100, and Tiny-ImageNet with 100 clients. Training and test data are partitioned across clients with a Dirichlet distribution with $P_c(x,y)$7, creating strong non-IID heterogeneity. The initial batch contains 80 clients. The remaining clients are added either as a 2-batch scenario with 20 new clients, a 5-batch scenario with 5 new clients per batch, or an 11-batch scenario with 2 new clients per batch. Existing clients are trained for 200 communication rounds; each new batch is trained for 100 rounds; in each round, 5% of currently available clients are sampled; and each selected client trains for one epoch with batch size 32 (Nguyen et al., 7 Aug 2025).

The comparison set includes FedAvg, FedPer, FedRep, pFedHN, FedWeIT, FedCLASS, and HR. All methods use ResNet-18 as the base backbone and similar training budgets. The principal reported result is that pFedDSH is the only method in the investigation scenario that achieves positive RI across all three datasets while also maintaining non-negative or positive PA.

On CIFAR-10, pFedDSH improves existing clients from 66.57 to 68.89, giving RI = +2.32, and yields PA = +1.42 for new clients, with mutual benefits +1.87. On CIFAR-100, it improves existing clients from 37.23 to 39.40, giving RI = +2.17, and yields PA = +8.37, with mutual benefits +5.37. On Tiny-ImageNet, it improves existing clients from 10.11 to 10.22, giving RI = +0.11, and yields PA = +0.27, with mutual benefits +0.19. By contrast, all baselines show negative RI, and several also show negative PA; the paper gives FedAvg PA $P_c(x,y)$8 on CIFAR-10 as an example (Nguyen et al., 7 Aug 2025).

The multi-batch analysis strengthens the same conclusion. For the first batch of clients, pFedDSH does not merely maintain accuracy across 11 batches; it increases it and then stabilizes. The paper reports CIFAR-10 first-batch accuracy from 66.67% at batch 1 to $P_c(x,y)$9 by later batches, CIFAR-100 from 37.23% to $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$0, and Tiny-ImageNet from 10.11% to $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$1. This is presented as direct evidence of positive backward transfer under a no-retraining constraint.

The ablation study isolates the two main mechanisms. Without neuron masking, CIFAR-10 5-batch accuracy falls to 51.22% with RI $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$2. Without data-free replay, CIFAR-10 5-batch accuracy remains relatively high at 64.15%, PA stays around 1.42%, but RI becomes $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$3, indicating stability without backward improvement. The effect of the mask sparsity parameter is also quantified: with $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$4, accuracy is 58.62% and all neurons are used; with $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$5, the method attains 68.40% with $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$6 of neurons used and sparsity 74%. Increasing the embedding dimension from 8 $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$7 16 $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$8 32 improves accuracy and RI, whereas 64 yields diminishing returns (Nguyen et al., 7 Aug 2025).

6. Positioning, limitations, and outlook

Relative to standard FL and static-client pFL, pFedDSH is positioned as a response to a specific failure mode: integrating new clients into a personalized system without degrading earlier ones. The paper argues that FedAvg drifts toward new data distributions and severely forgets early clients, and that FedPer, FedRep, and particularly pFedHN were designed for static client sets rather than progressive onboarding. The comparison to pFedHN is especially important because pFedDSH retains the hypernetwork-based personalization principle but adds batch-specific neuron masks and data-free replay, which are the mechanisms credited with converting negative RI into positive RI in the onboarding regime (Shamsian et al., 2021).

Relative to Federated Continual Learning, the paper places FedWeIT, FedCLASS, and HR in a neighboring but distinct category. Those methods focus on sequential tasks or changing classes. In the PCO-FL experiments they mitigate forgetting better than many pFL baselines, but still exhibit negative RI; for example, the paper reports HR RI $$\text{PA} = \mathbb{E}_{c \in \mathcal{B}_t}\big[ Acc_{\text{Trained}(c)} - Acc_{\text{Local}(c)} \big].$$9 on CIFAR-10. pFedDSH differs by targeting static task, dynamic clients, using sub-hypernetworks plus server-side replay once per batch, and controlling replay cost more tightly than approaches that replay every round (Nguyen et al., 7 Aug 2025).

The method’s limitations are explicit. DeepInversion is computationally expensive, and the server-side replay step adds non-trivial overhead, reported as 8s once per batch in the five-batch CIFAR-10 latency study. pFedDSH is also tailored to a fixed task/label space and does not directly solve genuine multi-task or label-incremental settings. Long-term scalability may be limited by capacity as the number of batches grows, even with mask sparsity and reuse. The replay mechanism depends on BN statistics, so architectures without BN would require an alternative inversion mechanism. Finally, although the method avoids raw data transfer and uses BN statistics rather than client datasets, the paper explicitly states that this is not formally differentially private (Nguyen et al., 7 Aug 2025).

The reported latency numbers place these trade-offs in context. In the five-batch CIFAR-10 setting, FedAvg requires 0.31s per round with RI $$\text{RI} = \mathbb{E}_{c \in \mathcal{B}_{<t}}\big[ Acc^{(t)}_{\text{Trained}(c)} - Acc^{(t-1)}_{\text{Trained}(c)} \big].$$0, pFedHN requires 0.38s per round with RI $$\text{RI} = \mathbb{E}_{c \in \mathcal{B}_{<t}}\big[ Acc^{(t)}_{\text{Trained}(c)} - Acc^{(t-1)}_{\text{Trained}(c)} \big].$$1, HR requires 2.52s per round (0.31 + 2.21) with RI $$\text{RI} = \mathbb{E}_{c \in \mathcal{B}_{<t}}\big[ Acc^{(t)}_{\text{Trained}(c)} - Acc^{(t-1)}_{\text{Trained}(c)} \big].$$2, and pFedDSH requires 0.38s for an ordinary round, 8s replay once per batch, and 0.42s average per round over 200 rounds, with RI +2.12. A plausible implication is that the framework’s main novelty lies not merely in retaining old performance, but in doing so with replay frequency reduced to the batch level rather than the round level.

In this sense, pFedDSH occupies an intersection of hypernetwork-based personalization, parameter isolation, and data-free knowledge transfer. Its central claim is that sub-hypernetworks defined by persistent batch-specific masks can preserve older client knowledge, while DeepInversion-based replay can convert that preserved structure into retro-active improvement. The method therefore formalizes a fixed-task federated regime in which onboarding is progressive, personalization is server-generated, and backward transfer is pursued without raw-data access (Nguyen et al., 7 Aug 2025).