MH-pFedHN: Heterogeneous Federated HyperNetworks

• The paper introduces MH-pFedHN, a framework that replaces direct model aggregation with a server-side multi-head hypernetwork to generate personalized model parameters.
• MH-pFedHN supports diverse client architectures and resource constraints by conditioning parameter generation on client-specific embedding vectors.
• MH-pFedHNGD extends the framework with a lightweight global model and distillation techniques, boosting performance and generalization by up to 50% over baselines.

MH-pFedHN, short for Model-Heterogeneous Personalized Federated HyperNetwork, is a data-free personalized federated learning framework for the model-heterogeneous personalized federated learning setting, where clients may differ not only in data distribution but also in model architecture, parameter count, and compute or memory budget. Its central mechanism is a server-side hypernetwork that takes client-specific embedding vectors as input and generates personalized model parameters for each client, replacing direct aggregation of heterogeneous client weights with conditional parameter generation. The framework introduces a multi-head hypernetwork so that clients with similar parameter counts can share heads, and its extension MH-pFedHNGD adds a lightweight global model and distillation to improve generalization (Zhang et al., 30 Jul 2025).

1. Problem setting and conceptual lineage

MH-pFedHN addresses model-heterogeneous personalized federated learning (MH-pFL). In this regime, clients are statistically non-IID and also differ in model architecture, parameter count, compute/memory budget, and sometimes face privacy constraints that prevent disclosure of architecture details. The paper positions existing MH-pFL approaches in three categories: partial training / submodel methods, which require architectural alignment and often suffer when parameter spaces do not match; federated distillation, which usually needs a public or synthetic dataset; and model decoupling, which shares only shallow components such as feature extractors or classifiers and may limit knowledge fusion (Zhang et al., 30 Jul 2025).

MH-pFedHN inherits its basic hypernetwork viewpoint from pFedHN. In pFedHN, personalized federated learning is formulated as minimizing average client risk over client-specific models rather than optimizing a single global model. Each client $i$ has its own distribution $P_i$, local dataset $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$, and model parameters $\theta_i$, while a server-side hypernetwork $h(\cdot;\varphi)$ maps a client descriptor $v_i$ to that client’s model weights, $\theta_i=h(v_i;\varphi)$. The original formulation emphasized that hypernetwork parameters remain on the server, so communication depends on the generated client model rather than the full trainable meta-model (Shamsian et al., 2021).

The model-heterogeneous extension changes the scope of that idea. Rather than generating one personalized model per client within a common target architecture, MH-pFedHN is designed for federations in which clients may use different neural network architectures and parameter sizes. In that sense, it reframes pFedHN’s conditional model generation as a mechanism for heterogeneous architecture support, while preserving the original emphasis on server-side sharing and client-specific personalization.

2. Server-side hypernetwork and multi-head parameter generation

The MH-pFedHN hypernetwork is denoted

$h(\cdot;\boldsymbol{\varphi}),$

with parameters $\boldsymbol{\varphi}$ decomposed into a shared feature extractor $\boldsymbol{\varphi}_f$ and multiple heads $P_i$0. For client $P_i$1, the server learns or stores a set of embedding vectors

$P_i$2

where $P_i$3 is determined by the parameter count $P_i$4 of the client model and the hypernetwork output dimension $P_i$5:

$P_i$6

The paper recommends that $P_i$7 be greater than the number of layers in the client model (Zhang et al., 30 Jul 2025).

Each embedding vector generates one parameter chunk:

$P_i$8

and the final client parameter vector is obtained by concatenation and truncation:

$P_i$9

equivalently,

$D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$0

This chunked construction is presented as the mechanism that allows the server to generate parameters for different model sizes, avoid requiring direct access to client architectures, and handle heterogeneous clients more efficiently than one-shot full-model generation.

A defining architectural feature is the multi-head structure. Clients are grouped by similar parameter counts, and clients with similar $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$1 use the same embedding configuration and share the same head $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$2. The paper’s example is that if two models both require three embedding vectors, they share one head with three output channels; that head outputs three subsets of size $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$3, which are then combined and trimmed to form the client-specific parameter vector. The stated motivations are threefold: knowledge sharing, because similarly sized models can share generation structure; efficiency, because parameters for multiple similarly sized models can be generated in one pass; and practicality, because a single-head hypernetwork would be too costly and may run out of memory.

3. Objective function, local optimization, and server update rule

The MH-pFL objective is written as minimization of the sum of client empirical risks:

$D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$4

Under the hypernetwork parameterization, this becomes

$D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$5

or equivalently

$D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$6

The optimization target is therefore the server’s mapping from embeddings to client parameters, together with the embeddings themselves (Zhang et al., 30 Jul 2025).

The appendix gives the training rule. For each client $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$7, the server first generates

$D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$8

The client then performs local SGD for $D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}$9 epochs:

$\theta_i$0

and returns the model-space update

$\theta_i$1

The server updates the hypernetwork parameters and client embedding by backpropagating through the generated model:

$\theta_i$2

$\theta_i$3

This is explicitly not FedAvg over model weights. The server does not average heterogeneous client models. Instead, it learns the embedding-to-parameter mapping, using client updates as supervision. The paper also characterizes the framework as data-free, because it does not rely on external datasets for cross-client knowledge transfer.

4. MH-pFedHNGD: lightweight global model and distillation

MH-pFedHNGD is an optional extension of MH-pFedHN that adds a lightweight global model generated by the same hypernetwork. The global model is

$\theta_i$4

where

$\theta_i$5

The global model reuses the head of the client with the smallest parameter count, which keeps it lightweight (Zhang et al., 30 Jul 2025).

The method has two phases per round. In the global update phase, the server sends $\theta_i$6 to all clients, clients train it locally and return

$\theta_i$7

and the server aggregates these updates into the hypernetwork with data-size weighting:

$\theta_i$8

where

$\theta_i$9

The same phase also updates the global embedding $h(\cdot;\varphi)$0.

In the personalized update phase, each client receives its personalized model and trains it using both its local task loss and a KL-divergence distillation term from the global model:

$h(\cdot;\varphi)$1

Here $h(\cdot;\varphi)$2 is Kullback–Leibler divergence and $h(\cdot;\varphi)$3 balances task loss and distillation loss.

The paper attributes four roles to the global model: it provides the hypernetwork with an extra round of learning signals, improves generalization, acts as a teacher during personalized training, and makes the hypernetwork learn a more globally coherent representation. In the ablation study, the ranking

$h(\cdot;\varphi)$4

is reported, which the paper interprets as evidence that the global model improves hypernetwork learning and that distillation provides an additional gain.

5. Experimental configuration and empirical behavior

The reported experiments use four benchmarks: EMNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet. Two non-IID partition schemes are considered: non-IID_1, described as quantity-based label imbalance, and non-IID_2, a Dirichlet partition with $h(\cdot;\varphi)$5. Homogeneous-model experiments use LeNet-style models. Heterogeneous-model experiments use a mixture of LeNet, VGGNet, 10/12/18-layer ResNets, MLP, and SqueezeNet variants in some generalization tests. The baselines are Local Training, FedAvg, pFedHN, pFedLA, FedGH, pFedLHN, PeFLL, and FedAKT. Training is conducted for up to 500 rounds with local epochs $h(\cdot;\varphi)$6, SGD on clients, Adam for the hypernetwork, embedding dimension 64, hypernetwork output size 3072, and LeNet-5 as the global model in MH-pFedHNGD; dataset-specific distillation temperatures and coefficients are tuned by validation (Zhang et al., 30 Jul 2025).

Selected results reported for MH-pFedHN and MH-pFedHNGD are summarized below.

Setting	MH-pFedHN	MH-pFedHNGD
CIFAR-100, 100 clients, non-IID_1	63.32	63.97
Tiny-ImageNet, 200 clients, non-IID_2	63.61	66.67
CIFAR-100, 50 clients, non-IID_1	57.09	60.11
Tiny-ImageNet, 100 clients, non-IID_2	58.93	61.11

In the homogeneous model setting, the paper reports that MH-pFedHN and MH-pFedHNGD outperform all baselines on CIFAR-100, Tiny-ImageNet, and EMNIST. In the heterogeneous model setting, both methods continue to outperform the baselines, with MH-pFedHNGD often producing the strongest results. For generalization to unseen clients on CIFAR-100, the hypernetwork is reported to generalize well to new clients, with MH-pFedHNGD generally stronger than MH-pFedHN; the reported gains are up to 20% in homogeneous generalization and up to 50% in heterogeneous generalization relative to baselines. The paper also states that for completely new client architectures, the method still works well and MH-pFedHNGD remains competitive, although the benefit of the global model is smaller.

6. Privacy, efficiency, and design trade-offs

The framework is presented as having several privacy and practicality properties. It requires no external/public dataset, and the server does not need to know the client model architecture, which the paper describes as preserving structural privacy for heterogeneous client models. The method is also characterized as suitable for resource-constrained environments because parameter generation is server-side, the multi-head design reduces memory cost, and MH-pFedHNGD adds only a lightweight global model rather than a large additional module. The paper further reports an iDLG gradient inversion attack experiment and states that data security is preserved even with the global-model extension (Zhang et al., 30 Jul 2025).

The ablation results specify several design trade-offs. Removing the multi-head structure causes a large drop in performance, while using a single head for everything often causes memory issues. Sharing heads by similar parameter sizes provides the best reported balance between efficiency and performance. When structurally different but same-size models are forced to share a head, performance is slightly worse than when they are given separate heads, but the shared-head setting is still reported as robust and privacy-preserving because the server typically does not know architecture details.

The paper also reports that uploading only the top 30% of parameter updates barely hurts performance, which it interprets as evidence of practical communication efficiency. For MH-pFedHNGD specifically, the additional cost over MH-pFedHN is described as modest, while the empirical results show better performance and stronger generalization.

A recurrent point of clarification is that MH-pFedHN is not a heterogeneous form of direct model aggregation. Its central object is the hypernetwork-induced mapping from client embeddings to client parameters. The personalized model, the head assignment, and—when used—the global model all serve that mapping-based view of federated personalization. Within the hypernetwork lineage initiated by pFedHN, MH-pFedHN can therefore be understood as a model-heterogeneous reformulation in which the principal shared object is no longer a common client architecture or an averaged weight vector, but a server-side generative parameterization of heterogeneous personalized models.