MH-pFedHNGD: Heterogeneous Personalized FL

• The paper introduces MH-pFedHNGD, a framework combining server-side hypernetworks, multi-head bucketing, and an optional global model to enable personalized federated learning for diverse client architectures.
• The methodology leverages client-specific embeddings and Jacobian-vector product updates to generate tailored model parameters while preserving the privacy of client architectures.
• Empirical results on datasets like CIFAR-100, Tiny-ImageNet, and EMNIST demonstrate MH-pFedHNGD’s superior performance over previous methods in both homogeneous and heterogeneous settings.

Searching arXiv for the specified paper to ground the article in the source publication. MH-pFedHNGD is a model-heterogeneous personalized federated learning framework introduced in "Hypernetworks for Model-Heterogeneous Personalized Federated Learning" (Zhang et al., 30 Jul 2025). It extends MH-pFedHN by combining a server-side hypernetwork, client-specific embedding vectors, and a multi-head structure that shares generation capacity among clients with similar model sizes, while optionally adding a lightweight global model and on-device knowledge distillation. The framework is designed for settings in which client models may differ in architecture $A_i$, parameter count $K_i$, and layer or operator types, and it targets personalized parameter generation without external datasets, without model decoupling, and without requiring disclosure of client model architectures to the server.

1. Problem setting and motivation

MH-pFedHNGD is formulated for model-heterogeneous personalized federated learning (MH-pFL), where there are $n$ clients with private datasets $D_i$ drawn from client-specific distributions $P_i$, and each client $i$ trains a personalized model $f_i$ with parameters $\theta_i$. The client architecture $A_i$ may differ across clients in layers, operators, widths, depths, parameter count $K_i$, and layer types such as CNN, FC, and BN. The objective is to deliver a $K_i$0 that fits $K_i$1 without sharing raw data and without requiring clients to disclose architecture details to the server (Zhang et al., 30 Jul 2025).

Formally, the target is

$K_i$2

with empirical objective

$K_i$3

where $K_i$4.

The central difficulty is that client model heterogeneity breaks weight-space alignment and makes traditional averaging or partial layer-sharing suboptimal or inapplicable. The paper positions hypernetworks as a mechanism that maps a compact client-specific embedding to the full parameter vector of a target network. In this formulation, hypernetworks act as a parameter-space unifier without requiring a common architecture; only the target parameter vector size matters. The paper also explicitly contrasts this design with prior MH-pFL strategies that use external or synthetic data for distillation, model decoupling, or partial learning, arguing that such strategies can be impractical, risky, sensitive to data quality or generator stability, or vulnerable to misaligned weights and representations.

2. Hypernetwork foundation and multi-head parameter generation

The MH-pFedHN foundation uses a server-side hypernetwork with parameters $K_i$5, where $K_i$6 is a shared feature extractor and $K_i$7 are multiple heads (Zhang et al., 30 Jul 2025). For each client $K_i$8, the server maintains customized embedding vectors

$K_i$9

where $n$0 depends on the required parameter count $n$1 and a chosen hypernetwork output block size $n$2.

At initialization, each client reports only $n$3 to the server. The server selects $n$4 and sets

$n$5

Clients with the same $n$6 are assigned to the same head $n$7. This bucketing mechanism is the basis of the multi-head structure: clients with similar sizes, measured by the number of embedding blocks, share a head, while the server need not access layer-by-layer architecture specifics.

For a client $n$8 assigned to head $n$9, each embedding block is mapped to a length-$D_i$0 vector:

$D_i$1

The full personalized flat parameter vector is then

$D_i$2

The server outputs $D_i$3 as a flat vector, and the client reshapes it into tensors per layer according to its private architecture $D_i$4 through a local reshape operator $D_i$5:

$D_i$6

This arrangement is central to how the method handles heterogeneous architectures. The server never learns layer shapes or operators; only the client knows how to reshape the flat vector. The paper states that this works for CNN, FC, BN parameters, and related cases. It also states that if clients change $D_i$7 during training, they update the local reshape map consistently. A plausible implication is that the framework separates parameter synthesis from architecture instantiation more strictly than approaches that assume shared layer semantics.

3. Objectives, gradient flow, and the role of the global model

For MH-pFedHN, the server optimizes the hypernetwork parameters and client embeddings to minimize the empirical losses on each client’s model instantiated from generated weights:

$D_i$8

Each round, the server materializes $D_i$9 and sends it to client $P_i$0. The client trains locally for $P_i$1 epochs on $P_i$2 and returns

$P_i$3

The server then updates the hypernetwork parameters and embeddings using Jacobian-vector products:

$P_i$4

$P_i$5

MH-pFedHNGD augments this with an optional lightweight global model $P_i$6 (Zhang et al., 30 Jul 2025). Let

$P_i$7

The global model $P_i$8 is generated by reusing the head of the smallest $P_i$9 group:

$i$0

with

$i$1

In practice, the paper states that $i$2 is instantiated as a small CNN, specifically LeNet-5 styled, while preserving the same flat-then-reshape generation path.

The role of $i$3 is twofold. First, it introduces a second lightweight update channel for the hypernetwork, which the paper states improves generalization. Second, it optionally serves as a teacher for client-specific models via on-device knowledge distillation. The coupled personalized-stage objective is

$i$4

For a batch $i$5 with personalized-model logits $i$6 and global-model logits $i$7 on the same input $i$8, the distillation term is

$i$9

The total client loss per batch is

$f_i$0

A common misconception would be to treat MH-pFedHNGD as a conventional shared-global-model method. The paper’s formulation is narrower: the global model is optional, lightweight, and generated through the hypernetwork itself, and its purpose is an additional update signal and optional teacher role rather than replacement of the personalized generation pathway.

4. Round structure, head assignment, and scalability

The paper provides two algorithms, one for MH-pFedHN and one for MH-pFedHNGD (Zhang et al., 30 Jul 2025). In MH-pFedHNGD, each round has a global pass followed by a personalized pass.

During the global pass, the server generates $f_i$1 using the head for the smallest-$f_i$2 group, broadcasts $f_i$3 to selected clients, and receives

$f_i$4

after local training. It then updates $f_i$5 and $f_i$6 with weighted Jacobian-vector products:

$f_i$7

$f_i$8

where $f_i$9.

During the personalized pass, the server generates $\theta_i$0 for each client according to the assigned head, clients optimize $\theta_i$1 locally, return $\theta_i$2, and the server updates $\theta_i$3 and $\theta_i$4 through the same Jacobian-vector-product mechanism.

The difference between MH-pFedHN and MH-pFedHNGD can be summarized as follows.

Variant	Components	Per-round structure
MH-pFedHN	Hypernetwork $\theta_i$5 with multi-heads, client embeddings $\theta_i$6	Single personalized pass
MH-pFedHNGD	MH-pFedHN components plus global model path $\theta_i$7 and optional KD	Global pass plus personalized pass

The scalability logic is explicitly tied to $\theta_i$8. The server stores $\theta_i$9 per client, the number of heads equals the number of distinct $A_i$0 values present, and dynamic creation of a new head for a new $A_i$1 is supported. For new architectures with new $A_i$2, the paper states that a new head can be added and the feature extractor $A_i$3 can be frozen while the new head is trained. The paper also states that removing heads severely degrades performance and that using one head for all clients causes OOM or poor accuracy, which is presented as evidence that multi-head bucketing is critical for both scalability and performance.

5. Empirical results and ablation findings

The reported evaluation covers EMNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet under two non-IID regimes: non-IID_1, defined as quantity-based label imbalance with per-client class subsets and random mass splits, and non-IID_2, defined as Dirichlet$A_i$4 partitioning (Zhang et al., 30 Jul 2025). The heterogeneous model suite includes LeNet-style CNN, VGG-8, and ResNets with 10-, 12-, and 18-layer variants. The hypernetwork feature extractor is a 3-layer MLP.

In homogeneous settings, the paper reports the following highlights. On CIFAR-100 with LeNet under non-IID_2 and 200 clients, MH-pFedHNGD reaches 82.54, compared with 79.49 for pFedLHN and 78.56 for FedAKT; MH-pFedHN reaches 81.60. On CIFAR-100 under non-IID_1 and 200 clients, MH-pFedHNGD reaches 61.59 versus 53.43 for pFedLHN, while MH-pFedHN reaches 60.11. On Tiny-ImageNet under non-IID_2 and 200 clients, MH-pFedHNGD reaches 66.67 versus 58.30 for pFedLHN, and MH-pFedHN reaches 63.61. For EMNIST, the paper states that performance is near-saturated for most methods and that the proposed methods are competitive at approximately 99%.

In heterogeneous settings, the paper reports that on CIFAR-100 with LeNet, VGG, and ResNets under non-IID_2 and 200 clients, MH-pFedHNGD reaches 76.46, MH-pFedHN 75.38, pFedLHN 74.60, and FedAKT 74.98. On Tiny-ImageNet under non-IID_2 and 200 clients, MH-pFedHNGD reaches 60.38, MH-pFedHN 58.79, and pFedLHN 57.53.

The reported generalization results are notable. The paper states that on unseen clients and architectures, MH-pFedHNGD and MH-pFedHN outperform other hypernetwork baselines by up to 20% in homogeneous generalization and up to 50% in heterogeneous generalization. For new architectures such as ResNet or SqueezeNet unseen in training, freezing $A_i$5 and adding a new head yields competitive performance. The paper also states that the global model gives limited extra gain in this setting, because learning relies mainly on new head adaptation to the new distribution. This suggests that the global-pass benefit is strongest when the pre-existing shared generator already has usable cross-client structure.

The ablation findings isolate three components. First, multi-head design is necessary: removing heads severely degrades performance. Second, the global pass is useful even without distillation, since HNG exceeds HN and HNGD exceeds HNG in most settings. Third, knowledge distillation remains effective under architecture mismatch: even when $A_i$6 and client architectures are entirely different, such as $A_i$7 and clients in $A_i$8, HNGD exceeds HN consistently. The paper also reports that communication pruning using the top-30% magnitudes of $A_i$9 yields negligible accuracy drop.

6. Privacy profile, communication costs, limitations, and implementation details

The privacy claim of MH-pFedHNGD is specific rather than absolute (Zhang et al., 30 Jul 2025). The framework does not require external, public, or synthetic data. Clients do not disclose their architectures; the server sees only $K_i$0 and uses $K_i$1 for bucketing. The server generates a flat $K_i$2 and does not require layer shapes, since reshaping occurs locally. The update vectors $K_i$3 and $K_i$4 are model-sized vectors, and the paper reports resilience to gradient inversion attacks, specifically iDLG, in practice. A plausible implication is that the method reduces architectural metadata exposure while still assuming that total parameter count is permissible metadata.

Communication and computation costs are also quantified. Client communication per round consists of uploading $K_i$5 with $K_i$6 floats and, optionally, $K_i$7 with $K_i$8 floats. The paper states that the additional cost of MH-pFedHNGD is an extra lightweight pass for $K_i$9 per round, and that the communication increase is small because the global model size is $K_i$00, the smallest among clients. On CIFAR-100 with 50 clients and 500 rounds under non-IID_1, the paper reports overall communication of approximately 1.83 MB for MH-pFedHN homogeneous, 3.66 MB for MH-pFedHNGD homogeneous, 3.00 MB for MH-pFedHN heterogeneous, and 4.83 MB for MH-pFedHNGD heterogeneous. Computation times grew by about 1.6–2.0$K_i$01 when adding the global pass.

The paper identifies several assumptions and limitations. Clients must be able to provide $K_i$02, store and reshape $K_i$03 according to their private $K_i$04, and run a small global model instance $K_i$05 locally. Bucketing by $K_i$06 is assumed sufficient to share heads among similar-size models. Extreme heterogeneity can introduce many heads, so server memory for $K_i$07 and multiple heads grows with the number of $K_i$08 buckets. The embedding dimension and output block size $K_i$09 trade off accuracy and compute. The paper also notes an accuracy drop when moving from homogeneous to heterogeneous model settings and frames better cross-architecture fusion as an open challenge. It further suggests that more principled head assignment beyond $K_i$10-based bucketing, such as data-driven clustering of size or behavior, could further improve sharing.

The implementation details reported in the paper are as follows. Training runs for up to 500 rounds with local epochs $K_i$11 and batch size 64. The client optimizer is SGD with learning rate $K_i$12, momentum $K_i$13, and weight decay $K_i$14. The hypernetwork optimizer is Adam with learning rate $K_i$15. The embedding dimension is $K_i$16, the hypernetwork output block size is $K_i$17, and $K_i$18 is a 3-layer MLP. In HNGD, the global model is LeNet-5 style by default, with $K_i$19 and $K_i$20. Distillation temperatures are dataset-specific: CIFAR-100 uses $K_i$21, Tiny-ImageNet $K_i$22, and EMNIST $K_i$23. The KD coefficients $K_i$24 are 0.01 for CIFAR-100, 0.2 for Tiny-ImageNet, and 0.1 for EMNIST. The paper states that the method is robust across participation ratios from 20% to 100%, that HNGD gains increase with participation, and that under very low participation and high heterogeneity in non-IID_1 the benefit of $K_i$25 can diminish because $K_i$26 reflects only a small, diverse subset. It also states that freezing client-side layers, such as the last FC or BN blocks, is supported by reshaping only the generated subset and retaining frozen parameters locally, with moderate accuracy drops under extreme constraints, and that HNGD improves robustness if at least 40% of clients can deploy $K_i$27.