---
title: 'MH-pFedHN: Heterogeneous Federated HyperNetworks'
url: https://www.emergentmind.com/topics/mh-pfedhn
type: topic
---

# MH-pFedHN: Heterogeneous Federated HyperNetworks

MH-pFedHN, short for **Model-Heterogeneous Personalized Federated HyperNetwork**, is a **data-free personalized federated learning framework** for the **model-heterogeneous personalized federated learning** setting, where clients may differ not only in data distribution but also in **model architecture**, **parameter count**, and **compute or memory budget**. Its central mechanism is a **server-side hypernetwork** that takes client-specific embedding vectors as input and generates personalized model parameters for each client, replacing direct aggregation of heterogeneous client weights with conditional parameter generation. The framework introduces a **multi-head** hypernetwork so that clients with similar parameter counts can share heads, and its extension **MH-pFedHNGD** adds a lightweight global model and distillation to improve generalization [2507.22330].

## 1. Problem setting and conceptual lineage

MH-pFedHN addresses **model-heterogeneous personalized federated learning (MH-pFL)**. In this regime, clients are statistically non-IID and also differ in **model architecture**, **parameter count**, **compute/memory budget**, and sometimes face **privacy constraints** that prevent disclosure of architecture details. The paper positions existing MH-pFL approaches in three categories: **partial training / submodel methods**, which require architectural alignment and often suffer when parameter spaces do not match; **federated distillation**, which usually needs a public or synthetic dataset; and **model decoupling**, which shares only shallow components such as feature extractors or classifiers and may limit knowledge fusion [2507.22330].

MH-pFedHN inherits its basic hypernetwork viewpoint from **pFedHN**. In pFedHN, personalized federated learning is formulated as minimizing average client risk over client-specific models rather than optimizing a single global model. Each client \(i\) has its own distribution \(P_i\), local dataset \(D_i=\{(x_j^{(i)},y_j^{(i)})\}_{j=1}^{m_i}\), and model parameters \(\theta_i\), while a server-side hypernetwork \(h(\cdot;\varphi)\) maps a client descriptor \(v_i\) to that client’s model weights, \(\theta_i=h(v_i;\varphi)\). The original formulation emphasized that hypernetwork parameters remain on the server, so communication depends on the generated client model rather than the full trainable meta-model [2103.04628].

The model-heterogeneous extension changes the scope of that idea. Rather than generating one personalized model per client within a common target architecture, MH-pFedHN is designed for federations in which clients may use **different neural network architectures and parameter sizes**. In that sense, it reframes pFedHN’s conditional model generation as a mechanism for heterogeneous architecture support, while preserving the original emphasis on server-side sharing and client-specific personalization.

## 2. Server-side hypernetwork and multi-head parameter generation

The MH-pFedHN hypernetwork is denoted
$$
h(\cdot;\boldsymbol{\varphi}),
$$
with parameters \(\boldsymbol{\varphi}\) decomposed into a **shared feature extractor** \(\boldsymbol{\varphi}_f\) and multiple **heads** \(\{\boldsymbol{\varphi}_{H_l}\}\). For client \(i\), the server learns or stores a set of embedding vectors
$$
\boldsymbol{v}_i = [\boldsymbol{v}_i^1,\dots,\boldsymbol{v}_i^{\tau_i}],
$$
where \(\tau_i\) is determined by the parameter count \(K_i\) of the client model and the hypernetwork output dimension \(N\):
$$
\tau_i = \left\lceil \frac{K_i}{N} \right\rceil.
$$
The paper recommends that \(\left\lceil K_i/N \right\rceil\) be greater than the number of layers in the client model [2507.22330].

Each embedding vector generates one parameter chunk:
$$
\boldsymbol{\theta}_i^j = h(\boldsymbol{v}_i^j;\boldsymbol{\varphi}_f,\boldsymbol{\varphi}_{H_l}),
$$
and the final client parameter vector is obtained by concatenation and truncation:
$$
\boldsymbol{\theta}_i := \operatorname{concat}(\boldsymbol{\theta}_i^1,\boldsymbol{\theta}_i^2,\cdots,\boldsymbol{\theta}_i^{\tau_i})_{[1:K_i]},
$$
equivalently,
$$
\boldsymbol{\theta}_i = h(\boldsymbol{v}_i;\boldsymbol{\varphi})_{[1:K_i]}.
$$
This chunked construction is presented as the mechanism that allows the server to generate parameters for different model sizes, avoid requiring direct access to client architectures, and handle heterogeneous clients more efficiently than one-shot full-model generation.

A defining architectural feature is the **multi-head structure**. Clients are grouped by **similar parameter counts**, and clients with similar \(K_i\) use the same embedding configuration and **share the same head** \(\boldsymbol{\varphi}_{H_l}\). The paper’s example is that if two models both require **three embedding vectors**, they share one head with **three output channels**; that head outputs three subsets of size \(N\), which are then combined and trimmed to form the client-specific parameter vector. The stated motivations are threefold: **knowledge sharing**, because similarly sized models can share generation structure; **efficiency**, because parameters for multiple similarly sized models can be generated in one pass; and **practicality**, because a single-head hypernetwork would be too costly and may run out of memory.

## 3. Objective function, local optimization, and server update rule

The MH-pFL objective is written as minimization of the sum of client empirical risks:
$$
\underset{\boldsymbol{\theta}_{1},\dots,\boldsymbol{\theta}_{n}}{\operatorname{arg\,min}}
\sum_{i=1}^n L_i(\boldsymbol{\theta}_i)
=
\underset{\boldsymbol{\theta}_{1},\dots,\boldsymbol{\theta}_{n}}{\operatorname{arg\,min}}
\sum_{i=1}^n \frac{1}{m_i}\sum_{j=1}^{m_i}\ell_i(x_j,y_j;\boldsymbol{\theta}_i).
$$
Under the hypernetwork parameterization, this becomes
$$
\underset{\boldsymbol{\varphi}, \boldsymbol{v}_1, \dots, \boldsymbol{v}_n}{\operatorname{arg\,min}}
\sum_{i=1}^n L_i\!\left(h(\boldsymbol{v}_i;\boldsymbol{\varphi})_{[1:K_i]}\right),
$$
or equivalently
$$
\underset{\boldsymbol{\varphi}, \boldsymbol{v}_1, \dots, \boldsymbol{v}_n}{\operatorname{arg\,min}}
\sum_{i=1}^n \frac{1}{m_i}\sum_{j=1}^{m_i}
\ell_i\!\left(x_j,y_j;h(\boldsymbol{v}_i;\boldsymbol{\varphi})_{[1:K_i]}\right).
$$
The optimization target is therefore the server’s mapping from embeddings to client parameters, together with the embeddings themselves [2507.22330].

The appendix gives the training rule. For each client \(i\), the server first generates
$$
\boldsymbol{\theta}_i = h(\boldsymbol{v}_i;\varphi)_{[1:K_i]}.
$$
The client then performs local SGD for \(E\) epochs:
$$
\widetilde{\boldsymbol{\theta}}_i \leftarrow
\widetilde{\boldsymbol{\theta}}_i - \eta \nabla_{\widetilde{\boldsymbol{\theta}}_i} L(\widetilde{\boldsymbol{\theta}}_i, B),
$$
and returns the model-space update
$$
\Delta\boldsymbol{\theta}_i = \widetilde{\boldsymbol{\theta}}_i - \boldsymbol{\theta}_i.
$$
The server updates the hypernetwork parameters and client embedding by backpropagating through the generated model:
$$
\varphi \leftarrow \varphi - \alpha \nabla_\varphi \boldsymbol{\theta}_i^T \Delta\boldsymbol{\theta}_i,
$$
$$
\boldsymbol{v}_i \leftarrow \boldsymbol{v}_i - \alpha \nabla_{\boldsymbol{v}_i}\varphi^T\nabla_\varphi \boldsymbol{\theta}_i^T\Delta\boldsymbol{\theta}_i.
$$

This is explicitly **not FedAvg over model weights**. The server does not average heterogeneous client models. Instead, it learns the **embedding-to-parameter mapping**, using client updates as supervision. The paper also characterizes the framework as **data-free**, because it does not rely on external datasets for cross-client knowledge transfer.

## 4. MH-pFedHNGD: lightweight global model and distillation

**MH-pFedHNGD** is an optional extension of MH-pFedHN that adds a **lightweight global model** generated by the same hypernetwork. The global model is
$$
\boldsymbol{w}_g := h(\boldsymbol{v}_g;\boldsymbol{\varphi})_{[1:K_g]},
$$
where
$$
K_g = \min\{K_1,\cdots,K_n\}.
$$
The global model reuses the head of the client with the smallest parameter count, which keeps it **lightweight** [2507.22330].

The method has **two phases per round**. In the **global update phase**, the server sends \(\boldsymbol{w}_g\) to all clients, clients train it locally and return
$$
\Delta \boldsymbol{w}_{g,i} = \boldsymbol{w}_{g,i} - \boldsymbol{w}_g,
$$
and the server aggregates these updates into the hypernetwork with data-size weighting:
$$
\varphi \leftarrow \varphi - \alpha \sum_{i=1}^n \frac{m_i}{M}\nabla_\varphi \boldsymbol{w}_g^T \Delta \boldsymbol{w}_{g,i},
$$
where
$$
M = \sum_i m_i.
$$
The same phase also updates the global embedding \(\boldsymbol{v}_g\).

In the **personalized update phase**, each client receives its personalized model and trains it using both its local task loss and a KL-divergence distillation term from the global model:
$$
\underset{\boldsymbol{\varphi}, \boldsymbol{v}_1, \dots, \boldsymbol{v}_n}{\operatorname{arg\,min}}
\sum_{i=1}^n \Big[
\lambda L_i(h(\boldsymbol{v}_i;\boldsymbol{\varphi})_{[1:K_i]})
+ (1-\lambda)L_{KL}\big(h(\boldsymbol{v}_i;\boldsymbol{\varphi})_{[1:K_i]}, h(\boldsymbol{v}_g;\boldsymbol{\varphi})_{[1:K_g]}\big)
\Big].
$$
Here \(L_{KL}\) is Kullback–Leibler divergence and \(\lambda\) balances task loss and distillation loss.

The paper attributes four roles to the global model: it provides the hypernetwork with an extra round of learning signals, improves generalization, acts as a teacher during personalized training, and makes the hypernetwork learn a more globally coherent representation. In the ablation study, the ranking
$$
\text{MH-pFedHNGD} > \text{MH-pFedHNG} > \text{MH-pFedHN}
$$
is reported, which the paper interprets as evidence that the global model improves hypernetwork learning and that distillation provides an additional gain.

## 5. Experimental configuration and empirical behavior

The reported experiments use four benchmarks: **EMNIST**, **CIFAR-10**, **CIFAR-100**, and **Tiny-ImageNet**. Two non-IID partition schemes are considered: **non-IID\_1**, described as quantity-based label imbalance, and **non-IID\_2**, a Dirichlet partition with \(Dir(0.01)\). Homogeneous-model experiments use **LeNet-style models**. Heterogeneous-model experiments use a mixture of **LeNet**, **VGGNet**, **10/12/18-layer ResNets**, **MLP**, and **SqueezeNet variants in some generalization tests**. The baselines are **Local Training**, **FedAvg**, **pFedHN**, **pFedLA**, **FedGH**, **pFedLHN**, **PeFLL**, and **FedAKT**. Training is conducted for up to **500 rounds** with local epochs \(E=2\), **SGD** on clients, **Adam** for the hypernetwork, embedding dimension **64**, hypernetwork output size **3072**, and **LeNet-5** as the global model in MH-pFedHNGD; dataset-specific distillation temperatures and coefficients are tuned by validation [2507.22330].

Selected results reported for MH-pFedHN and MH-pFedHNGD are summarized below.

| Setting | MH-pFedHN | MH-pFedHNGD |
|---|---:|---:|
| CIFAR-100, 100 clients, non-IID\_1 | 63.32 | 63.97 |
| Tiny-ImageNet, 200 clients, non-IID\_2 | 63.61 | 66.67 |
| CIFAR-100, 50 clients, non-IID\_1 | 57.09 | 60.11 |
| Tiny-ImageNet, 100 clients, non-IID\_2 | 58.93 | 61.11 |

In the **homogeneous model setting**, the paper reports that MH-pFedHN and MH-pFedHNGD outperform all baselines on **CIFAR-100**, **Tiny-ImageNet**, and **EMNIST**. In the **heterogeneous model setting**, both methods continue to outperform the baselines, with MH-pFedHNGD often producing the strongest results. For **generalization to unseen clients** on **CIFAR-100**, the hypernetwork is reported to generalize well to new clients, with MH-pFedHNGD generally stronger than MH-pFedHN; the reported gains are **up to 20%** in homogeneous generalization and **up to 50%** in heterogeneous generalization relative to baselines. The paper also states that for **completely new client architectures**, the method still works well and MH-pFedHNGD remains competitive, although the benefit of the global model is smaller.

## 6. Privacy, efficiency, and design trade-offs

The framework is presented as having several privacy and practicality properties. It requires **no external/public dataset**, and the server does **not need to know the client model architecture**, which the paper describes as preserving **structural privacy** for heterogeneous client models. The method is also characterized as suitable for **resource-constrained environments** because parameter generation is server-side, the multi-head design reduces memory cost, and MH-pFedHNGD adds only a **lightweight global model** rather than a large additional module. The paper further reports an **iDLG gradient inversion attack** experiment and states that data security is preserved even with the global-model extension [2507.22330].

The ablation results specify several design trade-offs. Removing the multi-head structure causes a **large drop** in performance, while using a **single head for everything** often causes **memory issues**. Sharing heads by similar parameter sizes provides the best reported balance between efficiency and performance. When **structurally different but same-size models** are forced to share a head, performance is **slightly worse** than when they are given separate heads, but the shared-head setting is still reported as robust and privacy-preserving because the server typically does not know architecture details.

The paper also reports that uploading only the **top 30%** of parameter updates **barely hurts performance**, which it interprets as evidence of practical communication efficiency. For MH-pFedHNGD specifically, the additional cost over MH-pFedHN is described as **modest**, while the empirical results show better performance and stronger generalization.

A recurrent point of clarification is that MH-pFedHN is not a heterogeneous form of direct model aggregation. Its central object is the **hypernetwork-induced mapping** from client embeddings to client parameters. The personalized model, the head assignment, and—when used—the global model all serve that mapping-based view of federated personalization. Within the hypernetwork lineage initiated by pFedHN, MH-pFedHN can therefore be understood as a model-heterogeneous reformulation in which the principal shared object is no longer a common client architecture or an averaged weight vector, but a server-side generative parameterization of heterogeneous personalized models.

Source: https://www.emergentmind.com/topics/mh-pfedhn