---
title: 'FedDH: Dynamic Heterogeneous Aggregation'
url: https://www.emergentmind.com/topics/dynamic-heterogeneous-model-aggregation-feddh
type: topic
---

# FedDH: Dynamic Heterogeneous Aggregation

Searching arXiv for the specified FedDH paper and closely related heterogeneous aggregation methods.
arXiv search query: `all:"Dynamic Heterogeneous model aggregation" OR id:2507.10430 OR id:2407.05098 OR id:2601.13608 OR id:2211.10878`
Dynamic Heterogeneous model aggregation, abbreviated **FedDH**, is a server-side aggregation method in federated learning that is designed to handle **statistical heterogeneity** by dynamically reweighting client updates according to each client’s estimated **non-IID degree**. In the FedDHAD framework, FedDH addresses **data heterogeneity** during aggregation, while **FedAD** addresses **device heterogeneity** via adaptive dropout. The central departure from FedAvg is that aggregation weights are not determined solely by client sample counts; instead, they are modulated by a heterogeneity-aware quantity derived from Jensen–Shannon divergence and updated over training rounds [2507.10430].

## 1. Problem setting and conceptual scope

FedDH is introduced against the background that standard federated aggregation, especially **FedAvg**, is too naive for non-IID data. In FedAvg, the server averages client models in proportion to local sample counts, ignoring the extent to which a client’s local data distribution deviates from the global distribution. The paper associates this with **accuracy degradation**, **client drift**, **training instability**, and poor performance under high statistical heterogeneity. Its key premise is that clients whose data are more representative of the global distribution should have higher aggregation influence, whereas clients with stronger distribution shift should be weighted differently based on measured heterogeneity [2507.10430].

The method is formulated as a **bi-level optimization problem**:
\[
\min_{w,Q}\left[F(w, Q)\triangleq\frac{1}{n}\sum_{k = 1}^N n_k F_k(w(Q))\right]
\]
with
\[
w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k
\]
subject to
\[
w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}
\]
and
\[
Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.
\]

In this notation, \(w\) denotes global model parameters, \(w_k\) the local model from device \(k\), \(Q\) the set of aggregation weights, \(q_k\) the aggregation weight for client \(k\), \(F_k(w)\) the local loss on device \(k\), \(n_k\) the number of samples on client \(k\), \(n\) the total number of samples, and \(N'\) the number of selected clients in a round. FedDH is therefore explicitly framed as **learning the aggregation weights \(Q\)** jointly with the global model [2507.10430].

A common source of confusion is the word **heterogeneous**. In FedDH, heterogeneity refers primarily to **statistical data heterogeneity**, not to clients training incompatible model architectures. That distinction matters because other methods, such as FedTSA, address **model-heterogeneous federated learning** under **system heterogeneity**, where clients may train different pruned architectures and direct weight averaging can become infeasible [2407.05098].

## 2. Non-IID degree and the dynamic weighting rule

FedDH uses a client-specific heterogeneity score called the **non-IID degree**. The paper first introduces a theoretical quantity:
\[
\Gamma_{k,t} = E[F_k({w}^*)] - E[F_k({w_k}^*)].
\]
This quantity appears in the convergence analysis and characterizes how much worse the global optimum \(w^*\) performs on client \(k\) relative to that client’s own local optimum \(w_k^*\). Larger \(\Gamma_{k,t}\) indicates stronger heterogeneity [2507.10430].

For practical aggregation, FedDH estimates the non-IID degree through a **linear transformation of Jensen–Shannon divergence**:
\[
D^{non-IID}_k(P_k, \upsilon_k, b_k) = \upsilon_k \mathcal{D}_{JS}(P_k) + b_k,
\]
where \(P_k\) is the label distribution of client \(k\), \(\upsilon_k\) is a learned scaling control parameter, and \(b_k\) is a learned bias control parameter. The Jensen–Shannon divergence is defined as
\[
\mathcal{D}_{JS}(P_k) = \frac{1}{2}\mathcal{D}_{KL}(P_k||\overline{P}) + \frac{1}{2}\mathcal{D}_{KL}(P_m||\overline{P}),
\]
with
\[
\overline{P} = \frac{1}{2}(P_k + P_m)
\]
and
\[
P_m = \frac{\sum_{k = 1}^{N}n_k P_k}{\sum_{k = 1}^{N}n_k}.
\]
The KL divergence used is
\[
\mathcal{D}_{KL}(P_i||P_j)= \sum_{y \in \mathcal{Y}} P_i(y)\log\!\left(\frac{P_i(y)}{P_j(y)}\right).
\]

The central FedDH aggregation rule assigns the round-\(t\) weight
\[
q^t_k = \frac{\frac{n_k}{D^{non-IID}_k(P_k, \upsilon^t_k, b^t_k)}}{\sum_{k \in S^t}\frac{n_k}{D^{non-IID}_k(P_k, \upsilon^t_k, b^t_k)}},
\]
where \(S^t\) is the set of selected clients. The resulting interpretation is direct: a client with more samples contributes more, but its contribution is reduced when its estimated distribution shift is larger. In proportional form, the aggregation weight is
\[
\frac{n_k}{D^{non-IID}_k}.
\]

FedDH also updates the control parameters dynamically:
\[
\Upsilon^t \gets \Upsilon^t - \lambda_\Upsilon^t \nabla_{\Upsilon^t} F(w^t(Q^t)),
\]
\[
B^t \gets B^t - \lambda_B^t \nabla_{B^t} F(w^t(Q^t)).
\]
These updates refine the mapping from JS divergence to effective non-IID degree as training proceeds. The paper contrasts this with both FedAvg, which uses only \(n_k\), and a static JS-based baseline **FedJS**, arguing that static divergence weighting alone is insufficient because FedDH adapts weights to the current training state [2507.10430].

## 3. Algorithmic structure and interaction with adaptive dropout

The FedDH procedure begins by computing \(\mathcal{D}_{JS}(P_k)\) for each client and initializing
\[
\upsilon_k^0 \gets 1, \qquad b_k^0 \gets 0.
\]
At each communication round, the server selects a client subset \(\mathcal{C}\subseteq\{1,\dots,N\}\). Each selected client then performs SGD-based local training:
\[
w^{t+1}_k \gets w^t_k - \eta^t \nabla_{w^t_k} F_k(w^t_k).
\]
After local updates, the server updates the global model \(w^t\), the control parameters \(\Upsilon^t\) and \(B^t\), and the aggregation weights \(Q^t\), and then aggregates local models using the dynamically determined \(q_k^t\) [2507.10430].

Within **FedDHAD**, FedDH is paired with **FedAD**. The paper describes the two components as complementary: FedAD reduces model size and accelerates training through adaptive dropout, whereas FedDH improves model quality by weighting updates according to data heterogeneity. The paper explicitly states that FedDH’s slightly higher server-side overhead is offset by FedAD’s efficiency gains [2507.10430].

When dropout is applied, not all parameters are present on all clients. The paper therefore gives two aggregation cases. If a parameter survives on all selected clients,
\[
m^i = \sum q_k m^i_k.
\]
If it is dropped on some clients,
\[
m^i = \frac{\sum_{k \notin Drop} q_k m^i_k}{\sum_{k \notin Drop} q_k}.
\]
In this formulation, FedDH still determines the weights \(q_k\), but actual parameter fusion is conditioned on whether a parameter exists in a client’s submodel. This makes FedDH a weighting rule that remains operative even when the trained submodels are partially masked by dropout rather than fully identical [2507.10430].

## 4. Theoretical role in convergence and drift control

FedDH’s theoretical justification is tied to the role of heterogeneity in optimization error. The paper introduces
\[
\Gamma = \max_{t}\sum_{k \in S_t} p_{k,t} \Gamma_{k,t},
\]
where \(p_{k,t}\) are aggregation weights. Theorem 2 states that if the weights are chosen as
\[
p_{k,t} = \frac{\frac{n_k}{\Gamma_{k,t}}}{\sum_{k'\in S_t}\frac{n_{k'}}{\Gamma_{k',t}}},
\]
then the error upper bound is **equal or smaller** than with standard sample-proportional averaging
\[
p_{k,t} = \frac{n_k}{\sum_{k'\in S_t}n_{k'}}.
\]
This is the formal rationale for heterogeneity-aware weighting: weighting by inverse heterogeneity can tighten the optimization bound relative to purely sample-size-based aggregation [2507.10430].

The paper’s account of FedDH is therefore not merely heuristic. It links the aggregation rule to a heterogeneity quantity that appears in convergence analysis and uses that link to motivate inverse-weighting by client-specific distribution mismatch. This suggests that FedDH should be understood as an attempt to control **client drift** at the aggregation layer rather than by altering local objectives alone.

That server-side emphasis distinguishes FedDH from methods such as **FedProx**, **FedNova**, **MOON**, and **FedDyn**, which mainly modify local training or use gradient or regularization mechanisms to reduce non-IID effects. A related but technically different line of work is **Fisher-Informed Parameterwise Aggregation (FIPA)**, which also argues that uniform client-level scalar weights are inadequate under non-IID data, but replaces them with parameter-specific matrix weights derived from low-rank Fisher or generalized Gauss–Newton curvature. FIPA is thus adaptive at the level of parameter directions, whereas FedDH is adaptive at the level of client weights [2601.13608].

## 5. Empirical evaluation and reported performance

The reported FedDH experiments use **100 clients**, with **10 selected per round**, on **CIFAR-10**, **CIFAR-100**, **SVHN**, and **TinyImageNet**. Models include **LeNet**, **CNN**, **VGG11**, and **ResNet18**. Non-IID partitioning is based on a Dirichlet distribution with default concentration \(\beta = 0.5\). Training settings are learning rate **0.1**, decay **0.99**, batch size **10**, local epochs \(\tau = 5\), and **500** rounds for LeNet, CNN, and VGG and **1000** rounds for ResNet [2507.10430].

FedDH is compared against **FedAvg**, **FedProx**, **FedNova**, **MOON**, **FedDyn**, **FedAS**, **FedKTL**, and **AugFL**, among others in the full evaluation. On **CIFAR-10 with LeNet**, FedDH achieves **0.621 accuracy**, compared with **0.565** for FedAvg, **0.588** for FedProx, **0.582** for FedNova, **0.610** for MOON, and **0.543** for FedDyn. The corresponding gains reported are **+5.6% over FedAvg**, **+3.3% over FedProx**, **+4.0% over FedNova**, **+1.1% over MOON**, and **+7.8% over FedDyn**. On **CIFAR-100**, the paper reports gains up to **+3.4% over FedAvg**, **+2.3% over FedProx**, **+1.1% over FedNova**, **+2.2% over MOON**, and **+8.3% over FedDyn**. On **SVHN** and **TinyImageNet**, FedDH yields higher accuracy than FedAvg by up to **0.9%**, over FedProx by up to **1.0%**, and over FedNova by up to **1.3%** [2507.10430].

The paper also reports auxiliary findings. If client distribution metadata \(P_k\) cannot be shared, the alternative estimator **FedDHE** uses gradients from a small balanced public or server dataset, and **FedDH** and **FedDHE** achieve **similar accuracy** with a difference of **less than 0.8%**. Against the static baseline **FedJS**, FedDH outperforms by up to **2.9% on LeNet** and **0.9% on CNN** for CIFAR-10. In ablation on CIFAR-10 with LeNet, **FedDHAD** achieves **0.633 accuracy, 2446 s**, **FedDH** achieves **0.621 accuracy, 2668 s**, and **FedAD** achieves **0.595 accuracy, 3942 s**. The paper interprets these results as evidence that dynamic aggregation contributes directly to the full system’s performance and that the combination of FedDH and FedAD improves both accuracy and time [2507.10430].

FedDH is not presented as an efficiency method in isolation. The reported server computation overhead is only up to **10.4%** for FedDH and up to **15.6%** for FedDHAD. The server overhead table includes values such as **0.086** for CIFAR-10/LeNet, **0.042** for CIFAR-10/CNN, **0.104** for CIFAR-100/LeNet, and **0.043** for CIFAR-100/CNN [2507.10430].

## 6. Relation to adjacent methods and interpretive boundaries

FedDH belongs to a broader family of aggregation methods that attempt to correct the limitations of FedAvg under heterogeneity, but its mechanism is specific. It is neither a model-heterogeneous architecture-matching method nor a synthetic-data correction method in the sense used by some server-side distillation approaches.

This boundary becomes clearer when contrasted with **FedTSA**, which is designed for **model-heterogeneous federated learning** under **system heterogeneity**. FedTSA clusters clients by processing capability using a proxy task, maps clusters to pruned model sizes, performs **cluster-wise weight averaging** among clients with identical architectures, and then performs **cross-cluster aggregation** by **deep mutual learning** on synthetic data generated with **Stable Diffusion v1-4**. In FedTSA, dynamic heterogeneous aggregation refers to switching aggregation modes according to whether models are homogeneous within a cluster or heterogeneous across clusters [2407.05098].

A second nearby method is **DynaFed**, which tackles client data heterogeneity by reserving a short early trajectory of global model snapshots, synthesizing a pseudo dataset that mimics the reserved trajectory, and then using that pseudo data to fine-tune later server aggregations. DynaFed is described as a dynamic aggregation enhancement that keeps the FedAvg backbone and adds a server-side correction learned from global dynamics rather than from client label-distribution metadata [2211.10878].

These comparisons help delimit the meaning of **FedDH**. In the FedDHAD paper, FedDH denotes a **dynamic, heterogeneity-aware client-weighting rule** based on estimated non-IID degree. It does not perform dynamic architecture assignment, cross-architecture knowledge transfer, or parameterwise curvature fusion. A plausible implication is that the phrase *dynamic heterogeneous model aggregation* can denote substantially different mechanisms across the federated-learning literature, even when the common objective is to move beyond uniform FedAvg-style averaging. Within that landscape, FedDH is most precisely classified as a **server-side dynamic reweighting method for statistical heterogeneity** [2507.10430].

Source: https://www.emergentmind.com/topics/dynamic-heterogeneous-model-aggregation-feddh