FedDH: Dynamic Heterogeneous Aggregation

• FedDH is a dynamic aggregation method that reweights client updates by quantifying non-IID degrees via Jensen–Shannon divergence.
• It formulates a bi-level optimization problem to jointly learn global model parameters and heterogeneity-aware client weights, reducing client drift.
• Empirical results on CIFAR-10, CIFAR-100, SVHN, and TinyImageNet demonstrate significant accuracy gains over FedAvg and similar baselines.

Searching arXiv for the specified FedDH paper and closely related heterogeneous aggregation methods. arXiv search query: all:"Dynamic Heterogeneous model aggregation" OR id:([2507.10430](/papers/2507.10430)) OR id:([2407.05098](/papers/2407.05098)) OR id:([2601.13608](/papers/2601.13608)) OR id:([2211.10878](/papers/2211.10878)) Dynamic Heterogeneous model aggregation, abbreviated FedDH, is a server-side aggregation method in federated learning that is designed to handle statistical heterogeneity by dynamically reweighting client updates according to each client’s estimated non-IID degree. In the FedDHAD framework, FedDH addresses data heterogeneity during aggregation, while FedAD addresses device heterogeneity via adaptive dropout. The central departure from FedAvg is that aggregation weights are not determined solely by client sample counts; instead, they are modulated by a heterogeneity-aware quantity derived from Jensen–Shannon divergence and updated over training rounds (Liu et al., 14 Jul 2025).

1. Problem setting and conceptual scope

FedDH is introduced against the background that standard federated aggregation, especially FedAvg, is too naive for non-IID data. In FedAvg, the server averages client models in proportion to local sample counts, ignoring the extent to which a client’s local data distribution deviates from the global distribution. The paper associates this with accuracy degradation, client drift, training instability, and poor performance under high statistical heterogeneity. Its key premise is that clients whose data are more representative of the global distribution should have higher aggregation influence, whereas clients with stronger distribution shift should be weighted differently based on measured heterogeneity (Liu et al., 14 Jul 2025).

The method is formulated as a bi-level optimization problem: $$\min_{w,Q}\left[F(w, Q)\triangleq\frac{1}{n}\sum_{k = 1}^N n_k F_k(w(Q))\right]$$ with

$w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$

subject to

$$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$

and

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$

In this notation, $w$ denotes global model parameters, $w_k$ the local model from device $k$, $Q$ the set of aggregation weights, $q_k$ the aggregation weight for client $k$, $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$0 the local loss on device $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$1, $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$2 the number of samples on client $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$3, $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$4 the total number of samples, and $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$5 the number of selected clients in a round. FedDH is therefore explicitly framed as learning the aggregation weights $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$6 jointly with the global model (Liu et al., 14 Jul 2025).

A common source of confusion is the word heterogeneous. In FedDH, heterogeneity refers primarily to statistical data heterogeneity, not to clients training incompatible model architectures. That distinction matters because other methods, such as FedTSA, address model-heterogeneous federated learning under system heterogeneity, where clients may train different pruned architectures and direct weight averaging can become infeasible (Fan et al., 2024).

2. Non-IID degree and the dynamic weighting rule

FedDH uses a client-specific heterogeneity score called the non-IID degree. The paper first introduces a theoretical quantity: $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$7 This quantity appears in the convergence analysis and characterizes how much worse the global optimum $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$8 performs on client $w(Q) = \sum_{k = 1, q_k \in Q}^{N'} q_k w_k$9 relative to that client’s own local optimum $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$0. Larger $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$1 indicates stronger heterogeneity (Liu et al., 14 Jul 2025).

For practical aggregation, FedDH estimates the non-IID degree through a linear transformation of Jensen–Shannon divergence: $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$2 where $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$3 is the label distribution of client $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$4, $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$5 is a learned scaling control parameter, and $$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$6 is a learned bias control parameter. The Jensen–Shannon divergence is defined as

$$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$7

with

$$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$8

and

$$w_k = \argmin_{w_k}F_k(w_k), \quad \forall k \in \{1, ..., N\}$$9

The KL divergence used is

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$0

The central FedDH aggregation rule assigns the round-$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$1 weight

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$2

where $$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$3 is the set of selected clients. The resulting interpretation is direct: a client with more samples contributes more, but its contribution is reduced when its estimated distribution shift is larger. In proportional form, the aggregation weight is

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$4

FedDH also updates the control parameters dynamically: $$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$5

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$6

These updates refine the mapping from JS divergence to effective non-IID degree as training proceeds. The paper contrasts this with both FedAvg, which uses only $$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$7, and a static JS-based baseline FedJS, arguing that static divergence weighting alone is insufficient because FedDH adapts weights to the current training state (Liu et al., 14 Jul 2025).

3. Algorithmic structure and interaction with adaptive dropout

The FedDH procedure begins by computing $$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$8 for each client and initializing

$$Q = \argmin_{Q}F(w, Q), \qquad \sum_{q_k \in Q}^{N'} q_k = 1.$$9

At each communication round, the server selects a client subset $w$0. Each selected client then performs SGD-based local training: $w$1 After local updates, the server updates the global model $w$2, the control parameters $w$3 and $w$4, and the aggregation weights $w$5, and then aggregates local models using the dynamically determined $w$6 (Liu et al., 14 Jul 2025).

Within FedDHAD, FedDH is paired with FedAD. The paper describes the two components as complementary: FedAD reduces model size and accelerates training through adaptive dropout, whereas FedDH improves model quality by weighting updates according to data heterogeneity. The paper explicitly states that FedDH’s slightly higher server-side overhead is offset by FedAD’s efficiency gains (Liu et al., 14 Jul 2025).

When dropout is applied, not all parameters are present on all clients. The paper therefore gives two aggregation cases. If a parameter survives on all selected clients,

$w$7

If it is dropped on some clients,

$w$8

In this formulation, FedDH still determines the weights $w$9, but actual parameter fusion is conditioned on whether a parameter exists in a client’s submodel. This makes FedDH a weighting rule that remains operative even when the trained submodels are partially masked by dropout rather than fully identical (Liu et al., 14 Jul 2025).

4. Theoretical role in convergence and drift control

FedDH’s theoretical justification is tied to the role of heterogeneity in optimization error. The paper introduces

$w_k$0

where $w_k$1 are aggregation weights. Theorem 2 states that if the weights are chosen as

$w_k$2

then the error upper bound is equal or smaller than with standard sample-proportional averaging

$w_k$3

This is the formal rationale for heterogeneity-aware weighting: weighting by inverse heterogeneity can tighten the optimization bound relative to purely sample-size-based aggregation (Liu et al., 14 Jul 2025).

The paper’s account of FedDH is therefore not merely heuristic. It links the aggregation rule to a heterogeneity quantity that appears in convergence analysis and uses that link to motivate inverse-weighting by client-specific distribution mismatch. This suggests that FedDH should be understood as an attempt to control client drift at the aggregation layer rather than by altering local objectives alone.

That server-side emphasis distinguishes FedDH from methods such as FedProx, FedNova, MOON, and FedDyn, which mainly modify local training or use gradient or regularization mechanisms to reduce non-IID effects. A related but technically different line of work is Fisher-Informed Parameterwise Aggregation (FIPA), which also argues that uniform client-level scalar weights are inadequate under non-IID data, but replaces them with parameter-specific matrix weights derived from low-rank Fisher or generalized Gauss–Newton curvature. FIPA is thus adaptive at the level of parameter directions, whereas FedDH is adaptive at the level of client weights (Chang et al., 20 Jan 2026).

5. Empirical evaluation and reported performance

The reported FedDH experiments use 100 clients, with 10 selected per round, on CIFAR-10, CIFAR-100, SVHN, and TinyImageNet. Models include LeNet, CNN, VGG11, and ResNet18. Non-IID partitioning is based on a Dirichlet distribution with default concentration $w_k$4. Training settings are learning rate 0.1, decay 0.99, batch size 10, local epochs $w_k$5, and 500 rounds for LeNet, CNN, and VGG and 1000 rounds for ResNet (Liu et al., 14 Jul 2025).

FedDH is compared against FedAvg, FedProx, FedNova, MOON, FedDyn, FedAS, FedKTL, and AugFL, among others in the full evaluation. On CIFAR-10 with LeNet, FedDH achieves 0.621 accuracy, compared with 0.565 for FedAvg, 0.588 for FedProx, 0.582 for FedNova, 0.610 for MOON, and 0.543 for FedDyn. The corresponding gains reported are +5.6% over FedAvg, +3.3% over FedProx, +4.0% over FedNova, +1.1% over MOON, and +7.8% over FedDyn. On CIFAR-100, the paper reports gains up to +3.4% over FedAvg, +2.3% over FedProx, +1.1% over FedNova, +2.2% over MOON, and +8.3% over FedDyn. On SVHN and TinyImageNet, FedDH yields higher accuracy than FedAvg by up to 0.9%, over FedProx by up to 1.0%, and over FedNova by up to 1.3% (Liu et al., 14 Jul 2025).

The paper also reports auxiliary findings. If client distribution metadata $w_k$6 cannot be shared, the alternative estimator FedDHE uses gradients from a small balanced public or server dataset, and FedDH and FedDHE achieve similar accuracy with a difference of less than 0.8%. Against the static baseline FedJS, FedDH outperforms by up to 2.9% on LeNet and 0.9% on CNN for CIFAR-10. In ablation on CIFAR-10 with LeNet, FedDHAD achieves 0.633 accuracy, 2446 s, FedDH achieves 0.621 accuracy, 2668 s, and FedAD achieves 0.595 accuracy, 3942 s. The paper interprets these results as evidence that dynamic aggregation contributes directly to the full system’s performance and that the combination of FedDH and FedAD improves both accuracy and time (Liu et al., 14 Jul 2025).

FedDH is not presented as an efficiency method in isolation. The reported server computation overhead is only up to 10.4% for FedDH and up to 15.6% for FedDHAD. The server overhead table includes values such as 0.086 for CIFAR-10/LeNet, 0.042 for CIFAR-10/CNN, 0.104 for CIFAR-100/LeNet, and 0.043 for CIFAR-100/CNN (Liu et al., 14 Jul 2025).

6. Relation to adjacent methods and interpretive boundaries

FedDH belongs to a broader family of aggregation methods that attempt to correct the limitations of FedAvg under heterogeneity, but its mechanism is specific. It is neither a model-heterogeneous architecture-matching method nor a synthetic-data correction method in the sense used by some server-side distillation approaches.

This boundary becomes clearer when contrasted with FedTSA, which is designed for model-heterogeneous federated learning under system heterogeneity. FedTSA clusters clients by processing capability using a proxy task, maps clusters to pruned model sizes, performs cluster-wise weight averaging among clients with identical architectures, and then performs cross-cluster aggregation by deep mutual learning on synthetic data generated with Stable Diffusion v1-4. In FedTSA, dynamic heterogeneous aggregation refers to switching aggregation modes according to whether models are homogeneous within a cluster or heterogeneous across clusters (Fan et al., 2024).

A second nearby method is DynaFed, which tackles client data heterogeneity by reserving a short early trajectory of global model snapshots, synthesizing a pseudo dataset that mimics the reserved trajectory, and then using that pseudo data to fine-tune later server aggregations. DynaFed is described as a dynamic aggregation enhancement that keeps the FedAvg backbone and adds a server-side correction learned from global dynamics rather than from client label-distribution metadata (Pi et al., 2022).

These comparisons help delimit the meaning of FedDH. In the FedDHAD paper, FedDH denotes a dynamic, heterogeneity-aware client-weighting rule based on estimated non-IID degree. It does not perform dynamic architecture assignment, cross-architecture knowledge transfer, or parameterwise curvature fusion. A plausible implication is that the phrase dynamic heterogeneous model aggregation can denote substantially different mechanisms across the federated-learning literature, even when the common objective is to move beyond uniform FedAvg-style averaging. Within that landscape, FedDH is most precisely classified as a server-side dynamic reweighting method for statistical heterogeneity (Liu et al., 14 Jul 2025).