FedDHAD: Heterogeneous FL Framework

• FedDHAD is a federated learning framework that integrates dynamic aggregation (FedDH) and adaptive dropout (FedAD) to handle both non-IID data and system heterogeneity.
• It achieves up to 6.7% higher accuracy, is up to 2.02 times faster in reaching target accuracy, and reduces computation cost by up to 15.0% compared to standard baselines.
• The framework employs bi-level optimization and device-adaptive dropout to mitigate client drift and straggler effects, ensuring efficient cross-device training.

FedDHAD is a federated learning framework designed to address two forms of heterogeneity that arise simultaneously in cross-device training: statistical heterogeneity, caused by non-IID client data, and system heterogeneity, caused by differences in computation and communication capability across devices. It combines two components—Dynamic Heterogeneous model aggregation (FedDH) and Adaptive Dropout (FedAD)—within a single server-coordinated protocol. FedDH reweights client contributions during aggregation according to a learned non-IID degree, while FedAD performs neuron-adaptive operations based on importance scores and device capability so as to reduce computation and communication and mitigate stragglers. In the reported experiments, the combined framework is stated to outperform state-of-the-art baselines in accuracy, efficiency, and computation cost, with gains of up to 6.7% higher accuracy, up to 2.02 times faster time-to-target accuracy, and up to 15.0% smaller computation cost (Liu et al., 14 Jul 2025).

1. Definition, scope, and naming

FedDHAD denotes the framework introduced in "Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout" (Liu et al., 14 Jul 2025). The acronym expands into two constituent methods: FedDH, for Dynamic Heterogeneous model aggregation, and FedAD, for Federated Adaptive Dropout. The framework is presented for horizontal cross-device FL with multiple edge devices collaboratively training a global model under non-IID data and heterogeneous device capabilities.

The central motivation is twofold. First, standard sample-size-weighted aggregation such as FedAvg can be suboptimal when local client distributions diverge substantially from the global distribution, yielding objective inconsistency and client drift. Second, full-model local training and full-model communication amplify the effect of stragglers when edge devices have limited or heterogeneous resources. FedDHAD addresses these issues jointly rather than treating them as separate optimization problems (Liu et al., 14 Jul 2025).

The term should be distinguished from similarly named but different methods. "FedDAH" refers to "Dynamic Allocation Hypernetwork with Adaptive Model Recalibration," a server-side federated continual learning method for asynchronous medical task streams rather than the heterogeneous FL framework considered here (Qi et al., 25 Mar 2025). "FedDAG" denotes "Federated DAG Structure Learning," which addresses decentralized causal DAG discovery under additive noise models and is unrelated to adaptive dropout or non-IID-aware aggregation in standard FL (Gao et al., 2021). This suggests that naming confusion can obscure substantial differences in objective, architecture, and evaluation domain.

2. Optimization target and heterogeneity model

FedDHAD adopts the standard federated objective

$F(w) = \frac{1}{n} \sum_{k=1}^{N} n_k F_k(w),$

with

$$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$

Here, $N$ is the number of clients, $n_k$ is the size of client $k$, and $w$ denotes the global model parameters (Liu et al., 14 Jul 2025).

The method is explicitly motivated by two heterogeneity regimes.

Statistical heterogeneity refers to non-IID data across edge devices. FedDHAD quantifies per-client non-IID degree through a divergence between the client label distribution $P_k$ and the global label distribution $P_m$. The paper defines

$$D^{non-IID}_k(P_k, \upsilon_k, b_k) = \upsilon_k \, \mathcal{D}_{JS}(P_k) + b_k,$$

where the Jensen–Shannon divergence is

$$\mathcal{D}_{JS}(P_k) = \frac{1}{2} \mathcal{D}_{KL}(P_k \,\|\, \overline{P}) + \frac{1}{2} \mathcal{D}_{KL}(P_m \,\|\, \overline{P}), \quad \overline{P} = \frac{1}{2}(P_k + P_m).$$

The global label distribution is defined as

$$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$0

The control parameters $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$1 are dynamically updated so that the JS-based score better aligns with the actual training impact of heterogeneity (Liu et al., 14 Jul 2025).

System heterogeneity refers to heterogeneous compute and communication capabilities. FedDHAD treats the resulting slow clients as a straggler problem. FedAD responds by reducing local model width through device-adaptive dropout, with the stated aim of equalizing training time across devices while preserving accuracy (Liu et al., 14 Jul 2025).

When label metadata cannot be shared, the framework includes an estimation variant, denoted FedDHE in the description. It estimates the label distribution from gradients on a balanced server-side small public dataset via

$$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$2

A plausible implication is that the method attempts to preserve the benefits of non-IID-aware aggregation while reducing direct exposure of client label statistics (Liu et al., 14 Jul 2025).

3. FedDH: dynamic heterogeneous model aggregation

FedDH reformulates server aggregation as a bi-level optimization problem. The inner level corresponds to local model training, while the outer level optimizes the aggregation weights $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$3. In the formulation provided,

$$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$4

where $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$5 is the set of selected clients at round $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$6, $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$7 are local models after local training, and $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$8 are aggregation weights constrained to sum to one (Liu et al., 14 Jul 2025).

The defining feature of FedDH is its inverse-non-IID weighting rule. At round $$F_k(w) = \frac{1}{n_k} \sum_{(x_k,y_k) \in \mathcal{D}_k} f(w,x_k,y_k).$$9, the server assigns

$N$0

Thus, clients with larger sample size receive more weight, but this effect is modulated by a penalty based on estimated statistical divergence from the global distribution. The method therefore departs from pure sample-size averaging and instead treats heterogeneity itself as a quantity to be optimized around (Liu et al., 14 Jul 2025).

The learnable control vectors $N$1 and $N$2, containing the per-client parameters $N$3 and $N$4, are updated through outer-level gradients:

$N$5

This converts the non-IID degree from a fixed statistic into a dynamically recalibrated quantity (Liu et al., 14 Jul 2025).

The theoretical analysis assumes $N$6-smoothness, $N$7-strong convexity, unbiased stochastic gradients, bounded local variance $N$8, and bounded expected squared gradient $N$9. Under these assumptions, Theorem 1 gives

$n_k$0

The analysis also defines

$n_k$1

and Theorem 2 states that using weights inversely proportional to $n_k$2 yields an upper bound that is less than or equal to that obtained with sample-size weights under the stated conditions. This suggests that FedDH is justified not only empirically but also through a convergence-bound argument specifically tied to non-IID-aware weighting (Liu et al., 14 Jul 2025).

4. FedAD: adaptive dropout for compute and communication heterogeneity

FedAD performs neuron/filter-adaptive dropout per device and per round. Its purpose is to reduce computation and communication, equalize local execution time across heterogeneous hardware, and mitigate stragglers without relying on uniform model truncation. The description distinguishes two importance mechanisms: HRank on feature maps for convolutional filters, and the sum of input weights for neurons in fully connected layers (Liu et al., 14 Jul 2025).

For each device $n_k$3 at round $n_k$4, the framework computes importance ranks

$n_k$5

over nodes in convolutional and fully connected layers. The dropout schedule is not updated every round. Instead, the method computes a variation term every $n_k$6 rounds using the Euclidean distance

$n_k$7

and triggers updates only when the paper’s stability conditions are met, such as no decrease after $n_k$8 rounds or after a significant reduction at a specific round. The stated objective is to avoid disrupting critical learning periods (Liu et al., 14 Jul 2025).

FedAD includes a device-level calibration heuristic based on a Hessian spectrum. For each device, a local dropout rate $n_k$9 is derived from the first index satisfying $k$0. The server aggregates these to obtain an average dropout rate for the largest device,

$k$1

and then estimates a target execution time

$k$2

Each device receives a dropout rate

$k$3

where $k$4 is the profiled local time without dropout (Liu et al., 14 Jul 2025).

Within convolutional and fully connected groups separately, the rank values are converted to per-node probabilities:

$k$5

The dropout mask is then sampled as

$k$6

Retained nodes are therefore rescaled in the standard dropout manner, while lower-importance nodes are more likely to be dropped. The framework emphasizes per-node dropout rather than whole-layer removal, because dropping entire layers can degrade accuracy significantly (Liu et al., 14 Jul 2025).

A key integration point is that FedAD changes the effective sub-model seen by each device. Aggregation must therefore operate over partially missing parameter sets. For parameters dropped on some devices, the server aggregates only over non-dropped clients and renormalizes the weights. This is the mechanism by which FedAD remains compatible with FedDH rather than becoming a separate compression stage (Liu et al., 14 Jul 2025).

5. Integrated protocol and algorithmic workflow

The full server-side procedure combines FedAD and FedDH in each round. The server selects clients $k$7, determines whether dropout rates should be updated, computes importance ranks on selected devices, derives the device-level rates, samples masks, and sends masked sub-models to participating clients. Clients perform local training for $k$8 epochs with SGD and return updated sub-models. The local update is written as

$k$9

The server then computes or estimates $w$0, evaluates $w$1, computes the aggregation weights $w$2, and aggregates with renormalization for dropped parameters (Liu et al., 14 Jul 2025).

The description reports several key hyperparameters: local epochs $w$3 such as $w$4, batch size such as $w$5, and local learning rate schedules such as $w$6 with $w$7 decay. It also lists the outer learning-rate families $w$8, decay$w$9, $P_k$0, and decay$P_k$1, as well as the dropout-scheduling parameters $P_k$2 and $P_k$3 (Liu et al., 14 Jul 2025).

The communication effect of FedAD is described as a reduction in message size roughly proportional to the fraction of active neurons or filters. Baseline FL sends the full model, whereas FedAD sends sub-models whose parameter count is reduced by dropout. The server-side overhead introduced by the bi-level updates is reported as up to 15.6% for FedDHAD and up to 10.4% for FedDH. This indicates that FedDHAD trades modest additional server computation for reduced client-side burden and faster overall convergence to target accuracy (Liu et al., 14 Jul 2025).

The framework is explicitly positioned as a joint treatment of statistical and system heterogeneity. FedDH improves aggregation under non-IID data; FedAD reduces local runtime and communication cost under device heterogeneity. Their interaction is not merely additive: FedAD changes which parameters are present on each device, and FedDH then aggregates the resulting heterogeneous sub-models using non-IID-aware weights with valid-weight renormalization (Liu et al., 14 Jul 2025).

6. Empirical evaluation and reported behavior

The experimental setup uses 100 devices total, with 10 sampled per round, and simulation on 44 Tesla V100 GPUs. The datasets are CIFAR-10, CIFAR-100, SVHN, and TinyImageNet. The models are LeNet5, CNN, VGG11, and ResNet18. The default non-IID partition uses a Dirichlet split with concentration parameter $P_k$4. Training lasts 500 rounds for LeNet, CNN, and VGG, and 1000 rounds for ResNet (Liu et al., 14 Jul 2025).

The baseline set includes FedAvg, FedProx, FedNova, MOON, FedDyn, FedDST, PruneFL, AFD, FedDrop, FjORD, and MOON+FjORD, with additional comparisons to FedAS, FedKTL, and AugFL in extended analysis. The reported metrics are accuracy, time-to-target accuracy, and computation cost measured in MFLOPs. Target accuracies include 0.54 for LeNet on CIFAR-10, 0.72 for CNN on CIFAR-10, 0.26 for LeNet on CIFAR-100, 0.41 for CNN on CIFAR-100, 0.85/0.88/0.91 for LeNet/CNN/VGG on SVHN, and 0.36 for ResNet on TinyImageNet (Liu et al., 14 Jul 2025).

The main quantitative claims are as follows.

Dimension	Reported result	Context
Accuracy	up to 6.7% higher	relative to baselines
Efficiency	up to 2.02 times faster	time-to-target accuracy
Computation cost	up to 15.0% smaller	MFLOPs than baselines

More specific comparisons are also reported. FedDH consistently improves accuracy over FedAvg, FedProx, FedNova, MOON, and FedDyn; for example, on CIFAR-10 with LeNet, the gain is stated as +5.6% versus FedAvg, and on CIFAR-100 with LeNet/CNN, up to +3.4% versus FedAvg. FedDHAD is reported to achieve 1.69 times speedup versus FedAvg, 1.51–2.57 times versus MOON on certain settings, up to 32.5% faster versus FjORD, and 51.4% versus MOON+FjORD. In the ablations, FedDHAD outperforms FedDH by accuracy up to 1.1% and time up to 14.7%, and outperforms FedAD by accuracy up to 3.8% and time up to 28.7% (Liu et al., 14 Jul 2025).

Sensitivity analyses indicate that the framework benefits more strongly under lower bandwidth, with speedup increases up to 82.7%, and under stronger non-IID heterogeneity, with accuracy advantages reported in the range +6.9% to +10.7%. In the scalability analysis with 200 devices, FedDHAD yields up to 13.5% accuracy advantage and shorter time-to-target than the 100-device runs. This suggests that the method is intended not only for moderate heterogeneity but also for settings in which per-client data become smaller and device variability becomes more consequential (Liu et al., 14 Jul 2025).

7. Assumptions, limitations, and relation to adjacent federated methods

The theoretical guarantees rely on assumptions that include $P_k$5-smoothness, $P_k$6-strong convexity, unbiased gradients, bounded variance, and bounded gradients. The paper notes that these assumptions are standard but may be restrictive for deep non-convex networks. As a result, the analysis is presented as guidance rather than a complete account of deep-network optimization behavior (Liu et al., 14 Jul 2025).

Several practical limitations are identified. Sharing label distributions $P_k$7 may raise privacy concerns; the gradient-based estimation alternative using a balanced public dataset is proposed to avoid sharing private metadata, and the reported performance difference between FedDH and FedDHE is stated to be small, specifically less than 0.8% in certain settings. Parameter sensitivity remains relevant: the aggregation control learning rates and decay schedules require tuning, and the dropout scheduling thresholds and importance scoring also require calibration. For large models such as Transformers, the method is described as adaptable—for example through dropout on dense layers and attention heads—but detailed implementation and empirical validation for LLMs are future work (Liu et al., 14 Jul 2025).

Relative to prior FL methods, the method occupies a specific position. FedAvg ignores non-IID degree in aggregation; FedDH reweights client models inversely to non-IID degree. FedProx and FedDyn mitigate drift through proximal or dynamic local objectives, whereas FedDH operates on server-side aggregation and is described as complementary. FedNova, MOON, and SCAFFOLD target statistical heterogeneity through normalization, contrastive learning, or variance reduction; FedDH instead targets aggregation. On the model-reduction side, FedDrop, AFD, FjORD, FedDST, and PruneFL reduce model size or communication, but FedAD differs by combining per-node importance-aware dropout, device-adaptive rates, and stable scheduling (Liu et al., 14 Jul 2025).

A common misconception is to treat FedDHAD as merely a compressed or pruned variant of FedAvg. The method description does not support that reading. FedDH is a non-IID-aware bi-level aggregation mechanism with learnable control parameters, and FedAD is a device-adaptive dropout mechanism tied to execution-time equalization and sub-model aggregation. Another misconception is to conflate FedDHAD with FedDAH or FedDAG because of acronym similarity. The former is a heterogeneous FL framework for non-IID data and stragglers (Liu et al., 14 Jul 2025), the latter a federated continual learning framework for asynchronous medical tasks (Qi et al., 25 Mar 2025), and the third a federated causal structure learning method under DAG and ANM assumptions (Gao et al., 2021).