Adaptive Dropout (FedAD) in Federated Learning

• Adaptive Dropout (FedAD) is an adaptive federated learning method that customizes dropout based on device capacity, loss-driven score maps, and data heterogeneity.
• It integrates resource-aware dropout allocation with non-IID-aware aggregation to generate client-specific subnetworks, optimizing efficiency and model quality.
• Empirical results show that FedAD improves accuracy and convergence speed while reducing computation time compared to traditional federated learning methods.

Searching arXiv for papers on federated adaptive dropout and closely related methods. Adaptive Dropout (FedAD) denotes a class of federated learning methods in which the dropout mechanism is made adaptive to device heterogeneity, data heterogeneity, or learned importance scores, so that each client trains a sub-model rather than the full global model. In the most explicit formulation, FedAD is the dropout component of the FedDHAD framework, where it performs neuron-adaptive operations in response to heterogeneous devices, while a companion aggregation method, FedDH, adjusts client aggregation weights under non-IID data (Liu et al., 14 Jul 2025). More broadly, closely related formulations appear as Adaptive Federated Dropout (AFD), which adaptively selects subsets of activations for client-side training based on loss-driven score maps (Bouacida et al., 2020), as resource-aware Federated Dropout (FedDrop), which assigns heterogeneous dropout rates according to communication and computation constraints (Wen et al., 2021), and as Bayesian Inference-based Adaptive Dropout (FedBIAD), which adaptively drops weight rows according to loss-correlated importance indicators in a variational Bayesian framework (Xue et al., 2023). Across these formulations, the common objective is to reduce communication and computation while preserving, and sometimes improving, model quality under heterogeneous clients and data.

1. Conceptual scope and historical positioning

In federated learning, dropout differs from classical centralized dropout because the dropped components are used to define a client-side subnetwork that is actually transmitted, trained, and aggregated. In FedDrop, the server generates different subnets from the global model and assigns one subnet per device, with dropout rates adapted to channel and computation state (Wen et al., 2021). AFD extends this principle by making the selection of activations adaptive rather than random, using score maps derived from loss improvements (Bouacida et al., 2020). FedAD, as named in FedDHAD, further emphasizes heterogeneous devices: it computes importance ranks for filters and neurons and combines them with execution-time-based device constraints to produce per-node dropout probabilities (Liu et al., 14 Jul 2025).

This lineage distinguishes FedAD from two adjacent uses of the phrase “adaptive dropout.” First, outside federated learning, adaptive dropout often refers to data- or gradient-dependent masking inside centralized models, such as Standout-style mechanisms or gradient-based Dynamic DropConnect (Yang et al., 27 Feb 2025). Second, in asynchronous decentralized federated learning, “adaptive strategies” for persistent client dropout refer to reconstructing missing clients rather than adaptively dropping neurons or filters (Stępka et al., 3 Aug 2025). In the federated model-compression literature, however, FedAD most precisely denotes adaptive, structured, client-aware submodel selection (Liu et al., 14 Jul 2025).

A useful organizing distinction is that FedDrop is primarily resource-aware, AFD is loss-adaptive, and FedBIAD is Bayesian importance-adaptive. FedAD in FedDHAD combines resource awareness with structural importance estimation, and is explicitly coupled to non-IID-aware aggregation (Liu et al., 14 Jul 2025).

2. Core mechanism in FedDHAD

FedAD in FedDHAD is described as Algorithm 2 and takes as input the current global round $t$, the global model $w^*$, the sets of fully connected layers $FL$ and convolutional layers $CL$, the participating devices $S^t$, the full device set $\mathcal{K}$, and a subset $SD^t \subseteq \mathcal{K}$ used to estimate pruning rates. Its output is a set of sub-models $w'^t = \{w'^t_k \mid k \in S^t\}$ for the selected devices (Liu et al., 14 Jul 2025).

The method separates into two phases. The first is a dropout-rate update phase, executed only when needed because rank and Hessian computation is expensive. The trigger is based on a model variation metric

$\Delta^t = \sqrt{(R^t - R^{t-h})^2},$

and dropout rates are updated if there is no decrease in variation after $\mathscr{C}$ rounds (Liu et al., 14 Jul 2025). The second phase performs mask sampling and subnet construction for participating clients every round.

FedAD computes importance ranks $w^*$0 for each device in $w^*$1. For convolutional filters, the rank is based on feature map statistics as in HRank. For fully connected neurons, the rank is based on weights, specifically the sum of the neuron’s input parameters in the weight matrix (Liu et al., 14 Jul 2025). These ranks are then aggregated across devices as

$w^*$2

which reduces the sensitivity of node importance estimation to local non-IID effects (Liu et al., 14 Jul 2025).

At the device level, FedAD derives a base dropout rate from local second-order information. For device $w^*$3, after local parameters move from $w^*$4 to $w^*$5, it defines

$w^*$6

computes the Hessian $w^*$7, sorts its eigenvalues $w^*$8, and identifies the first index $w^*$9 satisfying

$FL$0

where $FL$1 is the Lipschitz constant of $FL$2. The device-specific base dropout rate is then

$FL$3

with $FL$4 the rank of the Hessian (Liu et al., 14 Jul 2025). This uses a curvature-based separation between important and less-important directions.

FedAD next converts these local rates into a system-level time target. It computes a data-size-weighted average pruning rate

$FL$5

and sets

$FL$6

where $FL$7 is the profiled execution time per round of the device with the largest dataset using the full model (Liu et al., 14 Jul 2025). Then each device receives a resource-aware base dropout rate

$FL$8

with $FL$9 the profiled full-model execution time on device $CL$0 (Liu et al., 14 Jul 2025).

Finally, FedAD allocates this device-level dropout budget across nodes using importance-weighted softmax allocation: $CL$1 Large ranks imply smaller dropout probabilities, so more important filters or neurons are less likely to be dropped (Liu et al., 14 Jul 2025). This computation is done separately for convolutional and fully connected layers.

3. Mask sampling, submodel construction, and FL integration

Given the per-node dropout probabilities $CL$2, FedAD samples masks with inverted-dropout scaling: $CL$3 Using these masks, the server generates the submodel $CL$4 for device $CL$5 (Liu et al., 14 Jul 2025). The paper explicitly notes that dropping an entire layer, i.e. setting $CL$6 for all $CL$7 in a layer, is a special case but may cause significant accuracy degradation, so FedAD keeps dropout at neuron/filter granularity rather than removing whole layers (Liu et al., 14 Jul 2025).

Within FedDHAD, FedAD occupies the model-adaptation part of the communication round. The server first performs dropout to generate client-specific subnets, then sends these subnets to selected clients, each client trains only its subnetwork for $CL$8 epochs, uploads the updated subnet, and the server aggregates using FedDH (Liu et al., 14 Jul 2025). The global model remains dense on the server; local models are sparse subnetworks whose structure may change by client and by round.

A closely related but simpler mechanism appears in FedDrop. There, dropout is applied only to fully connected layers, and for a device-specific dropout rate $CL$9, the subnet parameter count and computation load scale as

$S^t$0

$S^t$1

yielding a reduction factor of roughly $S^t$2 in both communication and computation for the fully connected portion (Wen et al., 2021). FedAD generalizes this resource-aware logic to both convolutional and fully connected layers and replaces random selection with importance-aware selection (Liu et al., 14 Jul 2025).

AFD, by contrast, adapts the subset structure via score maps rather than Hessian-based rates. In Multi-Model AFD, the server maintains a score map $S^t$3 per client, records previously successful activation subsets, and updates activation scores whenever a client’s local loss improves; subsequent submodels are sampled with probabilities proportional to these scores (Bouacida et al., 2020). In Single-Model AFD, the same logic is applied globally using average loss across participating clients (Bouacida et al., 2020). This is adaptive, but the adaptation is driven by loss improvement rather than curvature and runtime profiling.

4. Device heterogeneity, non-IID data, and communication efficiency

FedAD is explicitly designed for heterogeneous devices. The defining device statistic is the profiled execution time $S^t$4, which may reflect CPU/GPU speed, memory, and possibly network throughput. Because $S^t$5, slower devices receive larger dropout rates and therefore smaller submodels, while faster devices receive smaller dropout rates and train denser subnetworks (Liu et al., 14 Jul 2025). This narrows the distribution of local execution times and directly targets straggler mitigation.

The second heterogeneity axis is non-IID data. In FedDHAD this is addressed primarily by FedDH, which reweights model aggregation according to data heterogeneity, but FedAD also contributes by preserving important model components under heterogeneous device constraints (Liu et al., 14 Jul 2025). The paper’s central claim is that the combination of FedDH and FedAD significantly outperforms state-of-the-art methods in accuracy, efficiency, and computation cost, with reported gains of up to 6.7% higher accuracy, up to 2.02 times faster efficiency, and up to 15.0% smaller computation cost (Liu et al., 14 Jul 2025).

FedDrop provides the canonical analytic resource-allocation perspective. With a target synchronous round latency $S^t$6, it derives the minimum feasible device-specific dropout rate

$S^t$7

where $S^t$8 and $S^t$9 are the latency contributions from convolutional and fully connected layers of the full model (Wen et al., 2021). This analytic dependence on bandwidth, spectrum efficiency, compute speed, and local dataset size is not used verbatim in FedAD, but it establishes the same principle: heterogeneous dropout rates are an effective device-aware allocation mechanism (Wen et al., 2021).

FedBIAD addresses communication from a different angle. It drops weight rows or filters according to an importance indicator $\mathcal{K}$0 built from local training-loss trends, and clients upload only the non-dropped rows of their variational means, producing a 2x uplink reduction with an accuracy increase of up to 2.41% and up to 72% decrease in training time (Xue et al., 2023). A plausible implication is that FedAD, FedDrop, AFD, and FedBIAD occupy different points on a shared design spectrum: resource-aware subnet selection, loss-adaptive subnet selection, and Bayesian importance-adaptive structured dropout.

5. Empirical results and comparative performance

FedAD is evaluated both as an isolated method and as part of FedDHAD on CIFAR-10, CIFAR-100, SVHN, and TinyImageNet with LeNet, CNN, VGG11, and ResNet18 (Liu et al., 14 Jul 2025). On CIFAR-10 with LeNet, FedAD achieves accuracy $\mathcal{K}$1 versus FedAvg $\mathcal{K}$2, FedDrop $\mathcal{K}$3, AFD $\mathcal{K}$4, and FjORD $\mathcal{K}$5 (Liu et al., 14 Jul 2025). On CIFAR-100 with LeNet, it achieves $\mathcal{K}$6 versus FedAvg $\mathcal{K}$7, FedDrop $\mathcal{K}$8, AFD $\mathcal{K}$9, and FjORD $SD^t \subseteq \mathcal{K}$0 (Liu et al., 14 Jul 2025). On SVHN with VGG, FedAD achieves $SD^t \subseteq \mathcal{K}$1 versus FedAvg $SD^t \subseteq \mathcal{K}$2 (Liu et al., 14 Jul 2025). On TinyImageNet with ResNet18, FedAD reaches $SD^t \subseteq \mathcal{K}$3, second only to FedDHAD at $SD^t \subseteq \mathcal{K}$4, while also having the shortest time to target accuracy and the lowest MFLOPs among the compared methods (Liu et al., 14 Jul 2025).

The time-to-target-accuracy gains are substantial. For LeNet on CIFAR-10 at target accuracy 0.54, the reported training times are 2366 s for FedAvg, 3993 s for FedDrop, 1663 s for FjORD, and 1394 s for FedAD, making FedAD 41% faster than FedAvg in that setting (Liu et al., 14 Jul 2025). More generally, the paper states that FedAD leads to up to 31.0% shorter training time than FedAvg and up to 55.1% shorter than FedDrop (Liu et al., 14 Jul 2025).

AFD emphasizes convergence-time reduction under communication bottlenecks. Combined with DGC and downlink quantization, AFD reports up to 57x reduction in convergence time relative to uncompressed FedAvg, while also improving model generalization by up to 1.7% (Bouacida et al., 2020). On non-IID FEMNIST, AFD + DGC reaches $SD^t \subseteq \mathcal{K}$5 accuracy versus $SD^t \subseteq \mathcal{K}$6 with no compression, while reducing time from 3233.2 min to 61.7 min (Bouacida et al., 2020). On non-IID Shakespeare, it reaches $SD^t \subseteq \mathcal{K}$7 versus $SD^t \subseteq \mathcal{K}$8, with time reduced from 762.5 min to 13.3 min (Bouacida et al., 2020). These are not FedAD results in the strict FedDHAD sense, but they establish the empirical viability of adaptive federated dropout as a broader family (Bouacida et al., 2020).

FedDrop reports that on CNNCifar, dropout rates up to 60% can reduce latency and bandwidth usage and may improve accuracy relative to full-model FL in overfitting regimes, while on CNNMnist the method causes only slight accuracy loss for moderate $SD^t \subseteq \mathcal{K}$9 (Wen et al., 2021). The paper also reports that FedDrop consistently outperforms uniform dropout because heterogeneous subnets provide better model coverage and diversity (Wen et al., 2021).

FedBIAD further shows that adaptive dropout can improve both communication and accuracy on image classification and next-word prediction. Relative to FedAvg, it reports 95.20% versus 95.06% on MNIST with 1.25x upload saving, 83.59% versus 81.18% on FMNIST with 2x saving, and 33.93% versus 31.68% on Reddit with 2x saving (Xue et al., 2023). These results reinforce a general pattern: adaptive rather than fixed subnet selection can improve the efficiency–accuracy trade-off.

6. Theoretical interpretations and broader variants

FedAD in FedDHAD does not come with a dedicated convergence theorem; the paper states that the formal convergence analysis is mainly for FedDH, not for FedAD (Liu et al., 14 Jul 2025). The justification for FedAD is therefore heuristic and structural: Hessian-spectrum-based rate selection, rank-based node importance, and known dropout regularization effects (Liu et al., 14 Jul 2025).

FedBIAD provides the strongest formal theory among the federated adaptive-dropout methods in the supplied corpus. It models each row or filter via a spike-and-slab variational distribution

$w'^t = \{w'^t_k \mid k \in S^t\}$0

updates the variational mean by masked gradients, and uses loss-trend-driven pattern exploration followed by score-based deterministic selection (Xue et al., 2023). Under Lipschitz activations and bounded optimal weights, it proves an upper bound on average generalization error and shows a minimax-optimal rate up to a squared logarithmic factor (Xue et al., 2023). This suggests that Bayesian adaptive dropout offers one route to giving FedAD-style mechanisms formal statistical guarantees.

A different theoretical lens comes from the duality between adaptive dropout and regularization. In the linear case, any dropout strategy that adapts to the weights in a monotonic way corresponds to an effective subquadratic regularization penalty and therefore leads to sparse solutions (LeJeune et al., 2021). This suggests that resource-aware and importance-aware federated dropout schemes can often be reinterpreted as inducing implicit sparse regularizers. A plausible implication is that FedAD’s node-wise dropout allocation may be viewed not only as communication control, but also as a structured regularization mechanism.

Dynamic DropConnect offers a non-federated but conceptually relevant analogue. It computes per-edge drop probabilities from gradient magnitudes and reports empirical superiority over Dropout, DropConnect, and Standout on MNIST, CIFAR-10, CIFAR-100, and NORB (Yang et al., 27 Feb 2025). The paper also states that its core mechanism can be naturally adapted to an “Adaptive Dropout (FedAD)” framework in federated learning, because the required per-weight gradients and layer statistics can be computed locally without cross-client aggregation (Yang et al., 27 Feb 2025). This is not a realized FedAD implementation, but it identifies gradient-aware masking as another viable design axis.

6. Limitations, assumptions, and open directions

FedAD in FedDHAD assumes centralized orchestration: the server must coordinate rank aggregation, subnet generation, and device profiling (Liu et al., 14 Jul 2025). Fully decentralized topologies are not directly addressed (Liu et al., 14 Jul 2025). The method also depends on Hessian and Lipschitz estimation, which can be computationally heavy and may require approximations in large models (Liu et al., 14 Jul 2025). The experiments focus on CNN-style architectures and ResNet18, with only a brief suggestion that Transformers could use heads or feed-forward components as nodes (Liu et al., 14 Jul 2025).

AFD has different limitations. Multi-Model AFD requires enough client participation frequency to maintain useful per-client score maps; when the fraction of clients per round is small, its score estimates become noisy (Bouacida et al., 2020). Single-Model AFD is less suitable for strongly non-IID settings because average loss across varying client subsets is an unstable signal for global score-map updates (Bouacida et al., 2020). FedDrop, meanwhile, applies dropout only to fully connected layers, which is effective when those layers dominate model size but less comprehensive for architectures where convolutional or attention layers dominate (Wen et al., 2021).

FedBIAD introduces additional concerns. Its adaptive dropout is built on model and loss statistics, and while theoretically grounded, it adds local computational overhead for score updates and stochastic variational training (Xue et al., 2023). Its use of structured dropping patterns is also primarily evaluated in centralized-server FL, not decentralized or asynchronous settings (Xue et al., 2023).

Across all these formulations, one persistent open question is how best to combine device-aware adaptation, data-aware adaptation, and theoretically grounded sparsity control. FedDHAD combines device-aware FedAD with heterogeneity-aware FedDH (Liu et al., 14 Jul 2025). AFD combines adaptive subnet selection with gradient compression (Bouacida et al., 2020). FedBIAD combines loss-aware adaptive dropout with Bayesian inference (Xue et al., 2023). This suggests a broader research direction toward multi-level adaptive dropout in federated systems, where client selection, subnet structure, aggregation weights, and communication compression are co-designed rather than optimized separately.