---
title: Adaptive Dropout (FedAD) in Federated Learning
url: https://www.emergentmind.com/topics/adaptive-dropout-fedad
type: topic
---

# Adaptive Dropout (FedAD) in Federated Learning

Searching arXiv for recent papers on federated adaptive dropout and closely related methods.
Adaptive Dropout (FedAD) denotes a class of federated learning methods in which the dropout mechanism is made adaptive to device heterogeneity, data heterogeneity, or learned importance scores, so that each client trains a sub-model rather than the full global model. In the most explicit formulation, FedAD is the dropout component of the FedDHAD framework, where it performs neuron-adaptive operations in response to heterogeneous devices, while a companion aggregation method, FedDH, adjusts client aggregation weights under non-IID data [2507.10430]. More broadly, closely related formulations appear as Adaptive Federated Dropout (AFD), which adaptively selects subsets of activations for client-side training based on loss-driven score maps [2011.04050], as resource-aware Federated Dropout (FedDrop), which assigns heterogeneous dropout rates according to communication and computation constraints [2109.15258], and as Bayesian Inference-based Adaptive Dropout (FedBIAD), which adaptively drops weight rows according to loss-correlated importance indicators in a variational Bayesian framework [2307.07172]. Across these formulations, the common objective is to reduce communication and computation while preserving, and sometimes improving, model quality under heterogeneous clients and data.

## 1. Conceptual scope and historical positioning

In federated learning, dropout differs from classical centralized dropout because the dropped components are used to define a client-side subnetwork that is actually transmitted, trained, and aggregated. In FedDrop, the server generates different subnets from the global model and assigns one subnet per device, with dropout rates adapted to channel and computation state [2109.15258]. AFD extends this principle by making the selection of activations adaptive rather than random, using score maps derived from loss improvements [2011.04050]. FedAD, as named in FedDHAD, further emphasizes heterogeneous devices: it computes importance ranks for filters and neurons and combines them with execution-time-based device constraints to produce per-node dropout probabilities [2507.10430].

This lineage distinguishes FedAD from two adjacent uses of the phrase “adaptive dropout.” First, outside federated learning, adaptive dropout often refers to data- or gradient-dependent masking inside centralized models, such as Standout-style mechanisms or gradient-based Dynamic DropConnect [2502.19948]. Second, in asynchronous decentralized federated learning, “adaptive strategies” for persistent client dropout refer to reconstructing missing clients rather than adaptively dropping neurons or filters [2508.01807]. In the federated model-compression literature, however, FedAD most precisely denotes adaptive, structured, client-aware submodel selection [2507.10430].

A useful organizing distinction is that FedDrop is primarily resource-aware, AFD is loss-adaptive, and FedBIAD is Bayesian importance-adaptive. FedAD in FedDHAD combines resource awareness with structural importance estimation, and is explicitly coupled to non-IID-aware aggregation [2507.10430].

## 2. Core mechanism in FedDHAD

FedAD in FedDHAD is described as Algorithm 2 and takes as input the current global round \(t\), the global model \(w^*\), the sets of fully connected layers \(FL\) and convolutional layers \(CL\), the participating devices \(S^t\), the full device set \(\mathcal{K}\), and a subset \(SD^t \subseteq \mathcal{K}\) used to estimate pruning rates. Its output is a set of sub-models \(w'^t = \{w'^t_k \mid k \in S^t\}\) for the selected devices [2507.10430].

The method separates into two phases. The first is a dropout-rate update phase, executed only when needed because rank and Hessian computation is expensive. The trigger is based on a model variation metric
\[
\Delta^t = \sqrt{(R^t - R^{t-h})^2},
\]
and dropout rates are updated if there is no decrease in variation after \(\mathscr{C}\) rounds [2507.10430]. The second phase performs mask sampling and subnet construction for participating clients every round.

FedAD computes importance ranks \(R^t_k = \{r^{t,i}_{k,l} \mid l \in CL \cup FL\}\) for each device in \(SD^t\). For convolutional filters, the rank is based on feature map statistics as in HRank. For fully connected neurons, the rank is based on weights, specifically the sum of the neuron’s input parameters in the weight matrix [2507.10430]. These ranks are then aggregated across devices as
\[
R^t = \sum_{k \in SD^t} \frac{n_k}{\sum_{k' \in SD^t} n_{k'}} R^t_k,
\]
which reduces the sensitivity of node importance estimation to local non-IID effects [2507.10430].

At the device level, FedAD derives a base dropout rate from local second-order information. For device \(k\), after local parameters move from \(W_k\) to \(W'_k\), it defines
\[
\Delta_k = W_k - W'_k,
\]
computes the Hessian \(H(W'_k)\), sorts its eigenvalues \(\{\mathscr{E}_k^m\}\), and identifies the first index \(m_k\) satisfying
\[
\mathscr{E}_{m_k+1} - \mathscr{E}_{m_k} > 4\mathscr{L}_k,
\]
where \(\mathscr{L}_k\) is the Lipschitz constant of \(B_k(\Delta_k) = H(W'_k) - \nabla L(W_k)\). The device-specific base dropout rate is then
\[
d_k = \frac{m_k}{h_k},
\]
with \(h_k\) the rank of the Hessian [2507.10430]. This uses a curvature-based separation between important and less-important directions.

FedAD next converts these local rates into a system-level time target. It computes a data-size-weighted average pruning rate
\[
d_{\text{avg}} = \sum_{k \in SD^t} \frac{n_k}{\sum_{k' \in SD^t} n_{k'}} d_k,
\]
and sets
\[
ET_{\max} = ET^o_{\max} \cdot d_{\text{avg}},
\]
where \(ET^o_{\max}\) is the profiled execution time per round of the device with the largest dataset using the full model [2507.10430]. Then each device receives a resource-aware base dropout rate
\[
d_k^* = \min\left\{\frac{ET_{\max}}{ET_k^o}, 1\right\},
\]
with \(ET_k^o\) the profiled full-model execution time on device \(k\) [2507.10430].

Finally, FedAD allocates this device-level dropout budget across nodes using importance-weighted softmax allocation:
\[
d_k^i = \frac{e^{-r_k^{t,i}}}{\sum_{j \in CL~\text{or}~FL} e^{-r_k^{t,j}}} \, N_{CL~\text{or}~FL}\, d_k^*.
\]
Large ranks imply smaller dropout probabilities, so more important filters or neurons are less likely to be dropped [2507.10430]. This computation is done separately for convolutional and fully connected layers.

## 3. Mask sampling, submodel construction, and FL integration

Given the per-node dropout probabilities \(d_k^i\), FedAD samples masks with inverted-dropout scaling:
\[
m_k^{i,t} =
\begin{cases}
\frac{1}{1-d_k^i}, & \text{with probability } 1-d_k^i, \\
0, & \text{with probability } d_k^i.
\end{cases}
\]
Using these masks, the server generates the submodel \(w_k'^t\) for device \(k\) [2507.10430]. The paper explicitly notes that dropping an entire layer, i.e. setting \(m_k^{i,t}=0\) for all \(i\) in a layer, is a special case but may cause significant accuracy degradation, so FedAD keeps dropout at neuron/filter granularity rather than removing whole layers [2507.10430].

Within FedDHAD, FedAD occupies the model-adaptation part of the communication round. The server first performs dropout to generate client-specific subnets, then sends these subnets to selected clients, each client trains only its subnetwork for \(\tau\) epochs, uploads the updated subnet, and the server aggregates using FedDH [2507.10430]. The global model remains dense on the server; local models are sparse subnetworks whose structure may change by client and by round.

A closely related but simpler mechanism appears in FedDrop. There, dropout is applied only to fully connected layers, and for a device-specific dropout rate \(p_k\), the subnet parameter count and computation load scale as
\[
M_k = M_{\text{conv}} + (1-p_k)^2 M_{\text{full}},
\]
\[
C_k = C_{\text{conv}} + (1-p_k)^2 C_{\text{full}},
\]
yielding a reduction factor of roughly \((1-p_k)^2\) in both communication and computation for the fully connected portion [2109.15258]. FedAD generalizes this resource-aware logic to both convolutional and fully connected layers and replaces random selection with importance-aware selection [2507.10430].

AFD, by contrast, adapts the subset structure via score maps rather than Hessian-based rates. In Multi-Model AFD, the server maintains a score map \(\mathcal{M}_c\) per client, records previously successful activation subsets, and updates activation scores whenever a client’s local loss improves; subsequent submodels are sampled with probabilities proportional to these scores [2011.04050]. In Single-Model AFD, the same logic is applied globally using average loss across participating clients [2011.04050]. This is adaptive, but the adaptation is driven by loss improvement rather than curvature and runtime profiling.

## 4. Device heterogeneity, non-IID data, and communication efficiency

FedAD is explicitly designed for heterogeneous devices. The defining device statistic is the profiled execution time \(ET_k^o\), which may reflect CPU/GPU speed, memory, and possibly network throughput. Because \(d_k^* = \min\{ET_{\max}/ET_k^o,1\}\), slower devices receive larger dropout rates and therefore smaller submodels, while faster devices receive smaller dropout rates and train denser subnetworks [2507.10430]. This narrows the distribution of local execution times and directly targets straggler mitigation.

The second heterogeneity axis is non-IID data. In FedDHAD this is addressed primarily by FedDH, which reweights model aggregation according to data heterogeneity, but FedAD also contributes by preserving important model components under heterogeneous device constraints [2507.10430]. The paper’s central claim is that the combination of FedDH and FedAD significantly outperforms state-of-the-art methods in accuracy, efficiency, and computation cost, with reported gains of up to 6.7% higher accuracy, up to 2.02 times faster efficiency, and up to 15.0% smaller computation cost [2507.10430].

FedDrop provides the canonical analytic resource-allocation perspective. With a target synchronous round latency \(T\), it derives the minimum feasible device-specific dropout rate
\[
p_k^{\min} = 1 - \sqrt{\frac{T - T_k^{\text{conv}}}{T_k^{\text{full}}}},
\]
where \(T_k^{\text{conv}}\) and \(T_k^{\text{full}}\) are the latency contributions from convolutional and fully connected layers of the full model [2109.15258]. This analytic dependence on bandwidth, spectrum efficiency, compute speed, and local dataset size is not used verbatim in FedAD, but it establishes the same principle: heterogeneous dropout rates are an effective device-aware allocation mechanism [2109.15258].

FedBIAD addresses communication from a different angle. It drops weight rows or filters according to an importance indicator \(E^k\) built from local training-loss trends, and clients upload only the non-dropped rows of their variational means, producing a 2x uplink reduction with an accuracy increase of up to 2.41% and up to 72% decrease in training time [2307.07172]. A plausible implication is that FedAD, FedDrop, AFD, and FedBIAD occupy different points on a shared design spectrum: resource-aware subnet selection, loss-adaptive subnet selection, and Bayesian importance-adaptive structured dropout.

## 5. Empirical results and comparative performance

FedAD is evaluated both as an isolated method and as part of FedDHAD on CIFAR-10, CIFAR-100, SVHN, and TinyImageNet with LeNet, CNN, VGG11, and ResNet18 [2507.10430]. On CIFAR-10 with LeNet, FedAD achieves accuracy \(0.595\) versus FedAvg \(0.565\), FedDrop \(0.585\), AFD \(0.511\), and FjORD \(0.593\) [2507.10430]. On CIFAR-100 with LeNet, it achieves \(0.295\) versus FedAvg \(0.263\), FedDrop \(0.254\), AFD \(0.270\), and FjORD \(0.292\) [2507.10430]. On SVHN with VGG, FedAD achieves \(0.9309\) versus FedAvg \(0.9286\) [2507.10430]. On TinyImageNet with ResNet18, FedAD reaches \(0.3763\), second only to FedDHAD at \(0.3765\), while also having the shortest time to target accuracy and the lowest MFLOPs among the compared methods [2507.10430].

The time-to-target-accuracy gains are substantial. For LeNet on CIFAR-10 at target accuracy 0.54, the reported training times are 2366 s for FedAvg, 3993 s for FedDrop, 1663 s for FjORD, and 1394 s for FedAD, making FedAD 41% faster than FedAvg in that setting [2507.10430]. More generally, the paper states that FedAD leads to up to 31.0% shorter training time than FedAvg and up to 55.1% shorter than FedDrop [2507.10430].

AFD emphasizes convergence-time reduction under communication bottlenecks. Combined with DGC and downlink quantization, AFD reports up to 57x reduction in convergence time relative to uncompressed FedAvg, while also improving model generalization by up to 1.7% [2011.04050]. On non-IID FEMNIST, AFD + DGC reaches \(80.6\% \pm 0.14\%\) accuracy versus \(78.9\% \pm 0.12\%\) with no compression, while reducing time from 3233.2 min to 61.7 min [2011.04050]. On non-IID Shakespeare, it reaches \(54.4\% \pm 0.36\%\) versus \(53.1\% \pm 0.22\%\), with time reduced from 762.5 min to 13.3 min [2011.04050]. These are not FedAD results in the strict FedDHAD sense, but they establish the empirical viability of adaptive federated dropout as a broader family [2011.04050].

FedDrop reports that on CNNCifar, dropout rates up to 60% can reduce latency and bandwidth usage and may improve accuracy relative to full-model FL in overfitting regimes, while on CNNMnist the method causes only slight accuracy loss for moderate \(p\) [2109.15258]. The paper also reports that FedDrop consistently outperforms uniform dropout because heterogeneous subnets provide better model coverage and diversity [2109.15258].

FedBIAD further shows that adaptive dropout can improve both communication and accuracy on image classification and next-word prediction. Relative to FedAvg, it reports 95.20% versus 95.06% on MNIST with 1.25x upload saving, 83.59% versus 81.18% on FMNIST with 2x saving, and 33.93% versus 31.68% on Reddit with 2x saving [2307.07172]. These results reinforce a general pattern: adaptive rather than fixed subnet selection can improve the efficiency–accuracy trade-off.

## 6. Theoretical interpretations and broader variants

FedAD in FedDHAD does not come with a dedicated convergence theorem; the paper states that the formal convergence analysis is mainly for FedDH, not for FedAD [2507.10430]. The justification for FedAD is therefore heuristic and structural: Hessian-spectrum-based rate selection, rank-based node importance, and known dropout regularization effects [2507.10430].

FedBIAD provides the strongest formal theory among the federated adaptive-dropout methods in the supplied corpus. It models each row or filter via a spike-and-slab variational distribution
\[
\tilde{\pi}(w_j) = \beta_j \mathcal{N}(\mu_j,\tilde{s}^2 I) + (1-\beta_j)\delta(0),
\]
updates the variational mean by masked gradients, and uses loss-trend-driven pattern exploration followed by score-based deterministic selection [2307.07172]. Under Lipschitz activations and bounded optimal weights, it proves an upper bound on average generalization error and shows a minimax-optimal rate up to a squared logarithmic factor [2307.07172]. This suggests that Bayesian adaptive dropout offers one route to giving FedAD-style mechanisms formal statistical guarantees.

A different theoretical lens comes from the duality between adaptive dropout and regularization. In the linear case, any dropout strategy that adapts to the weights in a monotonic way corresponds to an effective subquadratic regularization penalty and therefore leads to sparse solutions [2106.07769]. This suggests that resource-aware and importance-aware federated dropout schemes can often be reinterpreted as inducing implicit sparse regularizers. A plausible implication is that FedAD’s node-wise dropout allocation may be viewed not only as communication control, but also as a structured regularization mechanism.

Dynamic DropConnect offers a non-federated but conceptually relevant analogue. It computes per-edge drop probabilities from gradient magnitudes and reports empirical superiority over Dropout, DropConnect, and Standout on MNIST, CIFAR-10, CIFAR-100, and NORB [2502.19948]. The paper also states that its core mechanism can be naturally adapted to an “Adaptive Dropout (FedAD)” framework in federated learning, because the required per-weight gradients and layer statistics can be computed locally without cross-client aggregation [2502.19948]. This is not a realized FedAD implementation, but it identifies gradient-aware masking as another viable design axis.

## 6. Limitations, assumptions, and open directions

FedAD in FedDHAD assumes centralized orchestration: the server must coordinate rank aggregation, subnet generation, and device profiling [2507.10430]. Fully decentralized topologies are not directly addressed [2507.10430]. The method also depends on Hessian and Lipschitz estimation, which can be computationally heavy and may require approximations in large models [2507.10430]. The experiments focus on CNN-style architectures and ResNet18, with only a brief suggestion that Transformers could use heads or feed-forward components as nodes [2507.10430].

AFD has different limitations. Multi-Model AFD requires enough client participation frequency to maintain useful per-client score maps; when the fraction of clients per round is small, its score estimates become noisy [2011.04050]. Single-Model AFD is less suitable for strongly non-IID settings because average loss across varying client subsets is an unstable signal for global score-map updates [2011.04050]. FedDrop, meanwhile, applies dropout only to fully connected layers, which is effective when those layers dominate model size but less comprehensive for architectures where convolutional or attention layers dominate [2109.15258].

FedBIAD introduces additional concerns. Its adaptive dropout is built on model and loss statistics, and while theoretically grounded, it adds local computational overhead for score updates and stochastic variational training [2307.07172]. Its use of structured dropping patterns is also primarily evaluated in centralized-server FL, not decentralized or asynchronous settings [2307.07172].

Across all these formulations, one persistent open question is how best to combine device-aware adaptation, data-aware adaptation, and theoretically grounded sparsity control. FedDHAD combines device-aware FedAD with heterogeneity-aware FedDH [2507.10430]. AFD combines adaptive subnet selection with gradient compression [2011.04050]. FedBIAD combines loss-aware adaptive dropout with Bayesian inference [2307.07172]. This suggests a broader research direction toward multi-level adaptive dropout in federated systems, where client selection, subnet structure, aggregation weights, and communication compression are co-designed rather than optimized separately.

Source: https://www.emergentmind.com/topics/adaptive-dropout-fedad