Adaptive Federated Distillation (AdaFD)

• Adaptive Federated Distillation (AdaFD) is a framework that dynamically adjusts teacher weights using signals like confidence, loss, or entropy to improve knowledge transfer in federated settings.
• It replaces fixed averaging with adaptive mixture strategies, enabling robust performance under non-IID data and client heterogeneity across diverse applications such as image and textual data tasks.
• AdaFD enhances communication efficiency and model robustness by selectively aggregating client predictions, reducing data exchange while maintaining high accuracy in empirical evaluations.

Searching arXiv for the specified paper and closely related Adaptive Federated Distillation work. {"query":"arXiv (Ma et al., 2020) Adaptive Distillation for Decentralized Learning from Heterogeneous Clients Adaptive Federated Distillation related papers", "max_results": 10} Received arXiv search results including "Adaptive Distillation for Decentralized Learning from Heterogeneous Clients" (Ma et al., 2020) and related federated distillation papers such as (Wu et al., 2021, Liu et al., 2022, Su et al., 2022, Sturluson et al., 2021, Tsouvalas et al., 2024, Li et al., 2024), and (Xiao et al., 28 Aug 2025). Adaptive Federated Distillation (AdaFD) denotes a class of federated or server-coordinated distillation methods in which the transfer of client knowledge is not aggregated uniformly, but is modulated by an adaptive signal such as per-sample confidence, training loss, entropy, domain similarity, robustness statistics, or communication state. In the literature, the name is used explicitly for a multi-domain textual-data framework (Xiao et al., 28 Aug 2025), while earlier works such as Decentralized Learning via Adaptive Distillation (DLAD) instantiate the same operational pattern through adaptive weighting of heterogeneous client predictions on unlabeled distillation data (Ma et al., 2020). Within the broader federated distillation taxonomy, AdaFD occupies the part of the design space that replaces fixed averaging with sample-aware, client-aware, or system-aware weighting while retaining the central benefits of distillation: architecture heterogeneity, reduced parameter exchange, and training on public, unlabeled, or synthetic proxy data (Li et al., 2024).

1. Nomenclature and problem setting

In federated distillation, multiple clients hold private labeled datasets and a server coordinates knowledge transfer through outputs such as logits or soft labels, rather than through repeated averaging of full model parameters. The 2025 textual-data formulation states this explicitly for $K$ clients and one server: each client $k$ owns labeled private data $D^k=\{(x_i^k,y_i^k)\}$, the server and clients share access to an unlabeled public proxy dataset $D^0=\{x_i^0\}$, and the objective is to obtain a final central model without sharing raw private data or model parameters (Xiao et al., 28 Aug 2025). This operational pattern differs from conventional FL baselines such as FedAvg, which repeatedly transport full-precision model parameters and generally assume homogeneous architectures (Li et al., 2024).

The 2020 DLAD formulation studies a closely related setting with stronger heterogeneity: there are $M$ clients, each client $i$ holds a private dataset $D_i=(X_i,Y_i)$ and a local model $f_i:\mathcal{X}\to\Delta^{L_i}$, and client label sets $Y^{(i)}\subseteq Y$ may differ across clients. A shared unlabeled distillation set $U=X_{\text{dist}}$ is available, the server coordinates distillation, and clients share either model artifacts or predictions needed for distillation, while raw private data remain local. DLAD is single-shot rather than round-based, and it explicitly supports heterogeneous client architectures and heterogeneous label spaces via zero-padding of client outputs to the global label set (Ma et al., 2020).

A common source of ambiguity is terminological rather than algorithmic. The DLAD paper does not use the term “Adaptive Federated Distillation (AdaFD),” and several later methods are likewise named differently, including FedKD, FAS, FedRAD, and FedD3A. This suggests that AdaFD is best understood as an umbrella for adaptive distillation strategies in federated or decentralized learning, while particular papers instantiate that umbrella with different adaptive signals and communication protocols (Ma et al., 2020).

2. Canonical mechanism: per-sample adaptive aggregation

A canonical AdaFD mechanism is the DLAD formulation for decentralized learning from heterogeneous clients. For a distillation sample $k$0 and client $k$1, let $k$2 denote the client predictive distribution over the global label set $k$3; if $k$4 predicts only over $k$5, the vector is zero-padded over $k$6. DLAD then introduces a client-specific binary classifier $k$7 trained to distinguish client data $k$8 from the distillation data $k$9. The key quantity is the confidence score

$D^k=\{(x_i^k,y_i^k)\}$0

The rationale for this confidence is density-ratio based. If $D^k=\{(x_i^k,y_i^k)\}$1 is trained optimally, then

$D^k=\{(x_i^k,y_i^k)\}$2

where $D^k=\{(x_i^k,y_i^k)\}$3 and $D^k=\{(x_i^k,y_i^k)\}$4 are the density of $D^k=\{(x_i^k,y_i^k)\}$5 under $D^k=\{(x_i^k,y_i^k)\}$6 and $D^k=\{(x_i^k,y_i^k)\}$7, respectively. Because $D^k=\{(x_i^k,y_i^k)\}$8 increases monotonically with $D^k=\{(x_i^k,y_i^k)\}$9 for fixed $D^0=\{x_i^0\}$0, larger $D^0=\{x_i^0\}$1 indicates that $D^0=\{x_i^0\}$2 resembles client $D^0=\{x_i^0\}$3’s data more closely; accordingly, the associated client model is treated as more reliable on that sample (Ma et al., 2020).

DLAD converts these confidences into per-sample teacher weights through a temperature-scaled softmax:

$D^0=\{x_i^0\}$4

with $D^0=\{x_i^0\}$5 controlling smoothness and $D^0=\{x_i^0\}$6 used in the paper. The global teacher distribution is then

$D^0=\{x_i^0\}$7

and the global student model $D^0=\{x_i^0\}$8 is trained on the unlabeled set $D^0=\{x_i^0\}$9 by minimizing categorical cross-entropy

$M$0

No temperature is applied to class probabilities in the distillation loss; the temperature enters only through the weight softmax. The training protocol is server-coordinated and single-shot: local client training, confidence-classifier training, and then server-side distillation of the global student (Ma et al., 2020).

This formulation exposes the central AdaFD pattern in a particularly transparent way. Adaptation occurs at the granularity of client-sample pairs rather than only at the round level or the client level, and the distillation target is an adaptive mixture of teachers rather than an equal-weight average. In the broad sense of AdaFD, this is the defining mechanism: the teacher ensemble is conditioned on evidence of reliability.

3. Major adaptive variants

The literature implements AdaFD through several distinct adaptive signals. Some methods adapt how much each teacher contributes to each sample; others adapt the strength of mutual distillation, the subset of public samples to communicate, the compression level, or the teacher set itself.

Method	Adaptive signal	Distillation substrate
DLAD (Ma et al., 2020)	Discriminator confidence $M$1	Confidence-weighted client predictions on $M$2
AdaFD for textual data (Xiao et al., 28 Aug 2025)	Training-loss-based RNWC/ENWC or GPT-4-based LLMWC	Public-data logits
FedKD (Wu et al., 2021)	Inverse supervised loss	Mutual teacher–student and hidden-state distillation
FAS (Liu et al., 2022)	Entropy-based active public-sample selection	Selected public logits
FedRAD (Sturluson et al., 2021)	Median-based trust counts	Median-of-logits KD
FedD3A (Su et al., 2022)	Subspace-projection similarity	Sample-level teacher weighting

The explicit AdaFD framework for multi-domain textual data aggregates client logits as

$M$3

where the weights are produced by one of three schemes. RNWC uses inverse normalization of the minimum training loss, ENWC uses an exponential decay governed by a hyperparameter $M$4, and LLMWC uses GPT-4 prompted with training loss, F1, logits, and logits entropy. The server then distills by minimizing

$M$5

motivated by the observation that KL with temperature $M$6 can be approximated by $M$7 as $M$8. The framework supports both homogeneous and heterogeneous PLM settings because only logits are exchanged (Xiao et al., 28 Aug 2025).

FedKD realizes adaptation differently. Each client maintains a private teacher and a shared student, and mutual distillation intensity is controlled by prediction quality. The adaptive KD terms divide the KL and hidden-state alignment losses by the sum of the supervised teacher and student cross-entropies, so when predictions are poor the distillation signal is weak, and when predictions are good the distillation signal intensifies. The adaptive weight is therefore local, per-client, and per-sample or per-batch, rather than server-side (Wu et al., 2021).

FAS adapts the communication set instead of directly adapting teacher weights. Each client selects half of its public-sample budget from the lowest-entropy public examples, with class-proportion matching to the client’s private label distribution, and the other half uniformly at random. The server then averages logits by public-sample index and may apply Entropy Reduction Aggregation (ERA) with a sharpening temperature $M$9. In this design, the adaptive component is “what to communicate” rather than only “how to average” (Liu et al., 2022).

FedRAD and FedD3A represent two other important directions. FedRAD computes round-wise trust scores from median counts over client logits on an unlabeled server dataset and then performs ensemble knowledge distillation with an element-wise median-of-logits teacher, thereby targeting robustness to faulty and malicious clients (Sturluson et al., 2021). FedD3A computes a sample-level similarity between a server sample and each client domain using a client-uploaded subspace projection matrix, converts the similarity vector into z-score-softmax weights, and uses those weights to construct a domain-discrepancy-aware teacher ensemble for each server sample (Su et al., 2022).

4. Communication, heterogeneity, and systems design

A central reason AdaFD has attracted attention is that it separates collaborative learning from strict parameter homogeneity. In DLAD, clients may send either model parameters $i$0 and confidence classifiers $i$1 once, allowing the server to run inference on $i$2, or they may locally evaluate $i$3 and send per-sample logits $i$4 and confidences $i$5. Because the protocol is single-shot rather than iterative, no client-side participation is needed after the single upload. If clients transmit per-sample outputs on $i$6, the communication volume is $i$7 for logits and $i$8 for confidence scalars; if they transmit parameters, that per-sample overhead is avoided (Ma et al., 2020).

Several later systems make the communication objective explicit. FedCompress combines adaptive weight clustering and server-side knowledge distillation as a drop-in layer on top of FedAvg. The number of clusters $i$9 adapts over rounds according to a representation quality score computed locally from the effective rank of penultimate-layer embeddings, and the server then distills the aggregated model into a clustered student using out-of-distribution data. Reported Communication Compression Reduction values are 4.53× on CIFAR-10, 3.80× on CIFAR-100, 4.79× on PathMNIST, 5.04× on SpeechCommands, and 5.41× on VoxForge, with an average of approximately 4.5×; Model Compression Ratio ranges from 3.93× to 5.64×, with average approximately 4.14×, and inference speedups on edge accelerators are 1.10–1.15× in float32 and 1.16–1.25× in uint8 (Tsouvalas et al., 2024).

FedKD reduces communication by federating only the student model and compressing exchanged gradients with dynamic SVD. On MIND, the large-model federated baseline reports AUC $D_i=(X_i,Y_i)$0 with 2.05GB per-client communication, whereas FedKD_4 reports AUC $D_i=(X_i,Y_i)$1 with 0.19GB; on SMM4H, the federated UniLM baseline reports F1 $D_i=(X_i,Y_i)$2 with 1.37GB, while FedKD_4 reports F1 $D_i=(X_i,Y_i)$3 with 0.12GB. The method therefore couples adaptive distillation and adaptive compression (Wu et al., 2021).

Communication-aware AdaFD has also been extended to large-vocabulary LLM settings. One formulation uses adaptive Top-$D_i=(X_i,Y_i)$4 logit selection determined by real-time communication budgets, adaptive aggregation across heterogeneous sparse index sets without zero-padding, and LoRA-adapted hidden-layer projection distillation. In the reported Banking77 experiments, AdaFD reaches target accuracies with 16.6 MB, 28.0 MB, and 49.1 MB of total communication for accuracies 0.70, 0.75, and 0.79, whereas the All-logits baseline requires 763.1 MB to 2049.6 MB; the summary characterizes the reduction as approximately 50% relative to strong baselines (Zhang et al., 1 Sep 2025).

These system variants show that “adaptivity” in AdaFD is not confined to the aggregation rule. It may govern compression granularity, the number of transmitted logits, the subset of proxy samples, or whether distillation is one-shot or multi-round. A plausible implication is that AdaFD is better viewed as a joint optimization problem over teacher reliability, communication budget, and architectural heterogeneity than as a single distillation formula.

5. Empirical behavior across heterogeneous and non-IID settings

The most direct empirical evidence for the value of adaptive aggregation appears in DLAD. On MNIST with $D_i=(X_i,Y_i)$5 ResNet clients, client accuracies degrade sharply under non-IID partitions, with an example accuracy of 0.199 for NIID1, and the equal-averaging global model ranges widely, including 0.3954 for NIID1 and 0.5164 for NIID2. DLAD, by contrast, achieves approximately 0.982–0.984 across NIID1–3, essentially closing the non-IID gap and approaching the upper bound with labeled distillation. On CIFAR-10 with $D_i=(X_i,Y_i)$6 and heterogeneous clients split 50/50 between ResNet and DenseNet, the NIID1 global model under equal averaging reaches 0.3648, whereas DLAD reaches 0.6657; on CINIC-10 with $D_i=(X_i,Y_i)$7 ResNet clients, NIID1 reports 0.3534 for equal averaging and 0.6266 for DLAD (Ma et al., 2020).

The explicit textual-data AdaFD framework reports that AdaFD variants consistently outperform FD baselines and show robustness in both homogeneous and heterogeneous PLM regimes. Experiments use five Amazon review domains—automotive, baby, clothing, health, and sport—with 80% private and 20% public unlabeled data per domain, five communication rounds, and three local epochs per round. Tables report statistically significant improvements with $D_i=(X_i,Y_i)$8, and AdaFD with ENWC is identified as the final preferred model because it combines strong accuracy with no LLM overhead (Xiao et al., 28 Aug 2025).

FedD3A shows that sample-level domain-discrepancy-aware weighting can matter even when all methods operate on the same public server dataset. On DomainNet, the summary states that FedD3A exceeds FedDF by nearly 10% absolute accuracy in cross-silo experiments. In the teacher-weighting ablation with clipart as the server domain, the reported averages are 0.369 for uniform averaging, 0.363 for random weights, 0.420 for FedD3A-onehot, 0.439 for FedD3A-soft, and 0.445 for a ceiling oracle, indicating that soft discrepancy-aware weighting is substantially more effective than equal weighting (Su et al., 2022).

FedRAD demonstrates that adaptive distillation can also serve robustness rather than only heterogeneity handling. On MNIST with 10 faulty attackers, FedRAD reports error rates of $D_i=(X_i,Y_i)$9 under IID, $f_i:\mathcal{X}\to\Delta^{L_i}$0 under non-IID $f_i:\mathcal{X}\to\Delta^{L_i}$1, and $f_i:\mathcal{X}\to\Delta^{L_i}$2 under non-IID $f_i:\mathcal{X}\to\Delta^{L_i}$3; with 10 malicious attackers, the corresponding values are $f_i:\mathcal{X}\to\Delta^{L_i}$4, $f_i:\mathcal{X}\to\Delta^{L_i}$5, and $f_i:\mathcal{X}\to\Delta^{L_i}$6. The same summary notes that many baselines fail under these attack settings, and that median-count histograms show faulty clients are selected as median almost never (Sturluson et al., 2021).

Taken together, these results indicate a recurring empirical pattern: equal-weight logit averaging is often competitive when clients are similar, but adaptive aggregation becomes much more consequential when heterogeneity is induced by label skew, domain shift, client quality variation, or adversarial behavior. This suggests that the principal utility of AdaFD lies not in replacing standard distillation per se, but in stabilizing teacher quality under conditions where uniform averaging is structurally misaligned with the data-generating process.

6. Limitations, misconceptions, and open directions

A recurrent limitation is dependence on the quality of the proxy or distillation dataset. DLAD requires a large unlabeled set $f_i:\mathcal{X}\to\Delta^{L_i}$7 drawn from a domain relevant to clients, and the summary states that strong mismatch reduces the quality of $f_i:\mathcal{X}\to\Delta^{L_i}$8 and harms weights (Ma et al., 2020). The textual AdaFD framework likewise assumes an unlabeled public dataset $f_i:\mathcal{X}\to\Delta^{L_i}$9 and notes that mismatched proxies may reduce effectiveness (Xiao et al., 28 Aug 2025). FedD3A addresses server-to-client and client-to-client discrepancy explicitly, but it still relies on a public server dataset and on the quality of backbone features used for subspace-projection similarity (Su et al., 2022).

Another limitation is that privacy preservation is usually partial rather than formal. DLAD keeps raw data local but does not implement stronger privacy such as differential privacy (Ma et al., 2020). The multi-domain textual AdaFD work similarly states that no differential privacy mechanism is added, and it notes that logits on public inputs may leak limited model information (Xiao et al., 28 Aug 2025). The survey literature further records inversion and membership inference risks against FD-style logit exchange, together with defenses based on differential privacy, noise addition, one-shot communication, compression, and robust aggregation (Li et al., 2024).

Adversarial robustness is not automatic. DLAD does not include explicit defenses against adversarial clients, and a client could train $Y^{(i)}\subseteq Y$0 to yield inflated confidences; the summary notes that practical implementations may require trust assumptions, auditing, or secure aggregation (Ma et al., 2020). FedRAD is a counterexample in which robustness is a first-class objective, but its median-based design still presumes that fewer than half of the ensemble logits per class are adversarially corrupted (Sturluson et al., 2021).

A common misconception is that AdaFD refers to a single standardized algorithm. The literature does not support that reading. The 2020 paper introduces DLAD rather than AdaFD, the 2025 textual-data paper explicitly names its framework AdaFD, and the survey organizes the field around adaptive weighting, clustering, uncertainty-aware aggregation, class-wise distillation, and compression rather than around one canonical protocol (Ma et al., 2020). This suggests that AdaFD is more accurately treated as a methodological family characterized by adaptive knowledge transfer under federated constraints.

Open directions identified across the papers include integrating secure aggregation and privacy-preserving mechanisms, extending these methods to other modalities such as NLP and medical imaging, improving confidence calibration, handling stronger practical constraints such as network conditions and client availability, and providing tighter theoretical analyses of convergence and discrepancy-aware aggregation (Ma et al., 2020). In that sense, AdaFD remains an active design paradigm rather than a closed technique: its central idea is stable, but its concrete implementation continues to vary with the source of heterogeneity that must be controlled.