---
title: 'FedeKD: Reliable Federated Knowledge Distillation'
url: https://www.emergentmind.com/topics/fedekd
type: topic
---

# FedeKD: Reliable Federated Knowledge Distillation

FedeKD is a federated knowledge distillation framework for heterogeneous federated learning in which each client keeps a high-capacity private model for local learning and participates in collaboration through a lightweight shared proxy model. Its defining feature is an energy-based gating mechanism that turns private–proxy disagreement into sample-wise trust weights during backward distillation, so that transferred knowledge is emphasized on reliable samples and down-weighted on unreliable ones. The framework is explicitly designed to reduce negative transfer under heterogeneous data distributions and asymmetric model design, and it does so without relying on additional public data [2605.05553].

## 1. Definition and terminological position

The title **“FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings”** identifies a specific reliability-aware federated knowledge distillation method [2605.05553]. It belongs to the broader family of federated distillation methods in which collaboration proceeds through outputs or logits rather than full model parameters. In the foundational formulation of **Federated Knowledge Distillation** or **FD**, workers exchange model outputs whose dimensions are often much smaller than model size, making distillation-based collaboration attractive under limited communication resources [2011.02367].

Within the wider literature, closely related naming includes **FedGKD**, which regularizes local training with global-teacher distillation to alleviate client drift [2107.00051]; **FedKD**, which trains a local teacher and a shared student while communicating only the student and compressing exchanged gradients with dynamic SVD [2108.13323]; **FedDKD**, which performs decentralized knowledge distillation to approach the neural network map average in function space [2205.00706]; and **FedHKD**, which transmits class-wise means of local representations and corresponding soft predictions as “hyper-knowledge” [2301.08968]. This naming pattern suggests that “FedeKD” should be read not as a generic synonym for all federated distillation methods, but as a particular method within a broader FKD design space.

The conceptual distinction between FedeKD and parameter-averaging federated learning is structural. Standard FL typically exchanges gradients or weights, whereas federated distillation exchanges predictive behavior. FedeKD preserves that distillation-centered premise, but adds a reliability model: knowledge transfer is not assumed to be uniformly trustworthy across samples, clients, or rounds [2605.05553].

## 2. Problem setting: heterogeneity and negative transfer

FedeKD targets federated learning environments characterized by **data heterogeneity**, **model heterogeneity / asymmetry**, and uneven reliability of transferred knowledge. The motivating claim is that, under heterogeneous label or feature distributions and asymmetric client models, collaboration can become harmful rather than beneficial. The paper formalizes this failure mode as **negative transfer**, defined for client \(k\) as
\[
\Delta_k = \text{performance after collaboration} - \text{performance under local training},
\]
with harmful collaboration corresponding to \(\Delta_k < 0\) for accuracy-based metrics, or larger RMSE for regression [2605.05553].

This framing places FedeKD in direct contrast with methods that assume transferred knowledge is uniformly beneficial. The paper explicitly notes that parameter-averaging methods such as **FedAvg** and **FedProx** struggle under heterogeneity, and that many FKD methods either depend on public data or implicitly trust the transferred teacher signal on all samples [2605.05553]. FedeKD instead treats reliability as a sample-level quantity.

The method is formulated for both classification and regression. For classification, evaluation includes **Accuracy**, \(\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}\), **Avg \(\Delta\)**, **Worst \(\Delta\)**, **P10 \(\Delta\)**, mean accuracy, and worst-case accuracy. For regression, evaluation includes **RMSE**, \(\Delta_k = \text{RMSE}_k^{\text{method}} - \text{RMSE}_k^{\text{local}}\), **Avg \(\Delta\)**, **Worst \(\Delta\)**, **P90 \(\Delta\)**, mean RMSE, and worst-case RMSE [2605.05553]. This metric design makes the framework particularly concerned with tail behavior and the protection of vulnerable clients, not only average predictive quality.

## 3. Asymmetric architecture and round structure

Each client \(k\) in FedeKD maintains two models. The **private model** \(f_k\) is the higher-capacity model used for the main task and never leaves the client. The **proxy model** \(g_k\) is lightweight and is the only object communicated to the server. This asymmetric arrangement is the basic systems design of the framework: the private model supports strong local learning, while the proxy model enables cross-client knowledge exchange at lower communication and compatibility cost [2605.05553].

Training proceeds in a three-stage communication round. In **Stage 1**, each client freezes \(f_k\) and trains \(g_k\) to mimic \(f_k\) on local data; this is the forward proxy distillation step. In **Stage 2**, the server aggregates proxies by simple averaging,
\[
g \leftarrow \frac{1}{K}\sum_{k=1}^{K} g_k,
\]
to obtain a global proxy \(g\). In **Stage 3**, the global proxy is broadcast back to clients, and each client uses it to update its private model through energy-gated backward distillation [2605.05553].

The resulting information flow is
\[
f_k \rightarrow g_k \rightarrow g \rightarrow f_k.
\]
This captures the method’s defining asymmetry: the private model teaches the local proxy; the global proxy, after aggregation, teaches the private model. The private model objective is
\[
\mathcal{L}_{\text{private}} = \mathcal{L}_{\text{sup}} + \lambda_{\text{kd}}\mathcal{L}_{\text{BKD}},
\]
where \(\mathcal{L}_{\text{sup}}\) is the supervised local loss and \(\mathcal{L}_{\text{BKD}}\) is the energy-gated backward distillation loss [2605.05553]. The framework therefore remains supervised on local labels while adding a selectively trusted collaborative signal.

## 4. Energy-based gating and sample-wise trust estimation

The central technical contribution of FedeKD is the conversion of private–proxy disagreement into a **sample-wise trust weight**. For classification, the paper defines energy by an entropy-normalized symmetric KL divergence:
\[
E(x) = \frac{\frac{1}{2}\big( \mathrm{KL}(p \,\|\, q) + \mathrm{KL}(q \,\|\, p) \big)}{H(p)+H(q)+\epsilon_H},
\]
where \(p=\sigma(f_k(x))\), \(q=\sigma(g(x))\), and \(\epsilon_H = 10^{-8}\) [2605.05553]. The numerator measures bidirectional disagreement; the denominator normalizes that disagreement by predictive uncertainty. The intended effect is that confident contradiction is treated as more consequential than disagreement between uncertain predictors.

For regression, energy is defined as squared prediction disagreement:
\[
E(x)=\frac{1}{2}\|f_k(x)-g(x)\|_2^2.
\]
The paper explicitly notes that this regression form does not model uncertainty explicitly; it is a direct functional disagreement measure [2605.05553].

Trust is computed relative to the current minibatch. For minibatch \(B=\{x_i\}_{i=1}^{|B|}\), with energies \(E_i\), mean \(\mu_B\), and standard deviation \(s_B\), normalized energy is
\[
\widetilde{E}_i = \frac{E_i-\mu_B}{s_B+\epsilon_B},
\]
and the trust weight is
\[
w_i = \rho(-\beta \widetilde{E}_i), \qquad \rho(t)=\frac{1}{1+\exp(-t)},
\]
with \(\beta>0\) controlling gate sharpness and \(\epsilon_B = 10^{-8}\) preventing division by zero [2605.05553]. If a sample’s energy is below the batch mean, then \(w_i > 1/2\); if its energy is above the batch mean, then \(w_i < 1/2\). Reliability is therefore assessed through a soft, batch-relative ranking rather than a fixed threshold.

Backward distillation uses these trust weights through
\[
\mathcal{L}_{\text{BKD}} = \mathbb{E}_{B\sim \mathcal{D}_k} \left[ \frac{1}{|B|} \sum_{i=1}^{|B|} \operatorname{sg}(w_i)\,\ell_{\text{KD}}(x_i) \right],
\]
where \(\operatorname{sg}(\cdot)\) denotes stop-gradient [2605.05553]. For classification, \(\ell_{\text{KD}}(x_i)=\mathrm{KL}(q_i\|p_i)\); for regression, \(\ell_{\text{KD}}(x_i)=\|f_k(x_i)-g(x_i)\|_2^2\). Because the gate is detached, it rescales but does not redirect the distillation gradient. The paper also gives a variational interpretation of the logistic gate:
\[
w^\star=\rho(-\beta \widetilde{E}) = \arg\min_{w\in(0,1)} \left[ w\widetilde{E} + \frac{1}{\beta}\big(w\log w + (1-w)\log(1-w)\big) \right].
\]
This presents gating as a soft trust assignment balancing low energy preference with entropy regularization [2605.05553].

## 5. Empirical evaluation and reported findings

FedeKD is evaluated on six real-world datasets. The classification benchmarks are **FashionMNIST**, **CIFAR-10**, **OCTMNIST**, and **OrganAMNIST**. The regression benchmarks are **RetinaMNIST** and **Diabetic Retinopathy** [2605.05553]. Heterogeneity is simulated with **6 clients** and Dirichlet partitioning with \(\alpha \in \{0.1, 0.3, 0.5\}\), where smaller \(\alpha\) means stronger heterogeneity. For classification, the heterogeneity is label-skew; for regression, covariate shift is induced via K-means clustering into 5 bins and then Dirichlet splitting [2605.05553].

The architectural pattern is asymmetric throughout. For classification, the paper uses a private CNN with 3 convolutional layers followed by a fully connected layer, and a smaller 2-layer proxy CNN. Optimization uses **Adam**, learning rate \(10^{-4}\), **5 communication rounds**, **2 local epochs per round**, minibatch size **64**, evaluation batch size **256**, and FedeKD hyperparameters \(\beta = 1\) and \(\lambda_{\text{kd}}=1\) [2605.05553]. Baselines include **FedAvg**, **FedProx**, **FedDyn**, and **FedType**, together with ablations such as **No Gating** and alternative energy functions based on entropy, margin, LogSumExp, and feature distance [2605.05553].

The main empirical claim is that FedeKD is best or near-best across all four classification datasets and heterogeneity levels, especially at \(\alpha=0.1\), and best on **Avg \(\Delta\)**, **Worst \(\Delta\)**, and **P90 \(\Delta\)** across both regression datasets and all \(\alpha\) values [2605.05553]. The paper emphasizes that standard FL baselines remain harmful in regression, while FedeKD often drives degradation close to zero or below. It also reports strong standard predictive performance, with best or top-tier classification accuracy and lowest RMSE on regression tasks.

Ablation results support the gating hypothesis. Without gating, backward distillation can be harmful, particularly under strong heterogeneity. Alternative energy functions are usable, but the proposed KL-based disagreement energy is reported as the most consistent and stable overall [2605.05553]. Sensitivity analyses indicate that \(\beta\) should not be too small or too large: very small values make weights nearly uniform, while very large values make gating overly selective. The paper states that \(\beta\in[1,2]\) is a reliable range in the reported setting, and that \(\lambda_{\text{kd}}\) has a unimodal effect, with moderate values often working well in milder heterogeneity even though the default \(\lambda_{\text{kd}}=1\) is stable overall [2605.05553].

## 6. Relation to broader federated distillation research, limitations, and significance

FedeKD sits within a broader transition from parameter-space aggregation to function-space or output-space collaboration. The original FD formulation exchanges label-wise averaged logits and reports about **10,000× reduction in payload size per communication round** and over **40,000× reduction in total communication cost until convergence** in MNIST experiments, albeit with possible accuracy loss under non-IID data [2011.02367]. FedGKD addresses heterogeneity by distilling from the latest or historical global models to suppress client drift, while keeping a FedAvg-like communication protocol and avoiding proxy/public data, generators, or architectural changes [2107.00051]. FedDKD goes further in arguing that heterogeneous FL should approximate a neural network map average in function space rather than merely average parameters [2205.00706].

Other branches of the literature emphasize different axes. Communication-efficient **FedKD** shares only a small student and compresses exchanged student gradients by dynamic SVD [2108.13323]. **FedHKD** replaces public-data distillation with class-wise means of latent representations and soft predictions, aggregated as “hyper-knowledge,” and adds Gaussian noise for differential privacy [2301.08968]. **HFedCKD** combines data-free knowledge distillation with Inverse Probability Weighted Distillation and two-way contrast learning for low-participation, heterogeneous settings [2503.06511]. **FedKDNAS** couples federated distillation with client-side neural architecture selection and public-reference-set logit exchange [2605.21322]. In federated knowledge graph embedding, a different **FedKD** compresses a 256-d teacher into a 128-d student through client-side local distillation, with Adaptive Asymmetric Temperature Scaling and dynamic KD weighting [2408.05748]. Taken together, these methods suggest a broad design space in which the key choices are what is communicated, where teachers reside, whether public data are used, and how heterogeneity is controlled.

Against that backdrop, FedeKD’s distinctive contribution is its explicit **sample-wise reliability modeling**. Rather than assuming that the collaborative teacher signal should always be followed, it uses disagreement between the private and global proxy models as a trust proxy and gates backward distillation accordingly [2605.05553]. A plausible implication is that FedeKD is most useful when heterogeneity creates mixed-quality teacher signals rather than uniformly bad or uniformly good ones.

The paper also states several limitations. Experiments are conducted on public benchmarks with simulated cross-silo partitions; fairness across subpopulations is not directly evaluated; maintaining both private and proxy models introduces extra computation and synchronization cost; and the regression energy uses prediction disagreement rather than calibrated uncertainty [2605.05553]. These constraints define the current scope of the method. Even so, within the federated distillation literature, FedeKD represents a clear shift from transfer by uniform consensus toward transfer by calibrated trust, making reliability estimation a first-class component of heterogeneous federated knowledge distillation.

Source: https://www.emergentmind.com/topics/fedekd