FedeKD: Reliable Federated Knowledge Distillation

• FedeKD is a federated knowledge distillation method that uses an energy-based gating mechanism to compute sample-wise trust for reliable knowledge transfer.
• It employs asymmetric architecture with high-capacity private models and lightweight proxy models to mitigate negative transfer under heterogeneous data distributions.
• Empirical results on diverse classification and regression benchmarks demonstrate robust performance improvements over traditional federated learning methods.

FedeKD is a federated knowledge distillation framework for heterogeneous federated learning in which each client keeps a high-capacity private model for local learning and participates in collaboration through a lightweight shared proxy model. Its defining feature is an energy-based gating mechanism that turns private–proxy disagreement into sample-wise trust weights during backward distillation, so that transferred knowledge is emphasized on reliable samples and down-weighted on unreliable ones. The framework is explicitly designed to reduce negative transfer under heterogeneous data distributions and asymmetric model design, and it does so without relying on additional public data (Nguyen et al., 7 May 2026).

1. Definition and terminological position

The title “FedeKD: Energy-Based Gating for Robust Federated Knowledge Distillation under Heterogeneous Settings” identifies a specific reliability-aware federated knowledge distillation method (Nguyen et al., 7 May 2026). It belongs to the broader family of federated distillation methods in which collaboration proceeds through outputs or logits rather than full model parameters. In the foundational formulation of Federated Knowledge Distillation or FD, workers exchange model outputs whose dimensions are often much smaller than model size, making distillation-based collaboration attractive under limited communication resources (Seo et al., 2020).

Within the wider literature, closely related naming includes FedGKD, which regularizes local training with global-teacher distillation to alleviate client drift (Yao et al., 2021); FedKD, which trains a local teacher and a shared student while communicating only the student and compressing exchanged gradients with dynamic SVD (Wu et al., 2021); FedDKD, which performs decentralized knowledge distillation to approach the neural network map average in function space (Li et al., 2022); and FedHKD, which transmits class-wise means of local representations and corresponding soft predictions as “hyper-knowledge” (Chen et al., 2023). This naming pattern suggests that “FedeKD” should be read not as a generic synonym for all federated distillation methods, but as a particular method within a broader FKD design space.

The conceptual distinction between FedeKD and parameter-averaging federated learning is structural. Standard FL typically exchanges gradients or weights, whereas federated distillation exchanges predictive behavior. FedeKD preserves that distillation-centered premise, but adds a reliability model: knowledge transfer is not assumed to be uniformly trustworthy across samples, clients, or rounds (Nguyen et al., 7 May 2026).

2. Problem setting: heterogeneity and negative transfer

FedeKD targets federated learning environments characterized by data heterogeneity, model heterogeneity / asymmetry, and uneven reliability of transferred knowledge. The motivating claim is that, under heterogeneous label or feature distributions and asymmetric client models, collaboration can become harmful rather than beneficial. The paper formalizes this failure mode as negative transfer, defined for client $k$ as

$$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$

with harmful collaboration corresponding to $\Delta_k < 0$ for accuracy-based metrics, or larger RMSE for regression (Nguyen et al., 7 May 2026).

This framing places FedeKD in direct contrast with methods that assume transferred knowledge is uniformly beneficial. The paper explicitly notes that parameter-averaging methods such as FedAvg and FedProx struggle under heterogeneity, and that many FKD methods either depend on public data or implicitly trust the transferred teacher signal on all samples (Nguyen et al., 7 May 2026). FedeKD instead treats reliability as a sample-level quantity.

The method is formulated for both classification and regression. For classification, evaluation includes Accuracy, $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$, Avg $\Delta$, Worst $\Delta$, P10 $\Delta$, mean accuracy, and worst-case accuracy. For regression, evaluation includes RMSE, $$\Delta_k = \text{RMSE}_k^{\text{method}} - \text{RMSE}_k^{\text{local}}$$, Avg $\Delta$, Worst $\Delta$, P90 $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$0, mean RMSE, and worst-case RMSE (Nguyen et al., 7 May 2026). This metric design makes the framework particularly concerned with tail behavior and the protection of vulnerable clients, not only average predictive quality.

3. Asymmetric architecture and round structure

Each client $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$1 in FedeKD maintains two models. The private model $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$2 is the higher-capacity model used for the main task and never leaves the client. The proxy model $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$3 is lightweight and is the only object communicated to the server. This asymmetric arrangement is the basic systems design of the framework: the private model supports strong local learning, while the proxy model enables cross-client knowledge exchange at lower communication and compatibility cost (Nguyen et al., 7 May 2026).

Training proceeds in a three-stage communication round. In Stage 1, each client freezes $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$4 and trains $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$5 to mimic $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$6 on local data; this is the forward proxy distillation step. In Stage 2, the server aggregates proxies by simple averaging,

$$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$7

to obtain a global proxy $$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$8. In Stage 3, the global proxy is broadcast back to clients, and each client uses it to update its private model through energy-gated backward distillation (Nguyen et al., 7 May 2026).

The resulting information flow is

$$\Delta_k = \text{performance after collaboration} - \text{performance under local training},$$9

This captures the method’s defining asymmetry: the private model teaches the local proxy; the global proxy, after aggregation, teaches the private model. The private model objective is

$\Delta_k < 0$0

where $\Delta_k < 0$1 is the supervised local loss and $\Delta_k < 0$2 is the energy-gated backward distillation loss (Nguyen et al., 7 May 2026). The framework therefore remains supervised on local labels while adding a selectively trusted collaborative signal.

4. Energy-based gating and sample-wise trust estimation

The central technical contribution of FedeKD is the conversion of private–proxy disagreement into a sample-wise trust weight. For classification, the paper defines energy by an entropy-normalized symmetric KL divergence: $\Delta_k < 0$3 where $\Delta_k < 0$4, $\Delta_k < 0$5, and $\Delta_k < 0$6 (Nguyen et al., 7 May 2026). The numerator measures bidirectional disagreement; the denominator normalizes that disagreement by predictive uncertainty. The intended effect is that confident contradiction is treated as more consequential than disagreement between uncertain predictors.

For regression, energy is defined as squared prediction disagreement: $\Delta_k < 0$7 The paper explicitly notes that this regression form does not model uncertainty explicitly; it is a direct functional disagreement measure (Nguyen et al., 7 May 2026).

Trust is computed relative to the current minibatch. For minibatch $\Delta_k < 0$8, with energies $\Delta_k < 0$9, mean $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$0, and standard deviation $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$1, normalized energy is

$$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$2

and the trust weight is

$$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$3

with $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$4 controlling gate sharpness and $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$5 preventing division by zero (Nguyen et al., 7 May 2026). If a sample’s energy is below the batch mean, then $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$6; if its energy is above the batch mean, then $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$7. Reliability is therefore assessed through a soft, batch-relative ranking rather than a fixed threshold.

Backward distillation uses these trust weights through

$$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$8

where $$\Delta_k = \text{Acc}_k^{\text{method}} - \text{Acc}_k^{\text{local}}$$9 denotes stop-gradient (Nguyen et al., 7 May 2026). For classification, $\Delta$0; for regression, $\Delta$1. Because the gate is detached, it rescales but does not redirect the distillation gradient. The paper also gives a variational interpretation of the logistic gate: $\Delta$2 This presents gating as a soft trust assignment balancing low energy preference with entropy regularization (Nguyen et al., 7 May 2026).

5. Empirical evaluation and reported findings

FedeKD is evaluated on six real-world datasets. The classification benchmarks are FashionMNIST, CIFAR-10, OCTMNIST, and OrganAMNIST. The regression benchmarks are RetinaMNIST and Diabetic Retinopathy (Nguyen et al., 7 May 2026). Heterogeneity is simulated with 6 clients and Dirichlet partitioning with $\Delta$3, where smaller $\Delta$4 means stronger heterogeneity. For classification, the heterogeneity is label-skew; for regression, covariate shift is induced via K-means clustering into 5 bins and then Dirichlet splitting (Nguyen et al., 7 May 2026).

The architectural pattern is asymmetric throughout. For classification, the paper uses a private CNN with 3 convolutional layers followed by a fully connected layer, and a smaller 2-layer proxy CNN. Optimization uses Adam, learning rate $\Delta$5, 5 communication rounds, 2 local epochs per round, minibatch size 64, evaluation batch size 256, and FedeKD hyperparameters $\Delta$6 and $\Delta$7 (Nguyen et al., 7 May 2026). Baselines include FedAvg, FedProx, FedDyn, and FedType, together with ablations such as No Gating and alternative energy functions based on entropy, margin, LogSumExp, and feature distance (Nguyen et al., 7 May 2026).

The main empirical claim is that FedeKD is best or near-best across all four classification datasets and heterogeneity levels, especially at $\Delta$8, and best on Avg $\Delta$9, Worst $\Delta$0, and P90 $\Delta$1 across both regression datasets and all $\Delta$2 values (Nguyen et al., 7 May 2026). The paper emphasizes that standard FL baselines remain harmful in regression, while FedeKD often drives degradation close to zero or below. It also reports strong standard predictive performance, with best or top-tier classification accuracy and lowest RMSE on regression tasks.

Ablation results support the gating hypothesis. Without gating, backward distillation can be harmful, particularly under strong heterogeneity. Alternative energy functions are usable, but the proposed KL-based disagreement energy is reported as the most consistent and stable overall (Nguyen et al., 7 May 2026). Sensitivity analyses indicate that $\Delta$3 should not be too small or too large: very small values make weights nearly uniform, while very large values make gating overly selective. The paper states that $\Delta$4 is a reliable range in the reported setting, and that $\Delta$5 has a unimodal effect, with moderate values often working well in milder heterogeneity even though the default $\Delta$6 is stable overall (Nguyen et al., 7 May 2026).

6. Relation to broader federated distillation research, limitations, and significance

FedeKD sits within a broader transition from parameter-space aggregation to function-space or output-space collaboration. The original FD formulation exchanges label-wise averaged logits and reports about 10,000× reduction in payload size per communication round and over 40,000× reduction in total communication cost until convergence in MNIST experiments, albeit with possible accuracy loss under non-IID data (Seo et al., 2020). FedGKD addresses heterogeneity by distilling from the latest or historical global models to suppress client drift, while keeping a FedAvg-like communication protocol and avoiding proxy/public data, generators, or architectural changes (Yao et al., 2021). FedDKD goes further in arguing that heterogeneous FL should approximate a neural network map average in function space rather than merely average parameters (Li et al., 2022).

Other branches of the literature emphasize different axes. Communication-efficient FedKD shares only a small student and compresses exchanged student gradients by dynamic SVD (Wu et al., 2021). FedHKD replaces public-data distillation with class-wise means of latent representations and soft predictions, aggregated as “hyper-knowledge,” and adds Gaussian noise for differential privacy (Chen et al., 2023). HFedCKD combines data-free knowledge distillation with Inverse Probability Weighted Distillation and two-way contrast learning for low-participation, heterogeneous settings (Zheng et al., 9 Mar 2025). FedKDNAS couples federated distillation with client-side neural architecture selection and public-reference-set logit exchange (Medjadji et al., 20 May 2026). In federated knowledge graph embedding, a different FedKD compresses a 256-d teacher into a 128-d student through client-side local distillation, with Adaptive Asymmetric Temperature Scaling and dynamic KD weighting (Zhang et al., 2024). Taken together, these methods suggest a broad design space in which the key choices are what is communicated, where teachers reside, whether public data are used, and how heterogeneity is controlled.

Against that backdrop, FedeKD’s distinctive contribution is its explicit sample-wise reliability modeling. Rather than assuming that the collaborative teacher signal should always be followed, it uses disagreement between the private and global proxy models as a trust proxy and gates backward distillation accordingly (Nguyen et al., 7 May 2026). A plausible implication is that FedeKD is most useful when heterogeneity creates mixed-quality teacher signals rather than uniformly bad or uniformly good ones.

The paper also states several limitations. Experiments are conducted on public benchmarks with simulated cross-silo partitions; fairness across subpopulations is not directly evaluated; maintaining both private and proxy models introduces extra computation and synchronization cost; and the regression energy uses prediction disagreement rather than calibrated uncertainty (Nguyen et al., 7 May 2026). These constraints define the current scope of the method. Even so, within the federated distillation literature, FedeKD represents a clear shift from transfer by uniform consensus toward transfer by calibrated trust, making reliability estimation a first-class component of heterogeneous federated knowledge distillation.