Server-Client Distillation in FL

• Server-client distillation is a federated learning paradigm that exchanges soft labels computed on a shared proxy dataset to achieve efficient, bandwidth-reduced, and privacy-aware model updates.
• It supports model heterogeneity by enabling diverse neural architectures to participate without exchanging full model parameters, significantly reducing communication overhead compared to traditional methods.
• Adaptive aggregation strategies, such as weighted averaging and ensemble distillation, mitigate non-IID data challenges and enhance robust global performance.

Server-client distillation is a paradigm within federated learning (FL) where the server and clients exchange compact representations of learned knowledge—typically model logits or soft labels evaluated on a shared (often unlabeled) proxy dataset—rather than raw model parameters. This approach fundamentally alters the scalability, privacy, heterogeneity, and communication profile of collaborative learning systems. Server-client distillation enables flexible cross-device collaboration without enforcing architectural homogeneity, dramatically reducing communication load and enabling privacy-preserving and robust aggregation. Multiple algorithmic frameworks instantiate this concept with diverse technical and theoretical properties.

1. Core Principles and Workflow

In standard server-client distillation, the canonical FL communication loop is replaced by a bidirectional distillation-based knowledge transfer. Each client, after updating its local model on private data, evaluates model predictions on a shared proxy dataset and uploads only the soft logits (probability vectors) to the server. The server aggregates these logits (using mean, weighted sum, or more sophisticated strategies) to form an ensemble "teacher" distribution, which is broadcast back to clients. Clients then distill their models to match this soft consensus, combining the distillation loss (typically Kullback–Leibler divergence or cross-entropy with temperature scaling) with local supervised loss for continued private training. Unlike federated averaging, which requires strict architecture matching and massive parameter exchange, server-client distillation communicates information proportional to the proxy set size and class count, not model size, enabling model heterogeneity and vastly improved bandwidth efficiency (Li et al., 2024).

The process can be summarized as:

1. Clients compute and send soft predictions $Y_k^{\mathrm{pub}}$ over proxy data $X^{\mathrm{pub}}$.
2. Server aggregates the predictions into $Y^{\mathrm{pub}}_{\mathrm{agg}}$.
3. Server sends $Y^{\mathrm{pub}}_{\mathrm{agg}}$ to clients.
4. Clients minimize a composite loss:

$$\mathcal{L}_k = \mathcal{L}^{\mathrm{CE}}_{(k)} + \lambda \, \mathrm{KL}(q_{\mathrm{srv}}(x)\|q_k(x))$$

where $q_{\mathrm{srv}}$ is the server's consensus soft label and $q_k$ is the client output (Li et al., 2024).

This bidirectional distillation is often iterated over multiple communication rounds.

2. Motivations and Theoretical Underpinnings

Server-client distillation is motivated by several limitations in traditional FL:

• Communication Efficiency: Exchanging logits (typically $O(\#\mathrm{classes} \times |X^{\mathrm{pub}}|)$ floats) is orders of magnitude less bandwidth-intensive than exchanging full-precision weight vectors ($O(10^6$–$10^8)$ floats) (Sattler et al., 2020, Li et al., 2024).
• Support for Model Heterogeneity: It enables clients to use diverse neural architectures, since only function outputs (not weights) are exchanged. This supports hardware- and data-adaptive model selection (Chadha et al., 2024, Li et al., 2024).
• Privacy: Logits computed on a public proxy set leak considerably less private information than gradients or raw model weights, reducing the attack surface for inversion or membership attacks (Li et al., 2024).
• Non-IID Mitigation: Distillation over a common reference proxy allows implicit alignment of local models to a shared, globally informed teacher, reducing client drift without explicit parameter averaging (Li et al., 2024, Kwan et al., 2023).

Several works provide theoretical analysis of generalization in the presence of domain heterogeneity, quantifying the impact of server-to-client and client-to-client domain discrepancy (S2C and C2C), and motivating advanced sample-wise teacher selection and weighting strategies, such as projection-based adaptation (Su et al., 2022).

3. Algorithmic Variants and Frameworks

Server-client distillation admits a rich taxonomy based on the directionality of knowledge flow, manner of aggregation, and use of ensemble strategies:

Class	Example Algorithms	Description
Logits Aggregation	FedMD, FedGEMS	Mean or weighted aggregation of client soft outputs
Teacher–Student Modes	FedDF	Server or client acts as student/teacher per round
Ensemble Distillation	FedDF, FedSDD	Server ensembles multiple client models/logits as teacher
Domain-Adaptive	FedD³A, FedGO	Per-sample teacher weighting based on domain similarity
Mutual Distillation	FedBiKD, FedH2L, FedCM	Bidirectional knowledge exchange (server<->client)
Serverless	Serverless FedDF/FedMD	FaaS deployment with heterogeneous architectures

For instance, FedSDD introduces parallel aggregation of disjoint client groups into K global models, distilling only the "main" global model each round, which decouples server-side cost from client count and increases ensemble diversity through temporal pooling (Kwan et al., 2023). FedEMA-Distill leverages an exponential moving average of global models and robust aggregation of client logits via coordinate-wise median or trimmed mean, addressing Byzantine fault tolerance while maintaining model heterogeneity (Reguieg et al., 15 Feb 2026). FedGO provides theoretically grounded per-sample convex weights (inspired by GAN theory and client discriminators) to optimally ensemble client predictions for distillation (Jang et al., 10 Feb 2025).

4. Compression, Filtering, and Privacy Enhancements

A major trend in server-client distillation is the deployment of communication compression schemes:

• Quantized and Delta-Coded Logits: Compressed Federated Distillation (CFD) applies bit-quantization (down to 1-bit) and predictive/delta coding to logit vectors, together with entropy encoding, resulting in $X^{\mathrm{pub}}$0–$X^{\mathrm{pub}}$1 communication savings over FedAvg, with negligible accuracy loss (Sattler et al., 2020).
• Client-Side Filtering: EdgeFD incorporates a KMeans-based density-ratio estimator to filter out-of-distribution proxy samples locally, replacing computationally prohibitive estimators with a lightweight clustering-distance filter that is tractable on edge devices and reduces server latency (Mujtaba et al., 20 Aug 2025).
• Privacy-Preserving Distillation: TrajSyn enables server-side adversarial training by distilling a synthetic dataset from the trajectory of client model updates, avoiding exposure of raw data or explicit property inversion (Gupta et al., 17 Dec 2025). DiReDi encrypts user-informed updates in the form of weight differences, transmitting only the knowledge gap necessary to update cloud models while preserving both privacy and regulatory oversight (Sun et al., 2024).
• Dual-Distillation: To compensate for potential asynchrony or partial participation, some protocols perform distillation on both server and client sides, ensuring robustness under varied system constraints (Sattler et al., 2020).

5. Handling Heterogeneous and Non-IID Data

Non-IID data partitions and model heterogeneity are central challenges that server-client distillation explicitly addresses. Domain-adaptive aggregation, as in FedD³A, introduces per-sample adaptive teacher weighting based on domain similarity metrics computed via privacy-preserving local subspace projections; this approach demonstrably reduces both server-to-client and client-to-client risk terms, substantially improving cross-domain robustness (Su et al., 2022). Ensemble distillation using diverse global models (FedSDD) and temporal ensembling yields further decoherence, decorrelating local model updates and alleviating collapse under severe statistical heterogeneity (Kwan et al., 2023, Reguieg et al., 15 Feb 2026).

Empirical studies on image classification and NLP tasks confirm that, compared to FedAvg and earlier distillation methods, modern server-client distillation variants:

• Achieve near-centralized accuracy in highly non-IID settings.
• Cut communication cost by 10–1,000× (sometimes > 1,000×).
• Support system-level architectural diversity.
• Are robust to Byzantine and adversarial clients, with aggregate strategies (e.g., median, trimmed mean) sustaining accuracy even at 20–30% corrupted clients (Reguieg et al., 15 Feb 2026, Sattler et al., 2020).

6. Applications, Limitations, and Open Challenges

Server-client distillation has been successfully deployed in:

• Industrial IoT (edge AI for modulation classification, wind turbine monitoring) (Li et al., 2024).
• Mobile on-device speech recognition and keyword spotting (Li et al., 2024).
• Medical imaging, human activity recognition, and safety-critical edge scenarios (Sun et al., 2024, Gupta et al., 17 Dec 2025, Li et al., 2024).
• Large-scale serverless FL with Function-as-a-Service orchestration and arbitrary client architectures (Chadha et al., 2024).

Nevertheless, the paradigm faces several challenges:

• Domain Discrepancy: Public/synthetic proxy sets may not match all client domains, limiting transfer effectiveness. Recent advances address this via adaptive discrepancy-aware aggregation (Su et al., 2022) and discriminative weighting (Jang et al., 10 Feb 2025).
• Catastrophic Forgetting: Continual tasks or class-incremental learning still risk loss of historical performance unless maintained with historical teacher replays or synthetic data generators (Li et al., 2024).
• Privacy Amplification: While logits are less sensitive than weights/gradients, they are not immune to inversion; additional protections (DP noise, quantization) and careful system design remain essential (Sattler et al., 2020, Li et al., 2024).
• Scalability of Proxy Dataset: The size and representativeness of the proxy set constrain both transfer quality and communication cost; dynamic data selection is actively researched (Sattler et al., 2020, Mujtaba et al., 20 Aug 2025).

7. Experimental Results and Comparative Benchmarks

Multiple studies evaluate server-client distillation under diverse data and system conditions. Key findings include:

• FedEMA-Distill improves top-1 accuracy and reduces communication by >60× compared to FedAvg, with robust operation under strong non-IID drift and up to 30% Byzantine clients (Reguieg et al., 15 Feb 2026).
• CFD achieves up to $X^{\mathrm{pub}}$2 reduction in cumulative communication compared to FedAvg with no accuracy loss on CIFAR-10 and NLP tasks (Sattler et al., 2020).
• FedSDD, with scalable group-wise aggregation and temporal ensemble, outperforms FedAvg and FedDF under both accuracy and system scalability constraints (Kwan et al., 2023).
• FedD³A achieves up to 10 pp gains over baseline ensemble distillation on cross-silo/domain benchmarks via domain-adaptive teacher selection (Su et al., 2022).
• DiReDi and TrajSyn realize privacy-preserving domain adaptation and adversarial robustness, outperforming direct local fine-tuning or centralized retraining (Sun et al., 2024, Gupta et al., 17 Dec 2025).
• EdgeFD consistently exceeds prior state-of-the-art in non-IID settings while reducing computational footprint by an order of magnitude via KMeans filtering (Mujtaba et al., 20 Aug 2025).

These results collectively validate server-client distillation as a foundation for scalable, heterogeneous, privacy-aware, and communication-efficient federated learning.