Decentralized and Data-free KD Methods

• Decentralized and data-free KD is a set of techniques that use distilled model outputs instead of raw data to address privacy and non-IID challenges in distributed learning.
• It encompasses frameworks like HFedCKD, FedGen, and DIP-KD, which employ methods such as synthetic data generation, generator-based distillation, and peer-to-peer communication.
• These approaches achieve improved accuracy, lower communication costs, and enhanced scalability by transferring function-space knowledge rather than raw data.

Decentralized and Data-free Knowledge Distillation (KD) refers to a class of frameworks and methodologies for transferring knowledge across distributed learning agents or clients without requiring access to the raw training data and often outside the conventional centralized server-client paradigm. These approaches are motivated by severe privacy constraints, data heterogeneity, and scalability needs in federated learning (FL), edge computing, and other distributed AI deployments. The core idea is to enable model improvement or consensus via distilled “knowledge” (e.g., soft labels, logits, or synthetic samples) rather than direct data or weight sharing.

1. Overview and Problem Formulation

In decentralized and data-free KD, each participating node (client or agent) maintains local data—which can be highly non-IID and imbalanced—and aims to improve its model or contribute to a shared model without uploading raw data. The system topology may be fully peer-to-peer (no central server), hierarchical with local aggregators, or server-based but data-opaque.

Knowledge distillation replaces direct weight aggregation or data sharing by communicating function-space information, such as output logits, soft labels, or ensemble predictions, possibly on synthetic or public data, as in the function-space averaging objective: $$\min_{w}\;\int\!div\bigl(\Phi^*(x),\Phi(x;w)\bigr)\,P(dx)$$ where $div$ is a suitable function-space divergence (e.g., KL or TV distance), $P$ is a mixture of local data sources $P_k(x)$, and $\Phi(x;w)$ are global or local models (Li et al., 2022).

2. Approaches in Server-based Data-free Federated Learning

In the classical FL context, data-free KD is employed to address user and system heterogeneity. Notable frameworks include HFedCKD (Zheng et al., 9 Mar 2025) and FedGen (Zhu et al., 2021).

HFedCKD uses a parameter server to coordinate $N$ clients, of which only a sampled subset $\mathcal C_t$ ($|\mathcal C_t|=fN$) participates per round. Each uploads distilled knowledge rather than raw gradients or data. Contribution of each client $i$ is weighted by a combination of participation frequency $\pi_i$ and distributional divergence $div$0: $div$1 This weighting corrects for under-representation and distributional mismatch, and the weights are applied in an aggregation of distillation terms: $div$2 Baseline protocols perform two rounds of data-free KD per global round in both non-IID and model-heterogeneous scenarios (Zheng et al., 9 Mar 2025). Experiments focus on image (Fashion-MNIST, CIFAR-100, Tiny-ImageNet) and IoT datasets with rigorous participation controls.

FedGen proposes a generator-based distillation mechanism: the server learns a lightweight conditional generator $div$3 that approximates the feature distribution pooled across clients. Clients use this generator to augment local training with synthetic features, enforcing global consensus and reducing drift under heterogeneity. The key algorithmic steps are:

• Server maintains $div$4, $div$5, and $div$6
• Each client alternates local ERM with distillation on synthetic features $div$7
• Server aggregates predictors and updates the generator based on global label priors estimated from client label counts

These methods eschew any use of true data or public proxy sets at the server, relying entirely on model outputs for synthesis and distillation (Zhu et al., 2021).

3. Fully Decentralized Data-Free KD: Peer-to-Peer and Serverless Schemes

Protocols such as “Real-Time Decentralized Knowledge Transfer at the Edge” (Goldstein et al., 2020) and IDKD (“In-Distribution Knowledge Distillation”) (Ravikumar et al., 2023) realize serverless, peer-to-peer data-free KD by direct communication between clients arranged in network topologies (e.g., ring, fully connected, small-world).

Real-Time Decentralized KD

• Agents share only model architectures and class histograms.
• Each agent selects a peer set $div$8 based on task overlap and model compatibility.
• Training alternates supervised local updates and remote distillation:
  • Local: cross-entropy loss on private labels
  • Remote: KL divergence on peer softmax outputs, adaptively aligned via small “pipeline” modules to support heterogeneous architectures
  • Loss: $div$9
• No raw data or synthetic samples are exchanged; only soft outputs and model weights (Goldstein et al., 2020).

In-Distribution KD (IDKD)

• Assumes full copies of a public auxiliary dataset $P$0 at each node.
• Each node uses its own model to score $P$1 for in-distribution examples using negative entropy or energy-based criteria, forming $P$2.
• Nodes exchange only soft labels $P$3 with neighbors, average them, and fine-tune via: $P$4 where $P$5 is the empirical loss on private data, $P$6 the distillation loss on public examples (Ravikumar et al., 2023).

Empirically, such schemes yield lower communication (soft-label vectors rather than gradient or parameter blocks) and outperform decentralized SGD in non-IID regimes by up to $P$7 on CIFAR-10, as shown in (Ravikumar et al., 2023).

4. Data-free KD in Black-box and Privacy-restricted Environments

Recent work has extended data-free KD to settings where even teacher logits or features are inaccessible: only top-1 predictions are available, as exemplified by DIP-KD (“Diverse Image Priors for Black-box Data-free Knowledge Distillation”) (Vo et al., 28 Apr 2026).

DIP-KD implements a three-phase pipeline:

1. Synthesis: Generates diverse synthetic images via hierarchical noise, nonlinear transformations, and CutMix, forming a prior pool $P$8.
2. Contrast: Applies contrastive learning to maximize feature diversity in the synthetic set using a primer student $P$9, thereby increasing the efficacy of hard-label KD.
3. Distillation: Final student $P_k(x)$0 is trained using $P_k(x)$1, a sum of hard-label KD (using teacher top-1) and soft-KD (using $P_k(x)$2's logits).

This architecture achieves state-of-the-art results in black-box, query-limited settings, with up to $P_k(x)$3 accuracy gains over earlier methods on Imagenette (Vo et al., 28 Apr 2026). Importantly, all phases can run locally or via distributed collaboration, with the teacher acting purely as a labeling oracle. The requirement for neither teacher features nor original data makes DIP-KD a candidate for the strictest privacy-first deployments.

5. Theoretical Justification and Guarantees

Several theoretical analyses provide insight and guarantees for data-free decentralized KD:

• (Li et al., 2022) shows that parameter averaging is suboptimal under heterogeneity and that function-space (distillation-based) objectives more closely approach the integrated global optimum, especially when the divergence metric satisfies the triangle inequality.
• Generalization theory (see Theorem 1, (Zhu et al., 2021)) demonstrates that adding synthetic “global” feature samples reduces both distributional discrepancy and estimation error, tightening risk bounds. Provided that the synthetic generator well approximates aggregate client predictors, such approaches improve statistical efficiency and final accuracy.
• Under mild smoothness and variance assumptions, decentralized KD schemes converge to near-optimal nodes of a surrogate global loss, at rates similar to decentralized SGD (see (Ravikumar et al., 2023)).

6. Empirical Performance and Communication Complexity

Experiments across methods consistently show that decentralized and data-free KD:

• Achieves absolute accuracy gains of $P_k(x)$4–$P_k(x)$5 over FedAvg/SGD, with up to $P_k(x)$6 fewer communication rounds and $P_k(x)$7–$P_k(x)$8 fewer local update steps for fixed accuracy (Li et al., 2022, Zhu et al., 2021).
• Dramatically reduces the per-round communication cost: e.g., one node transmits $P_k(x)$9 KB/round (soft labels) in IDKD, compared to $\Phi(x;w)$0 MB/round (full gradients) in DSGD (Ravikumar et al., 2023).
• Enables robust operation under high client heterogeneity, low participation, and with partial activation in each round.

A summary table comparing representative methods:

Framework	Topology	Data-Free	Synthetic/Proxy Data	Communication Unit
HFedCKD (Zheng et al., 9 Mar 2025)	Server-client	Yes	Yes (synth. generator)	Model, distillation loss (weighted)
FedGen (Zhu et al., 2021)	Server-client	Yes	Yes (synth. generator)	Model, generator
Real-Time Dec (Goldstein et al., 2020)	Peer-to-peer	Yes	No	Soft logits, model weights
IDKD (Ravikumar et al., 2023)	Peer-to-peer	Yes	Yes (public data)	Soft labels
DIP-KD (Vo et al., 28 Apr 2026)	Student-teacher	Yes	Yes (synth. images)	Label queries

7. Practical Considerations and Limitations

Key operational aspects include:

• Scalability: Peer-to-peer and hybrid schemes are naturally scalable if agents maintain reasonable communications degree (Goldstein et al., 2020).
• Privacy: Data-free KD minimizes privacy risk; however, generator-based methods may be vulnerable to inversion unless properly regularized (Zhu et al., 2021).
• Generator/Proxy Quality: Generator- or synthetic-prior-based approaches are sensitive to the quality and diversity of generated samples. Poorly trained generators can degrade performance (Zhu et al., 2021, Vo et al., 28 Apr 2026).
• Public Data Limitations: In schemes reliant on auxiliary public datasets (IDKD), success depends critically on sufficient overlap between $\Phi(x;w)$1 and the true distribution (Ravikumar et al., 2023).
• Communication/Computation Tradeoffs: While data-free KD reduces communication load, computational cost may rise when optimizing synthetic data pools (e.g., DIP-KD uses millions of teacher queries for contrastive phase) (Vo et al., 28 Apr 2026).

Extensions to asynchronous updates, model heterogeneity, and privacy-preserving generator updates are active research directions. In summary, decentralized and data-free KD represents a convergence of privacy, communication-efficiency, and robustness in distributed learning, with rigorous theoretical foundations and empirical success across diverse scenarios.