Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Learning via Input-Output Collaborative Distillation (2312.14478v1)

Published 22 Dec 2023 in cs.LG

Abstract: Federated learning (FL) is a machine learning paradigm in which distributed local nodes collaboratively train a central model without sharing individually held private data. Existing FL methods either iteratively share local model parameters or deploy co-distillation. However, the former is highly susceptible to private data leakage, and the latter design relies on the prerequisites of task-relevant real data. Instead, we propose a data-free FL framework based on local-to-central collaborative distillation with direct input and output space exploitation. Our design eliminates any requirement of recursive local parameter exchange or auxiliary task-relevant data to transfer knowledge, thereby giving direct privacy control to local users. In particular, to cope with the inherent data heterogeneity across locals, our technique learns to distill input on which each local model produces consensual yet unique results to represent each expertise. Our proposed FL framework achieves notable privacy-utility trade-offs with extensive experiments on image classification and segmentation tasks under various real-world heterogeneous federated learning settings on both natural and medical images.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint arXiv:1909.08097.
  2. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629.
  3. Bakas, S. S. 2020. Brats MICCAI Brain tumor dataset.
  4. Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer. arXiv preprint arXiv:1912.11279.
  5. Synthetic learning: Learn from distributed asynchronized discriminator GAN without sharing medical image data. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13856–13866.
  6. Gs-wgan: A gradient-sanitized approach for learning differentially private generators. Advances in Neural Information Processing Systems, 33: 12673–12684.
  7. DCAN: deep contour-aware networks for accurate gland segmentation. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2487–2496.
  8. Data-free learning of student networks. In Proceedings of IEEE/CVF International Conference on Computer Vision, 3514–3522.
  9. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248–255. Ieee.
  10. Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data. In Proceedings of Conference on Neural Information Processing Systems.
  11. Data-free adversarial distillation. arXiv preprint arXiv:1912.11006.
  12. Inverting Gradients–How easy is it to break privacy in federated learning? arXiv:2003.14053.
  13. Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation. In Association for the Advancement of Artificial Intelligence.
  14. Federated Learning with Privacy-Preserving Ensemble Attention Distillation. IEEE Transactions on Medical Imaging.
  15. Generative adversarial networks. Communications of the ACM, 63(11): 139–144.
  16. Multi-Institutional Collaborations for Improving Deep Learning-Based Magnetic Resonance Image Reconstruction Using Federated Learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  18. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  19. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.
  20. Federated Visual Classification with Real-World Data Distribution. In Proceedings of European Conference on Computer Vision.
  21. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1125–1134.
  22. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479.
  23. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of European conference on computer vision, 694–711. Springer.
  24. Scaffold: Stochastic controlled averaging for on-device federated learning. In Proceedings of International Conference on Machine Learning.
  25. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Transactions on Medical Imaging, 36(7): 1550–1560.
  26. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581.
  27. Practical one-shot federated learning for cross-silo setting. In Proceedings of International Joint Conference on Artificial Intelligence.
  28. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127.
  29. Fair resource allocation in federated learning. In Proceedings of International Conference on Learning Representations.
  30. Ensemble Distillation for Robust Model Fusion in Federated Learning. In Proceedings of Conference on Neural Information Processing Systems.
  31. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282. PMLR.
  32. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10): 1993–2024.
  33. Zero-shot knowledge distillation in deep networks. In Proceedings of International Conference on Machine Learning, 4743–4751. PMLR.
  34. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Transactions on Medical Imaging, 38(2): 448–459.
  35. Scalable private learning with pate. arXiv preprint arXiv:1802.08908.
  36. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
  37. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention.
  38. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
  39. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 0–0.
  40. Methods for segmentation and classification of digital microscopy tissue images. Frontiers in bioengineering and biotechnology, 53.
  41. Federated learning with matched averaging. In Proceedings of International Conference on Learning Representations.
  42. Distilled person re-identification: Towards a more scalable system. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1187–1196.
  43. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In Proceedings of European Conference on Computer Vision, 247–263. Springer.
  44. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8715–8724.
  45. fastMRI: An Open Dataset and Benchmarks for Accelerated MRI. ArXiv 1811.08839.
  46. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10174–10183.
  47. FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models. In 2022 IEEE 42nd International Conference on Distributed Computing Systems, 928–938. IEEE.
  48. Deep leakage from gradients. In Proceedings of Conference on Neural Information Processing Systems, 14774–14784.
  49. Data-free knowledge distillation for heterogeneous federated learning. In Proceedings of International Conference on Machine Learning, 12878–12889. PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Xuan Gong (16 papers)
  2. Shanglin Li (7 papers)
  3. Yuxiang Bao (5 papers)
  4. Barry Yao (1 paper)
  5. Yawen Huang (40 papers)
  6. Ziyan Wu (59 papers)
  7. Baochang Zhang (113 papers)
  8. Yefeng Zheng (197 papers)
  9. David Doermann (54 papers)
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now