Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients (2401.02433v1)

Published 16 Nov 2023 in cs.CV, cs.AI, and cs.LG

Abstract: With the rapid development of imaging sensor technology in the field of remote sensing, multi-modal remote sensing data fusion has emerged as a crucial research direction for land cover classification tasks. While diffusion models have made great progress in generative models and image classification tasks, existing models primarily focus on single-modality and single-client control, that is, the diffusion process is driven by a single modal in a single computing node. To facilitate the secure fusion of heterogeneous data from clients, it is necessary to enable distributed multi-modal control, such as merging the hyperspectral data of organization A and the LiDAR data of organization B privately on each base station client. In this study, we propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Our key insight is that diffusion models driven by different modalities are inherently complementary in terms of potential denoising steps on which bilateral connections can be built. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure, and introduce a lightweight communication module. Qualitative and quantitative experiments validate the superiority of our framework in terms of image quality and conditional consistency.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. K. Song, L. Huang, A. Gong, and Y. Yan, “Multiple graph affinity interactive network and a variable illumination dataset for rgbt image salient object detection,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  2. Y. Wang, Y. Mao, Q. Liu, and Y. Dai, “Decomposed guided dynamic filters for efficient rgb-guided depth completion,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  3. M. Ma, W. Ma, L. Jiao, X. Liu, F. Liu, L. Li, and S. Yang, “Mbsi-net: Multimodal balanced self-learning interaction network for image classification,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  4. K. Ding, T. Lu, W. Fu, S. Li, and F. Ma, “Global–local transformer network for hsi and lidar data joint classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.
  5. W. Xiong, Z. Xiong, Y. Cui, L. Huang, and R. Yang, “An interpretable fusion siamese network for multi-modality remote sensing ship image retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  6. S. K. Roy, A. Deria, D. Hong, B. Rasti, A. Plaza, and J. Chanussot, “Multimodal fusion transformer for remote sensing image classification,” arXiv preprint arXiv:2203.16952, 2022.
  7. Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte, and L. Van Gool, “Ddfm: denoising diffusion model for multi-modality image fusion,” arXiv preprint arXiv:2303.06840, 2023.
  8. Z. Wu, Y. Li, A. Plaza, J. Li, F. Xiao, and Z. Wei, “Parallel and distributed dimensionality reduction of hyperspectral data on cloud computing architectures,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 9, no. 6, pp. 2270–2278, 2016.
  9. N. Li, H. Zhang, Z. Lv, L. Min, and Z. Guo, “Simultaneous screening and detection of rfi from massive sar images: A case study on european sentinel-1,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–17, 2022.
  10. C. Utz, M. Degeling, S. Fahl, F. Schaub, and T. Holz, “(un) informed consent: Studying gdpr consent notices in the field,” in Proceedings of the 2019 Acm Sigsac Conference on Computer and Communications Security, 2019, pp. 973–990.
  11. J. Liang, D. Hu, Y. Wang, R. He, and J. Feng, “Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8602–8617, 2021.
  12. M. Jegorova, C. Kaul, C. Mayor, A. Q. O’Neil, A. Weir, R. Murray-Smith, and S. A. Tsaftaris, “Survey: Leakage and privacy at inference time,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–20, 2022.
  13. B. Yang, H.-W. Yeh, T. Harada, and P. C. Yuen, “Model-induced generalization error bound for information-theoretic representation learning in source-data-free unsupervised domain adaptation,” IEEE Transactions on Image Processing, vol. 31, pp. 419–432, 2021.
  14. Z. Huang, K. C. Chan, Y. Jiang, and Z. Liu, “Collaborative diffusion for multi-modal face generation and editing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 6080–6090.
  15. N. Audebert, B. Le Saux, and S. Lefèvre, “Beyond rgb: Very high resolution urban remote sensing with multimodal deep networks,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 140, pp. 20–32, 2018, geospatial Computer Vision. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0924271617301818
  16. Q. He, X. Sun, W. Diao, Z. Yan, F. Yao, and K. Fu, “Multimodal remote sensing image segmentation with intuition-inspired hypergraph modeling,” IEEE Transactions on Image Processing, vol. 32, pp. 1474–1487, 2023.
  17. D. Hong, L. Gao, R. Hang, B. Zhang, and J. Chanussot, “Deep encoder–decoder networks for classification of hyperspectral and lidar data,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
  18. P. Bellmann, P. Thiam, and F. Schwenker, “Using meta labels for the training of weighting models in a sample-specific late fusion classification architecture,” in 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 2604–2611.
  19. Y. Guo, Z. Li, X. Zhang, E.-x. Chen, L. Bai, X. Tian, Q. He, Q. Feng, and W. Li, “Optimal support vector machines for forest above-ground biomass estimation from multisource remote sensing data,” in 2012 IEEE International Geoscience and Remote Sensing Symposium, 2012, pp. 6388–6391.
  20. X. Luo, G. Fu, J. Yang, Y. Cao, and Y. Cao, “Multi-modal image fusion via deep laplacian pyramid hybrid network,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  21. S. Fang, K. Li, and Z. Li, “S22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTenet: Spatial–spectral cross-modal enhancement network for classification of hyperspectral and lidar data,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2021.
  22. D. Hoffmann, K. N. Clasen, and B. Demir, “Transformer-based multi-modal learning for multi label remote sensing image classification,” arXiv preprint arXiv:2306.01523, 2023.
  23. P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 873–12 883.
  24. D. Baranchuk, I. Rubachev, A. Voynov, V. Khrulkov, and A. Babenko, “Label-efficient semantic segmentation with diffusion models,” arXiv preprint arXiv:2112.03126, 2021.
  25. X. Han, H. Zheng, and M. Zhou, “Card: Classification and regression diffusion models,” Advances in Neural Information Processing Systems, vol. 35, pp. 18 100–18 115, 2022.
  26. J. Yue, L. Fang, S. Xia, Y. Deng, and J. Ma, “Dif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion models,” arXiv preprint arXiv:2301.08072, 2023.
  27. V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  28. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020.
  29. T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450, 2020.
  30. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning.   PMLR, 2020, pp. 5132–5143.
  31. J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” Advances in Neural Information Processing Systems, vol. 33, pp. 7611–7623, 2020.
  32. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics.   PMLR, 2017, pp. 1273–1282.
  33. Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE).   IEEE, 2022, pp. 965–978.
  34. K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, B. McMahan et al., “Towards federated learning at scale: System design,” Proceedings of Machine Learning and Systems, vol. 1, pp. 374–388, 2019.
  35. C. Van Berkel, “Multi-core for mobile phones,” in 2009 Design, Automation & Test in Europe Conference & Exhibition.   IEEE, 2009, pp. 1260–1265.
  36. J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492, 2016.
  37. W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, and E. Xing, “High-performance distributed ml at scale through parameter server consistency models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1, 2015.
  38. J. Ren, Y. He, D. Wen, G. Yu, K. Huang, and D. Guo, “Scheduling for cellular federated edge learning with importance and channel awareness,” IEEE Transactions on Wireless Communications, vol. 19, no. 11, pp. 7690–7703, 2020.
  39. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International Conference on Machine Learning.   PMLR, 2015, pp. 2256–2265.
  40. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
  41. W. Gedara Chaminda Bandara, N. Gopalakrishnan Nair, and V. M. Patel, “Remote sensing change detection (segmentation) using denoising diffusion probabilistic models,” arXiv e-prints, pp. arXiv–2206, 2022.
  42. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4713–4726, 2022.
  43. H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, “Multiscale vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6824–6835.
  44. F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Transactions on Geoscience and Remote Sensing, vol. 42, no. 8, pp. 1778–1790, 2004.
  45. K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, “Deep supervised learning for hyperspectral data classification through convolutional neural networks,” in 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).   IEEE, 2015, pp. 4959–4962.
  46. K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” arXiv preprint arXiv:1409.1259, 2014.
  47. D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 5, pp. 4340–4354, 2020.
  48. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Weiying Xie (31 papers)
  2. YiBing Lu (1 paper)
  3. Yunsong Li (41 papers)
  4. Leyuan Fang (26 papers)
  5. Daixun Li (9 papers)
  6. Zixuan Wang (82 papers)
Citations (9)
View on Semantic Scholar