Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation (2403.11803v1)

Published 18 Mar 2024 in cs.CV

Abstract: Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Federated learning and differential privacy for medical image analysis. Scientific Reports, 12(1): 1953.
  2. SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. In International Conference on Medical Imaging with Deep Learning, 48–62. PMLR.
  3. Bakas, S.; et al. 2018. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629.
  4. FedDis: Disentangled federated learning for unsupervised brain pathology segmentation. arXiv preprint arXiv:2103.03705.
  5. FedNorm: Modality-based normalization in federated learning for multi-modal liver segmentation. arXiv preprint arXiv:2205.11096.
  6. Brain tumor segmentation with deep convolutional symmetric neural network. Neurocomputing, 392: 305–313.
  7. FedMSplit: Correlation-adaptive federated multi-task learning across multimodal split networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 87–96.
  8. Dual-force convolutional neural networks for accurate brain tumor segmentation. Pattern Recognition, 88: 90–100.
  9. A unified framework for generalized low-shot medical image segmentation with scarce data. IEEE Transactions on Medical Imaging, 40(10): 2656–2671.
  10. A multi-path adaptive fusion network for multimodal brain tumor segmentation. Neurocomputing, 412: 19–30.
  11. RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3975–3984.
  12. Hetero-modal variational encoder-decoder for joint modality completion and segmentation. In Medical Image Computing and Computer Assisted Intervention, 74–82. Springer.
  13. Federated visual classification with real-world data distribution. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, 76–92. Springer.
  14. Hu, M.; et al. 2020. Knowledge distillation from multi-modal to mono-modal segmentation networks. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23., 772–781. Springer.
  15. Current clinical state of advanced magnetic resonance imaging for brain tumor diagnosis and follow up. In Seminars in Roentgenology, volume 53, 45–61. Elsevier.
  16. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6): 305–311.
  17. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3): 50–60.
  18. Federated learning for vision-and-language grounding problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 11572–11579.
  19. MacQueen, J.; et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, 281–297. Oakland, CA, USA.
  20. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282. PMLR.
  21. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10): 1993–2024.
  22. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In Fourth International Conference on 3D Vision, 565–571. IEEE.
  23. FedProc: Prototypical contrastive federated learning on non-IID data. Future Generation Computer Systems, 143: 93–104.
  24. Myronenko, A. 2018. 3D MRI brain tumor segmentation using autoencoder regularization. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4., 311–320. Springer.
  25. Multi-anchor active domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 9112–9122.
  26. Brain tumor segmentation on MRI with missing modalities. In Information Processing in Medical Imaging, 417–428. Springer.
  27. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems.
  28. FedProto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8432–8440.
  29. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30: 1195–1204.
  30. Attention is all you need. Advances in Neural Information Processing Systems, 30: 5998–6008.
  31. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252.
  32. ACN: Adversarial co-training network for brain tumor segmentation with missing modalities. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VII 24., 410–420. Springer.
  33. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, 478–487. PMLR.
  34. A unified framework for multi-modal federated learning. Neurocomputing, 480: 110–118.
  35. Experiments of federated learning for COVID-19 chest X-ray images. In Advances in Artificial Intelligence and Security, 41–53. Springer International Publishing.
  36. Variation-aware federated learning with multi-source decentralized medical image data. IEEE Journal of Biomedical and Health Informatics, 25(7): 2615–2628.
  37. Multimodal federated learning via contrastive representation ensemble. In The Eleventh International Conference on Learning Representations.
  38. Multimodal federated learning on IoT data. In IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation, 43–54. IEEE.
  39. One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Transactions on Image Processing, 2020, 29: 4516-4529., 29: 4516–4529.
  40. Latent correlation representation learning for brain tumor segmentation with missing MRI modalities. IEEE Transactions on Image Processing, 30: 4263–4274.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Qian Dai (3 papers)
  2. Dong Wei (70 papers)
  3. Hong Liu (394 papers)
  4. Jinghan Sun (12 papers)
  5. Liansheng Wang (48 papers)
  6. Yefeng Zheng (197 papers)
Citations (1)
View on Semantic Scholar