Bridging Data Islands: Geographic Heterogeneity-Aware Federated Learning for Collaborative Remote Sensing Semantic Segmentation (2404.09292v2)
Abstract: Remote sensing semantic segmentation (RSS) is an essential technology in earth observation missions. Due to concerns over geographic information security, data privacy, storage bottleneck and industry competition, high-quality annotated remote sensing images are often isolated and distributed across institutions. The issue of remote sensing data islands poses challenges for fully utilizing isolated datasets to train a global model. Federated learning (FL), a privacy-preserving distributed collaborative learning technology, offers a potential solution to leverage isolated remote sensing data. Typically, remote sensing images from different institutions exhibit significant geographic heterogeneity, characterized by coupled class-distribution heterogeneity and object-appearance heterogeneity. However, existing FL methods lack consideration of them, leading to a decline in the performance of the global model when FL is directly applied to RSS. We propose a novel Geographic heterogeneity-aware Federated learning (GeoFed) framework to bridge data islands in RSS. Our framework consists of three modules, including the Global Insight Enhancement (GIE) module, the Essential Feature Mining (EFM) module and the Local-Global Balance (LoGo) module. Through the GIE module, class distribution heterogeneity is alleviated by introducing a prior global class distribution vector. We design an EFM module to alleviate object appearance heterogeneity by constructing essential features. Furthermore, the LoGo module enables the model to possess both global generalization capability and local adaptation. Extensive experiments on three public datasets (i.e., FedFBP, FedCASID, FedInria) demonstrate that our GeoFed framework consistently outperforms the current state-of-the-art methods.
- Differentially private secure multi-party computation for federated learning in financial applications. In Proceedings of the First ACM International Conference on AI in Finance, pages 1–9, 2020.
- Stereo attention cross-decoupling fusion-guided federated neural learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 61:1–16, 2023.
- Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pages 654–672. Springer, 2022.
- Elastic aggregation for federated optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12187–12197, 2023a.
- Fraug: Tackling federated learning with non-iid features via representation augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4849–4859, 2023b.
- Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
- Federated learning-based aerial image segmentation for collision-free movement and landing. In Proceedings of the 4th ACM MobiCom Workshop on Drone Assisted Wireless Communications for 5G and Beyond, pages 13–18, 2021.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
- Federated incremental semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3934–3943, 2023a.
- Large-scale land cover mapping with fine-grained classes via class-aware semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16783–16793, 2023b.
- Feddrive: Generalizing federated learning to semantic segmentation in autonomous driving. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11504–11511, 2022.
- Michael F Goodchild. The validity and usefulness of laws in geographic information science and geography. Annals of the Association of American Geographers, 94(2):300–303, 2004.
- Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Mapping local climate zones for cities: A large review. Remote Sensing of Environment, 292:113573, 2023a.
- Style projected clustering for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3061–3071, 2023b.
- Rethinking federated learning with domain shift: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16312–16322, 2023c.
- A decentralized federated learning paradigm for semantic segmentation of geospatial data. In Intelligent Computing & Optimization: Proceedings of the 4th International Conference on Intelligent Computing and Optimization 2021 (ICO2021) 3, pages 196–206. Springer, 2022.
- Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978, 2022.
- Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, pages 429–450, 2020.
- Sinolc-1: the first 1 m resolution national-scale land-cover map of china created with a deep learning framework and open-access data. Earth System Science Data, 15(11):4749–4780, 2023.
- Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1013–1023, 2021a.
- A large-scale climate-aware satellite image dataset for domain adaptive land-cover semantic segmentation. ISPRS Journal of Photogrammetry and Remote Sensing, 205:98–114, 2023.
- Federated learning in the sky: Aerial-ground air quality sensing framework with uav swarms. IEEE Internet of Things Journal, 8(12):9827–9837, 2021b.
- Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pages 3226–3229, 2017.
- Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
- Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8397–8406, 2022.
- Fedseg: Class-heterogeneous federated learning for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8042–8052, 2023.
- Federated learning via attentive margin of semantic feature representations. IEEE Internet of Things Journal, 10(2):1517–1535, 2023.
- Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10061–10071, 2022.
- Land use mapping using sentinel-1 and sentinel-2 time series in a heterogeneous landscape in niger, sahel. ISPRS Journal of Photogrammetry and Remote Sensing, 178:97–111, 2021.
- Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2218–2224, 2022.
- Amp: Adaptive masked proxies for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
- Federated target recognition for multiradar sensor data security. IEEE Transactions on Geoscience and Remote Sensing, 61:1–10, 2023.
- Adaptive resource optimized edge federated learning in real-time image sensing classifications. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:10929–10940, 2021.
- Google earth engine for geo-big data applications: A meta-analysis and systematic review. ISPRS Journal of Photogrammetry and Remote Sensing, 164:152–170, 2020.
- Dynamicearthnet: Daily multi-spectral satellite dataset for semantic change segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21158–21167, 2022.
- Enabling country-scale land cover mapping with meter-resolution satellite imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 196:178–196, 2023.
- Dafkd: Domain-aware federated knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20412–20421, 2023.
- Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3349–3364, 2021.
- Endpoints weight fusion for class incremental semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7204–7213, 2023.
- Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
- Closing the generalization gap of cross-silo federated medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20866–20875, 2022.
- Ai security for geoscience and remote sensing: Challenges and future trends. IEEE Geoscience and Remote Sensing Magazine, 11(2):60–85, 2023.
- Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
- Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56(3):1–44, 2023.
- Global balanced experts for federated long-tailed learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4815–4825, 2023.
- Prototype-based clustered federated learning for semantic segmentation of aerial images. In IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, pages 2227–2230, 2022.
- Federated deep learning with prototype matching for object extraction from very-high-resolution remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 61:1–16, 2023a.
- Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10795–10816, 2023b.
- Local differential privacy based membership-privacy-preserving federated learning for deep-learning-driven remote sensing. Remote Sensing, 15(20):5050, 2023c.
- Mining the regularity of landscape-structure heterogeneity to improve urban land-cover mapping. Remote Sensing of Environment, 214:14–32, 2018.
- Global urban high-resolution land-use mapping: From benchmarks to multi-megacity applications. Remote Sensing of Environment, 298:113758, 2023.