Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach (2404.03702v1)

Published 4 Apr 2024 in cs.LG and cs.AI

Abstract: The existing federated learning (FL) methods for spatio-temporal forecasting fail to capture the inherent spatio-temporal heterogeneity, which calls for personalized FL (PFL) methods to model the spatio-temporally variant patterns. While contrastive learning approach is promising in addressing spatio-temporal heterogeneity, the existing methods are noneffective in determining negative pairs and can hardly apply to PFL paradigm. To tackle this limitation, we propose a novel PFL method, named Federated dUal sEmantic aLignment-based contraStive learning (FUELS), which can adaptively align positive and negative pairs based on semantic similarity, thereby injecting precise spatio-temporal heterogeneity into the latent representation space by auxiliary contrastive tasks. From temporal perspective, a hard negative filtering module is introduced to dynamically align heterogeneous temporal representations for the supplemented intra-client contrastive task. From spatial perspective, we design lightweight-but-efficient prototypes as client-level semantic representations, based on which the server evaluates spatial similarity and yields client-customized global prototypes for the supplemented inter-client contrastive task. Extensive experiments demonstrate that FUELS outperforms state-of-the-art methods, with communication cost decreasing by around 94%.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Federated learning with personalization layers. arXiv preprint arXiv:1912.00818, 2019.
  2. A multi-source dataset of urban life in the city of milan and the province of trentino. Scientific data, 2(1):1–15, 2015.
  3. Federated user representation learning. arXiv preprint arXiv:1909.12535, 2019.
  4. Efficient personalized federated learning via sparse model-adaptation. arXiv preprint arXiv:2305.02776, 2023.
  5. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.  1597–1607. PMLR, 2020.
  6. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pp.  2089–2099. PMLR, 2021.
  7. Personalized federated learning with moreau envelopes. In Advances in Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020.
  8. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557–3568, 2020.
  9. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586–19597, 2020.
  10. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  11. Spatio-temporal self-supervised learning for traffic flow prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.  4356–4364, 2023.
  12. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  13. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp.  5132–5143. PMLR, 2020.
  14. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  15. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  16. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  17. Fedsae: A novel self-adaptive federated learning framework in heterogeneous systems. In 2021 International Joint Conference on Neural Networks (IJCNN), pp.  1–10. IEEE, 2021a.
  18. Spatial-temporal fusion graph neural networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.  4189–4196, 2021.
  19. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10713–10722, 2021b.
  20. Mining spatio-temporal relations via self-paced graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.  936–944, 2022.
  21. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  22. Federated meta-learning for spatial-temporal prediction. Neural Computing and Applications, 34(13):10355–10374, 2022.
  23. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR ’18), 2018.
  24. Privacy-preserving spatiotemporal scenario generation of renewable energies: A federated deep generative learning approach. IEEE Transactions on Industrial Informatics, 18(4):2310–2320, 2021c.
  25. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
  26. A data-driven base station sleeping strategy based on traffic prediction. IEEE Transactions on Network Science and Engineering, 2021.
  27. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  28. Online spatio-temporal correlation-based federated learning for traffic flow forecasting. arXiv preprint arXiv:2302.08658, 2023.
  29. When do contrastive learning signals help spatio-temporal graph forecasting? In Proceedings of the 30th International Conference on Advances in Geographic Information Systems, pp.  1–12, 2022.
  30. Privacy-preserving traffic flow prediction: A federated learning approach. IEEE Internet of Things Journal, 7(8):7751–7763, 2020.
  31. A clustering-driven approach to predict the traffic load of mobile networks for the analysis of base stations deployment. Journal of Sensor and Actuator Networks, 9(4):53, 2020.
  32. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  33. Cross-node federated graph neural network for spatio-temporal data modeling. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp.  1202–1211, 2021.
  34. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, 143:93–104, 2023.
  35. Federated learning for 5g base station traffic forecasting. Computer Networks, 235:109950, 2023.
  36. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
  37. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022a.
  38. Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 35:19332–19344, 2022b.
  39. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  2495–2504, 2021.
  40. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575, 2022.
  41. Multi-center federated learning. arXiv preprint arXiv:2108.08647, 2021.
  42. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp.  2174–2178. IEEE, 2021.
  43. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812–5823, 2020.
  44. Multimodal federated learning via contrastive representation ensemble. In The Eleventh International Conference on Learning Representations, 2022.
  45. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.  8980–8987, 2022.
  46. Dual attention-based federated learning for wireless traffic prediction. In IEEE INFOCOM 2021-IEEE conference on computer communications, pp.  1–10. IEEE, 2021.
  47. Efficient wireless traffic prediction at the edge: A federated meta-learning approach. IEEE Communications Letters, 26(7):1573–1577, 2022.
  48. Cellular network traffic prediction incorporating handover: A graph convolutional approach. In 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp.  1–9. IEEE, 2020.
  49. Joint traffic prediction and base station sleeping for energy saving in cellular networks. In ICC 2021-IEEE International Conference on Communications, pp.  1–6. IEEE, 2021.
  50. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pp.  2069–2080, 2021a.
  51. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning, pp.  12878–12889. PMLR, 2021b.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets