Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning (2311.13250v2)

Published 22 Nov 2023 in cs.CV and cs.LG

Abstract: Federated Learning (FL) enables joint training across distributed clients using their local data privately. Federated Multi-Task Learning (FMTL) builds on FL to handle multiple tasks, assuming model congruity that identical model architecture is deployed in each client. To relax this assumption and thus extend real-world applicability, we introduce a novel problem setting, Hetero-Client Federated Multi-Task Learning (HC-FMTL), to accommodate diverse task setups. The main challenge of HC-FMTL is the model incongruity issue that invalidates conventional aggregation methods. It also escalates the difficulties in accurate model aggregation to deal with data and task heterogeneity inherent in FMTL. To address these challenges, we propose the FedHCA$2$ framework, which allows for federated training of personalized models by modeling relationships among heterogeneous clients. Drawing on our theoretical insights into the difference between multi-task and federated optimization, we propose the Hyper Conflict-Averse Aggregation scheme to mitigate conflicts during encoder updates. Additionally, inspired by task interaction in MTL, the Hyper Cross Attention Aggregation scheme uses layer-wise cross attention to enhance decoder interactions while alleviating model incongruity. Moreover, we employ learnable Hyper Aggregation Weights for each client to customize personalized parameter updates. Extensive experiments demonstrate the superior performance of FedHCA$2$ in various HC-FMTL scenarios compared to representative methods. Our code will be made publicly available.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (92)
  1. Federated learning based on dynamic regularization. In ICLR, 2021.
  2. Federated learning with personalization layers. CoRR, abs/1912.00818, 2019.
  3. How to backdoor federated learning. In AISTATS, pages 2938–2948, 2020.
  4. Advancing COVID-19 diagnosis with privacy-preserving collaboration in artificial intelligence. Nature Machine Intelligence, 3(12):1081–1089, 2021.
  5. Federated disentangled representation learning for unsupervised brain anomaly detection. Nature Machine Intelligence, 4(8):685–695, 2022.
  6. Automated search for resource-efficient branched multi-task networks. In BMVC, page 359, 2020.
  7. Exploring relational context for multi-task dense prediction. In ICCV, pages 15869–15878, 2021.
  8. Many-task federated learning: A new problem setting and a simple baseline. In CVPR, pages 5037–5045, 2023.
  9. Rich Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997.
  10. On bridging generic and personalized federated learning for image classification. In ICLR, 2022.
  11. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801–818, 2018a.
  12. Fedbone: Towards large-scale federated multi-task learning. arXiv preprint arXiv:2306.17465, 2023.
  13. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pages 794–803, 2018b.
  14. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In NeurIPS, 2020.
  15. Exploiting shared representations for personalized federated learning. In ICML, pages 2089–2099, 2021.
  16. Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.
  17. Federated learning for predicting clinical outcomes in patients with COVID-19. Nature Medicine, 27(10):1735–1743, 2021.
  18. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In ECCV, 2021.
  19. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. NeurIPS, 33:3557–3568, 2020.
  20. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In CVPR, pages 3205–3214, 2019.
  21. Learning to branch for multi-task learning. In ICML, pages 3854–3863, 2020.
  22. Spreadgnn: Decentralized multi-task federated learning for graph neural networks on molecular data. In AAAI, pages 6865–6873, 2022.
  23. Planning-oriented autonomous driving. In CVPR, pages 17853–17862, 2023.
  24. Personalized cross-silo federated learning on non-iid data. In AAAI, pages 7865–7873, 2021a.
  25. Evaluating gradient inversion attacks and defenses in federated learning. NeurIPS, 34:7232–7241, 2021b.
  26. Rotograd: Gradient homogenization in multitask learning. In ICLR, 2022.
  27. Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021.
  28. Reparameterizing convolutions for incremental multi-task learning without task interference. In ECCV, pages 689–707, 2020.
  29. SCAFFOLD: stochastic controlled averaging for federated learning. In ICML, pages 5132–5143, 2020.
  30. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pages 7482–7491, 2018.
  31. Federated optimization: Distributed optimization beyond the datacenter. CoRR, abs/1511.03575, 2015.
  32. Federated learning: Strategies for improving communication efficiency. CoRR, abs/1610.05492, 2016.
  33. Model-contrastive federated learning. In CVPR, pages 10713–10722, 2021a.
  34. Federated optimization in heterogeneous networks. In MLSys, 2020.
  35. Ditto: Fair and robust federated learning through personalization. In ICML, pages 6357–6368, 2021b.
  36. FedBN: Federated learning on non-IID features via local batch normalization. In ICLR, 2021c.
  37. Fedphp: Federated personalization with inherited private models. In ECML PKDD, pages 587–602, 2021d.
  38. Conflict-averse gradient descent for multi-task learning. In NeurIPS, pages 18878–18890, 2021a.
  39. On privacy and personalization in cross-silo federated learning. NeurIPS, 35:5925–5940, 2022a.
  40. Towards impartial multi-task learning. In ICLR, 2021b.
  41. End-to-end multi-task learning with attention. In CVPR, pages 1871–1880, 2019.
  42. Vertical federated learning. CoRR, abs/2211.12814, 2022b.
  43. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021c.
  44. Learning multiple tasks with multilinear relationship networks. NeurIPS, 30, 2017.
  45. Decoupled weight decay regularization. In ICLR, 2019.
  46. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, pages 5334–5343, 2017.
  47. Prompt guided transformer for multi-task dense prediction. arXiv preprint arXiv:2307.15362, 2023.
  48. MAMDR: A model agnostic learning framework for multi-domain recommendation. In ICDE, pages 3079–3092, 2023.
  49. Layer-wised model aggregation for personalized federated learning. In CVPR, pages 10092–10101, 2022.
  50. Attentive single-tasking of multiple tasks. In CVPR, pages 1851–1860, 2019.
  51. Federated multi-task learning under a mixture of distributions. In NeurIPS, pages 15434–15447, 2021.
  52. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273–1282, 2017a.
  53. Communication-efficient learning of deep networks from decentralized data. In AISTATS, pages 1273–1282, 2017b.
  54. Multi-task federated learning for personalised deep neural networks in edge computing. TPDS, 33(3):630–641, 2021.
  55. Cross-stitch networks for multi-task learning. In CVPR, pages 3994–4003, 2016.
  56. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891–898, 2014.
  57. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019.
  58. Vision transformers for dense prediction. In ICCV, pages 12179–12188, 2021.
  59. Balanced meta-softmax for long-tailed visual recognition. NeurIPS, 33:4175–4186, 2020.
  60. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
  61. Latent multi-task architecture learning. In AAAI, pages 4822–4829, 2019.
  62. Multi-task learning as multi-objective optimization. In NeurIPS, pages 525–536, 2018.
  63. Indoor segmentation and support inference from rgbd images. In ECCV, pages 746–760, 2012.
  64. Scale-aware task message transferring for multi-task learning. In ICME, pages 1859–1864, 2023.
  65. Federated multi-task learning. In NeurIPS, pages 4424–4434, 2017.
  66. Task switching network for multi-task learning. In ICCV, pages 8291–8300, 2021.
  67. Personalized federated learning with moreau envelopes. NeurIPS, 33:21394–21405, 2020.
  68. Towards Personalized Federated Learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1–17, 2022.
  69. Mti-net: Multi-scale task interaction networks for multi-task learning. In ECCV, pages 527–543, 2020.
  70. Multi-task learning for dense prediction tasks: A survey. IEEE TPAMI, 44(7):3614–3633, 2021.
  71. Task adaptive parameter sharing for multi-task learning. In CVPR, pages 7561–7570, 2022.
  72. Federated learning with matched averaging. In ECCV, 2020a.
  73. Tackling the objective inconsistency problem in heterogeneous federated optimization. In NeurIPS, 2020b.
  74. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pages 568–578, 2021a.
  75. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In ICLR, 2021b.
  76. Perada: Parameter-efficient and generalizable federated learning personalization with guarantees. CoRR, abs/2302.06637, 2023.
  77. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In CVPR, pages 675–684, 2018.
  78. Multi-task learning with multi-query transformer for dense prediction. IEEE TCSVT, 2023.
  79. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10(2):12:1–12:19, 2019.
  80. Multi-objective meta learning. In NeurIPS, pages 21338–21351, 2021.
  81. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, pages 514–530, 2022.
  82. Invpt++: Inverted pyramid multi-task transformer for visual scene understanding. arXiv preprint arXiv:2306.04842, 2023.
  83. Fed2: Feature-aligned federated learning. In ACM SIGKDD, pages 2066–2074, 2021.
  84. Gradient surgery for multi-task learning. In NeurIPS, 2020.
  85. Bayesian nonparametric federated learning of neural networks. In ICML, pages 7252–7261, 2019.
  86. Transfer vision patterns for multi-task pixel learning. In ACM MM, pages 97–106, 2021.
  87. Joint task-recursive learning for semantic segmentation and depth estimation. In ECCV, pages 235–251, 2018.
  88. Pattern-affinitive propagation across depth, surface normal and semantic segmentation. In CVPR, pages 4106–4115, 2019.
  89. Pattern-structure diffusion for multi-task learning. In CVPR, pages 4514–4523, 2020.
  90. Delayed gradient averaging: Tolerate the communication latency for federated learning. NeurIPS, 34:29995–30007, 2021a.
  91. Data-free knowledge distillation for heterogeneous federated learning. In ICML, pages 12878–12889, 2021b.
  92. Mas: Towards resource-efficient federated multiple-task learning. In ICCV, pages 23414–23424, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Yuxiang Lu (26 papers)
  2. Suizhi Huang (10 papers)
  3. Yuwen Yang (21 papers)
  4. Shalayiding Sirejiding (6 papers)
  5. Yue Ding (49 papers)
  6. Hongtao Lu (76 papers)
Citations (2)
View on Semantic Scholar