Papers
Topics
Authors
Recent
Search
2000 character limit reached

Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning

Published 18 Jul 2023 in cs.LG and cs.DC | (2307.09619v2)

Abstract: We introduce Dataset Grouper, a library to create large-scale group-structured (e.g., federated) datasets, enabling federated learning simulation at the scale of foundation models. This library facilitates the creation of group-structured versions of existing datasets based on user-specified partitions and directly leads to a variety of useful heterogeneous datasets that can be plugged into existing software frameworks. Dataset Grouper offers three key advantages. First, it scales to settings where even a single group's dataset is too large to fit in memory. Second, it provides flexibility, both in choosing the base (non-partitioned) dataset and in defining partitions. Finally, it is framework-agnostic. We empirically demonstrate that Dataset Grouper enables large-scale federated language modeling simulations on datasets that are orders of magnitude larger than in previous work, allowing for federated training of LLMs with hundreds of millions, and even billions, of parameters. Our experimental results show that algorithms like FedAvg operate more as meta-learning methods than as empirical risk minimization methods at this scale, suggesting their utility in downstream personalization and task-specific adaptation. Dataset Grouper is available at https://github.com/google-research/dataset_grouper.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (92)
  1. Advances and Open Problems in Federated Learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021a.
  2. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In ICML, pages 1126–1135. PMLR, 2017.
  3. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography Conference, volume 3876 of Lecture Notes in Computer Science, pages 265–284, 2006.
  4. Deep Learning with Differential Privacy. In CCS, pages 308–318, 2016.
  5. How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy. J. Artif. Intell. Res., 77:1113–1201, 2023.
  6. The Power of Scale for Parameter-Efficient Prompt Tuning. In EMNLP, pages 3045–3059, 2021.
  7. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022a.
  8. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. arXiv Preprint, 2021.
  9. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
  10. What Does it Mean for a Language Model to Preserve Privacy? In FAccT, pages 2280–2292, 2022.
  11. TensorFlow Federated: Machine Learning on Decentralized Data. https://www.tensorflow.org/federated.
  12. LEAF: A benchmark for federated settings. arXiv Preprint, 2018.
  13. FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks. In NAACL, 2022.
  14. FedML: A Research Library and Benchmark for Federated Machine Learning. NeurIPS Federated Learning Workshop, 2020.
  15. FedScale: Benchmarking Model and System Performance of Federated Learning at Scale. In ICML, volume 162, pages 11814–11827, 2022.
  16. TensorFlow Datasets, a collection of ready-to-use datasets. https://www.tensorflow.org/datasets.
  17. Datasets: A community library for natural language processing. In EMNLP, pages 175–184, 2021.
  18. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
  19. Advances and Open Problems in Federated Learning. Found. Trends Mach. Learn., 14(1-2):1–210, 2021b.
  20. Flower: A Friendly Federated Learning Research Framework. arXiv Preprint, 2020.
  21. FLBench: A Benchmark Suite for Federated Learning. In Intelligent Computing and Block Chain, pages 166–176. Springer, 2021.
  22. The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems. ACM Trans. Intell. Syst. Technol., 13(4), 2022b. ISSN 2157-6904. doi: 10.1145/3510540.
  23. FedJAX: Federated learning simulation with JAX. arXiv Preprint, 2021.
  24. Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization. In NeurIPS, pages 11080–11092, 2019.
  25. Communication Trade-offs for Local-SGD with Large Step Size. In NeurIPS, pages 13579–13590, 2019.
  26. On the Convergence of FedAvg on Non-IID Data. In ICLR, 2020a.
  27. Tighter Theory for Local SGD on Identical and Heterogeneous Data. In International Conference on Artificial Intelligence and Statistics, 2020.
  28. SCAFFOLD: stochastic controlled averaging for federated learning. In ICML, volume 119, pages 5132–5143, 2020.
  29. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. In NeurIPS, 2020.
  30. Adaptive Federated Optimization. In ICLR, 2021.
  31. On large-cohort training for federated learning. NeurIPS, 34:20461–20475, 2021.
  32. Learning Differentially Private Recurrent Language Models. In ICLR, 2018.
  33. Federated Learning With Differential Privacy: Algorithms and Performance Analysis. IEEE Transactions on Information Forensics and Security, 15:3454–3469, 2020.
  34. The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation. In ICML, volume 139, pages 5201–5212, 2021c.
  35. Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams . In NeurIPS, volume 35, pages 5910–5924, 2022.
  36. Robust Aggregation for Federated Learning. IEEE Transactions on Signal Processing, 2022a.
  37. Sageflow: Robust federated learning against both stragglers and adversaries. NeurIPS, 2021.
  38. Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing. In ICLR, 2022.
  39. Robustness and Personalization in Federated Learning: A Unified Approach via Regularization. In IEEE EDGE, 2022.
  40. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In IEEE Symposium on Security and Privacy. IEEE, 2022.
  41. Fair Resource Allocation in Federated Learning. In ICLR, 2020b.
  42. Device Heterogeneity in Federated Learning: A Superquantile Approach. arXiv preprint, 2020.
  43. Distributionally Robust Federated Averaging. In NeurIPS, 2020.
  44. Robust Federated Learning: The Case of Affine Distribution Shifts. In NeurIPS, 2020.
  45. Federated learning with superquantile aggregation for heterogeneous data. Machine Learning, pages 1–68, 2023.
  46. Personalized Federated Learning with Moreau Envelopes. In NeurIPS, volume 33, pages 21394–21405, 2020.
  47. Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications. arXiv Preprint, 2021.
  48. Exploiting Shared Representations for Personalized Federated Learning. In ICML, volume 139, pages 2089–2099, 2021.
  49. Differentially Private Model Personalization. In NeurIPS, pages 29723–29735, 2021.
  50. Federated Learning with Partial Model Personalization. In ICML, volume 162, pages 17716–17758, 2022b.
  51. Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning. In ICML, volume 162, pages 1945–1962, 2022.
  52. FLAIR: Federated Learning Annotated Image Repository. In NeurIPS, 2022.
  53. FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings. In NeurIPS, 2022.
  54. Motley: Benchmarking Heterogeneity and Personalization in Federated Learning. arXiv Preprint, 2022.
  55. pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning. In NeurIPS, 2022.
  56. Scaling Language Model Size in Cross-Device Federated Learning. arXiv Preprint, 2022.
  57. WILDS: A Benchmark of in-the-Wild Distribution Shifts. In Marina Meila and Tong Zhang, editors, ICML, volume 139, pages 5637–5664. PMLR, 2021.
  58. Extending the WILDS Benchmark for Unsupervised Adaptation. In ICLR, 2022.
  59. How do I access data from only one group? Github Issue #73 for p-lambda/wilds. https://github.com/p-lambda/wilds/issues/73. Accessed on June 1, 2023.
  60. Ellie Pavlick Stefanie Tellex Aaron Gokaslan, Vanya Cohen. OpenWebText Corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019.
  61. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv Preprint, 2020.
  62. Scaling Laws for Neural Language Models. arXiv Preprint, 2020.
  63. Big Bird: Transformers for Longer Sequences. NeurIPS, 33:17283–17297, 2020.
  64. Longformer: The Long-Document Transformer. arXiv Preprint, 2020.
  65. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In ICLR, 2022.
  66. PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models–Federated Learning in Age of Foundation Model. arXiv Preprint, 2022.
  67. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022.
  68. Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answering. arXiv Preprint, 2023.
  69. Towards Building the Federated GPT: Federated Instruction Tuning. arXiv Preprint, 2023.
  70. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015.
  71. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. arXiv Preprint, 2019.
  72. Array programming with NumPy. Nature, 585:357–362, 2020. doi: 10.1038/s41586-020-2649-2.
  73. PyTorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019.
  74. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
  75. Steven T Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112–1130, 2014.
  76. George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books, 2016.
  77. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In ICCV, December 2015.
  78. Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus. arXiv Preprint, 2021.
  79. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv Preprint, 2016.
  80. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, pages 4171–4186, 2019.
  81. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 20–22 Apr 2017. URL https://proceedings.mlr.press/v54/mcmahan17a.html.
  82. Iterated vector fields and conservatism, with applications to federated learning. In COLT, pages 130–147, 2022.
  83. Improving Federated Learning Personalization via Model Agnostic Meta Learning. arXiv Preprint, 2019.
  84. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach. In NeurIPS, 2020.
  85. Convergence and accuracy trade-offs in federated learning and meta-learning. In AISTATS, pages 2575–2583, 2021.
  86. Fast Federated Learning by Balancing Communication Trade-Offs. IEEE Transactions on Communications, 69(8):5168–5182, 2021.
  87. Fine-tuning is Fine in Federated Learning. arXiv Preprint, 3, 2021.
  88. FedAvg with Fine Tuning: Local Updates Lead to Representation Learning. In NeurIPS, 2022.
  89. On First-Order Meta-Learning Algorithms. arXiv Preprint, 2018.
  90. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research. In NeurIPS Datasets and Benchmarks, 2021.
  91. tf.data: A machine learning data processing framework. arXiv Preprint, 2021.
  92. Letter value plots: Boxplots for large data. Journal of Computational and Graphical Statistics, 26(3):469–477, 2017.
Citations (19)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up