Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Decomposable Submodular Maximization in Federated Setting (2402.00138v2)

Published 31 Jan 2024 in cs.DS, cs.AI, cs.DC, and cs.LG

Abstract: Submodular functions, as well as the sub-class of decomposable submodular functions, and their optimization appear in a wide range of applications in machine learning, recommendation systems, and welfare maximization. However, optimization of decomposable submodular functions with millions of component functions is computationally prohibitive. Furthermore, the component functions may be private (they might represent user preference function, for example) and cannot be widely shared. To address these issues, we propose a {\em federated optimization} setting for decomposable submodular optimization. In this setting, clients have their own preference functions, and a weighted sum of these preferences needs to be maximized. We implement the popular {\em continuous greedy} algorithm in this setting where clients take parallel small local steps towards the local solution and then the local changes are aggregated at a central server. To address the large number of clients, the aggregation is performed only on a subsampled set. Further, the aggregation is performed only intermittently between stretches of parallel local steps, which reduces communication cost significantly. We show that our federated algorithm is guaranteed to provide a good approximate solution, even in the presence of above cost-cutting measures. Finally, we show how the federated setting can be incorporated in solving fundamental discrete submodular optimization problems such as Maximum Coverage and Facility Location.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. cpsgd: Communication-efficient and differentially-private distributed SGD. In NeurIPS 2018, pages 7575–7586, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/21ce689121e39821d07d04faab328370-Abstract.html.
  2. The skellam mechanism for differentially private federated learning. In NeurIPS 2021, pages 5052–5064, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/285baacbdf8fda1de94b19282acd23e2-Abstract.html.
  3. The power of randomization: Distributed submodular maximization on massive datasets. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1236–1244. JMLR.org, 2015. URL http://proceedings.mlr.press/v37/barbosa15.html.
  4. Secure single-server aggregation with (poly)logarithmic overhead. In CCS, pages 1253–1269. ACM, 2020. doi: 10.1145/3372297.3417885. URL https://doi.org/10.1145/3372297.3417885.
  5. Practical secure aggregation for federated learning on user-held data. CoRR, abs/1611.04482, 2016. URL http://arxiv.org/abs/1611.04482.
  6. Maximizing a monotone submodular function subject to a matroid constraint. SIAM J. Comput., 40(6):1740–1766, 2011. doi: 10.1137/080733991. URL https://doi.org/10.1137/080733991.
  7. Differentially private decomposable submodular maximization. In AAAI, pages 6984–6992, 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/16860.
  8. The fundamental price of secure aggregation in differentially private federated learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 3056–3089. PMLR, 2022a. URL https://proceedings.mlr.press/v162/chen22c.html.
  9. Optimal client sampling for federated learning. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856. URL https://openreview.net/forum?id=8GvRCWKHIL.
  10. Federated frank-wolfe algorithm. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS), 2022.
  11. An improved approximation algorithm for combinatorial auctions with submodular bidders. In SODA, pages 1064–1073, 2006. URL http://dl.acm.org/citation.cfm?id=1109557.1109675.
  12. Non-metric affinity propagation for unsupervised image categorization. In ICCV, pages 1–8, 2007. doi: 10.1109/ICCV.2007.4408853. URL https://doi.org/10.1109/ICCV.2007.4408853.
  13. Jack Edmonds. Submodular functions, matroids, and certain polyhedra. In Combinatorial Optimization - Eureka, You Shrink!, pages 11–26, 2001.
  14. Uriel Feige. A threshold of ln n for approximating set cover. J. ACM, 45(4):634–652, 1998. doi: 10.1145/285055.285059. URL https://doi.org/10.1145/285055.285059.
  15. Uriel Feige. On maximizing welfare when utility functions are subadditive. In STOC, pages 41–50, 2006. doi: 10.1145/1132516.1132523. URL https://doi.org/10.1145/1132516.1132523.
  16. Approximation algorithms for allocation problems: Improving the factor of 1 - 1/e. In FOCS, pages 667–676, 2006. doi: 10.1109/FOCS.2006.14. URL https://doi.org/10.1109/FOCS.2006.14.
  17. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95–110, 1956.
  18. Budgeted nonparametric learning from data streams. In ICML, pages 391–398, 2010. URL https://icml.cc/Conferences/2010/papers/433.pdf.
  19. Local SGD: unified theory and new efficient methods. In AISTATS, volume 130 of Proceedings of Machine Learning Research, pages 3556–3564. PMLR, 2021. URL http://proceedings.mlr.press/v130/gorbunov21a.html.
  20. Differentially private combinatorial optimization. In Moses Charikar, editor, SODA, pages 1106–1125. SIAM, 2010. doi: 10.1137/1.9781611973075.90. URL https://doi.org/10.1137/1.9781611973075.90.
  21. Gradient methods for submodular maximization. In NeurIPS, pages 5841–5851, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/24b43fb034a10d78bec71274033b4096-Abstract.html.
  22. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 5201–5212. PMLR, 2021. URL http://proceedings.mlr.press/v139/kairouz21a.html.
  23. SCAFFOLD: stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5132–5143. PMLR, 2020. URL http://proceedings.mlr.press/v119/karimireddy20a.html.
  24. Maximizing the spread of influence through a social network. In KDD, pages 137–146, 2003. URL https://doi.org/10.1145/956750.956769.
  25. Cut sparsification and succinct representation of submodular hypergraphs. CoRR, abs/2307.09110, 2023. doi: 10.48550/ARXIV.2307.09110. URL https://doi.org/10.48550/arXiv.2307.09110.
  26. Inapproximability results for combinatorial auctions with submodular utility functions. Algorithmica, 52(1):3–18, 2008. doi: 10.1007/s00453-007-9105-7. URL https://doi.org/10.1007/s00453-007-9105-7.
  27. Federated optimization: distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
  28. Near-optimal nonmyopic value of information in graphical models. In UAI, pages 324–331, 2005.
  29. Fast greedy algorithms in mapreduce and streaming. ACM Trans. Parallel Comput., 2(3):14:1–14:22, 2015. doi: 10.1145/2809814. URL https://doi.org/10.1145/2809814.
  30. Combinatorial auctions with decreasing marginal utilities. Games Econ. Behav., 55(2):270–296, 2006. doi: 10.1016/j.geb.2005.02.006. URL https://doi.org/10.1016/j.geb.2005.02.006.
  31. On the convergence of fedavg on non-iid data. In ICLR. OpenReview.net, 2020. URL https://openreview.net/forum?id=HJxNAnVtDS.
  32. A class of submodular functions for document summarization. In HLT, pages 510–520, 2011.
  33. Communication-efficient learning of deep networks from decentralized data. In AISTATS, volume 54 of Proceedings of Machine Learning Research, pages 1273–1282. PMLR, 2017. URL http://proceedings.mlr.press/v54/mcmahan17a.html.
  34. Tight information-theoretic lower bounds for welfare maximization in combinatorial auctions. In EC, pages 70–77. ACM, 2008. doi: 10.1145/1386790.1386805. URL https://doi.org/10.1145/1386790.1386805.
  35. Baharan Mirzasoleiman. Big Data Summarization Using Submodular Functions. PhD thesis, ETH Zurich, Zürich, Switzerland, 2017.
  36. Fast constrained submodular maximization: Personalized data summarization. In ICML, volume 48, pages 1358–1367, 2016a. URL http://proceedings.mlr.press/v48/mirzasoleiman16.html.
  37. Distributed submodular maximization. J. Mach. Learn. Res., 17:238:1–238:44, 2016b. URL http://jmlr.org/papers/v17/mirzasoleiman16a.html.
  38. Differentially private submodular maximization: Data summarization in disguise. In ICML, pages 2478–2487, 2017. URL http://proceedings.mlr.press/v70/mitrovic17a.html.
  39. Conditional gradient method for stochastic submodular maximization: Closing the gap. In AISTATS, volume 84 of Proceedings of Machine Learning Research, pages 1886–1895. PMLR, 2018a. URL http://proceedings.mlr.press/v84/mokhtari18a.html.
  40. Decentralized submodular maximization: Bridging discrete and continuous settings. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 3613–3622. PMLR, 2018b. URL http://proceedings.mlr.press/v80/mokhtari18a.html.
  41. An analysis of approximations for maximizing submodular set functions - I. Math. Program., 14(1):265–294, 1978. doi: 10.1007/BF01588971. URL https://doi.org/10.1007/BF01588971.
  42. On the hardness of being truthful. In FOCS, pages 250–259. IEEE Computer Society, 2008. doi: 10.1109/FOCS.2008.54. URL https://doi.org/10.1109/FOCS.2008.54.
  43. Fedsplit: an algorithmic framework for fast federated optimization. In NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/4ebd440d99504722d80de606ea8507da-Abstract.html.
  44. Fast and private submodular and k-submodular functions maximization with matroid constraints. In ICML, pages 7887–7897, 2020. URL http://proceedings.mlr.press/v119/rafiey20a.html.
  45. Sparsification of decomposable submodular functions. In AAAI, pages 10336–10344. AAAI Press, 2022. URL https://ojs.aaai.org/index.php/AAAI/article/view/21275.
  46. Sebastian U. Stich. Local SGD converges fast and communicates little. In ICLR. OpenReview.net, 2019. URL https://openreview.net/forum?id=S1g2JnRcFX.
  47. Learning mixtures of submodular functions for image collection summarization. In NeurIPS, pages 1413–1421, 2014. URL https://proceedings.neurips.cc/paper/2014/hash/a8e864d04c95572d1aece099af852d0a-Abstract.html.
  48. Jan Vondrák. Optimal approximation for the submodular welfare problem in the value oracle model. In STOC, pages 67–74, 2008. doi: 10.1145/1374376.1374389. URL https://doi.org/10.1145/1374376.1374389.
  49. Cooperative SGD: A unified framework for the design and analysis of local-update SGD algorithms. J. Mach. Learn. Res., 22:213:1–213:50, 2021. URL http://jmlr.org/papers/v22/20-147.html.
  50. Adaptive federated learning in resource constrained edge computing systems. IEEE J. Sel. Areas Commun., 37(6):1205–1221, 2019. doi: 10.1109/JSAC.2019.2904348. URL https://doi.org/10.1109/JSAC.2019.2904348.
  51. Federated submodular maximization with differential privacy. IEEE Internet of Things Journal, 2023.
  52. Is local SGD better than minibatch sgd? In ICML, volume 119 of Proceedings of Machine Learning Research, pages 10334–10343. PMLR, 2020. URL http://proceedings.mlr.press/v119/woodworth20a.html.
  53. Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works for deep learning. In AAAI, pages 5693–5700. AAAI Press, 2019. doi: 10.1609/aaai.v33i01.33015693. URL https://doi.org/10.1609/aaai.v33i01.33015693.
  54. Decentralized randomized block-coordinate frank-wolfe algorithms for submodular maximization over networks. IEEE Trans. Syst. Man Cybern. Syst., 52(8):5081–5091, 2022. doi: 10.1109/TSMC.2021.3112691. URL https://doi.org/10.1109/TSMC.2021.3112691.
  55. Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data. CoRR, abs/2005.11418, 2020. URL https://arxiv.org/abs/2005.11418.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now