Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Many-Objective Multi-Solution Transport (2403.04099v1)

Published 6 Mar 2024 in cs.LG

Abstract: Optimizing the performance of many objectives (instantiated by tasks or clients) jointly with a few Pareto stationary solutions (models) is critical in machine learning. However, previous multi-objective optimization methods often focus on a few number of objectives and cannot scale to many objectives that outnumber the solutions, leading to either subpar performance or ignored objectives. We introduce Many-objective multi-solution Transport (MosT), a framework that finds multiple diverse solutions in the Pareto front of many objectives. Our insight is to seek multiple solutions, each performing as a domain expert and focusing on a specific subset of objectives while collectively covering all of them. MosT formulates the problem as a bi-level optimization of weighted objectives for each solution, where the weights are defined by an optimal transport between the objectives and solutions. Our algorithm ensures convergence to Pareto stationary solutions for complementary subsets of objectives. On a range of applications in federated learning, multi-task learning, and mixture-of-prompt learning for LLMs, MosT distinctly outperforms strong baselines, delivering high-quality, diverse solutions that profile the entire Pareto frontier, thus ensuring balanced trade-offs across many objectives.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Uci machine learning repository, 2007.
  2. Convex optimization. Cambridge university press, 2004.
  3. Brualdi, R. A. Combinatorial matrix classes, volume 13. Cambridge University Press, 2006.
  4. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018.
  5. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.
  6. Coello, C. C. Evolutionary multi-objective optimization: a historical view of the field. IEEE computational intelligence magazine, 1(1):28–36, 2006.
  7. Emnist: an extension of mnist to handwritten letters. arXiv preprint arXiv:1702.05373, 2017.
  8. Court, U. S. Griggs v. duke power co, 1971.
  9. Désidéri, J.-A. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313–318, 2012.
  10. A tutorial on multiobjective optimization: fundamentals and evolutionary methods. Natural computing, 17:585–609, 2018.
  11. Complexity of gradient descent for multiobjective optimization. Optimization Methods and Software, 34(5):949–959, 2019.
  12. Fujishige, S. Lexicographically optimal base of a polymatroid with respect to a weight vector. Mathematics of Operations Research, 5(2):186–196, 1980.
  13. An efficient framework for clustered federated learning. In Advances in Neural Information Processing Systems, 2020.
  14. Caltech-256 object category dataset. 2007.
  15. Exact algorithms for multiobjective linear optimization problems with integer variables: A state of the art survey. Journal of Multi-Criteria Decision Analysis, 29(5-6):341–363, 2022.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  17. Federated learning meets multi-objective optimization. IEEE Transactions on Network Science and Engineering, 9(4):2039–2051, 2022.
  18. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.
  19. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp.  252–262, 2018.
  20. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pp.  201–216, 2018.
  21. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020.
  22. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
  23. Pareto multi-task learning. Advances in neural information processing systems, 32, 2019.
  24. Sparsity-constrained optimal transport. arXiv preprint arXiv:2209.15466, 2022a.
  25. Profiling pareto front with multi-objective stein variational gradient descent. Advances in Neural Information Processing Systems, 34:14721–14733, 2021.
  26. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11976–11986, 2022b.
  27. Multi-task learning with user preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning, pp.  6597–6607. PMLR, 2020.
  28. Communication-efficient learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics, 2017.
  29. Miettinen, K. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
  30. A multi-objective/multi-task learning framework induced by pareto stationarity. In International Conference on Machine Learning, pp.  15895–15907. PMLR, 2022.
  31. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  1406–1415, 2019.
  32. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808.09121, 2018.
  33. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599, 2021.
  34. Optimization on pareto sets: On a theory of multi-objective optimization. arXiv preprint arXiv:2308.02145, 2023.
  35. Adapting visual category models to new domains. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp.  213–226. Springer, 2010.
  36. A hybrid framework for evolutionary multi-objective optimization. IEEE Transactions on Evolutionary Computation, 17(4):495–511, 2012.
  37. Federated multi-task learning. In Advances in Neural Information Processing Systems, 2017.
  38. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
  39. Motley: Benchmarking heterogeneity and personalization in federated learning. arXiv preprint arXiv:2206.09262, 2022.
  40. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in artificial intelligence, pp.  433–453. PMLR, 2020.
  41. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pp.  962–970. PMLR, 2017.
  42. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257–271, 1999.
  43. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation, 8(2):173–195, 2000.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets