2000 character limit reached
Partial Rankings of Optimizers (2402.16565v3)
Published 26 Feb 2024 in cs.LG and stat.ML
Abstract: We introduce a framework for benchmarking optimizers according to multiple criteria over various test functions. Based on a recently introduced union-free generic depth function for partial orders/rankings, it fully exploits the ordinal information and allows for incomparability. Our method describes the distribution of all partial orders/rankings, avoiding the notorious shortcomings of aggregation. This permits to identify test functions that produce central or outlying rankings of optimizers and to assess the quality of benchmarking suites.
- Kenneth J Arrow. A difficulty in the concept of social welfare. Journal of political economy, 58(4):328–346, 1950.
- Performance evaluation of an advanced local search evolutionary algorithm. In IEEE Congress on Evolutionary Computation. IEEE, 2005.
- Michael Bacharach. Group decisions in the face of differences of opinion. Management Science, 22(2):182–191, 1975.
- PD-MORL: Preference-driven multi-objective reinforcement learning algorithm. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
- Depth functions for partial orders with a descriptive analysis of machine learning algorithms. In Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications (ISIPTA), volume 215, pp. 59–71. PMLR, 2023.
- Jean Charles de Borda. Mémoire sur les élections au scrutin. Histoire de l’Académie Royale des Sciences, 12, 1781.
- Marquis de Condorcet. Essai sur l’application de l’analyse a la probabilite des decisions rendues a la pluralite des voix, 1785. Paris.
- Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023.
- A strategy for ranking optimization methods using multiple criteria. In Workshop on Automatic Machine Learning, pp. 11–20. PMLR, 2016.
- Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pp. 613–622, 2001.
- Jürgen Eckhoff. Chapter 2.1 - Helly, Radon, and Carathéodory type theorems. In Handbook of Convex Geometry, pp. 389–448. North-Holland, Amsterdam, 1993.
- Statistical Decision Theory: Kendall’s Library of Statistics 9. Wiley, 2010.
- On a general definition of depth for functional data. Statistical Science, 32(4):630–639, 2017.
- Min-max multi-objective bilevel optimization with applications in robust machine learning. In International Conference on Learning Representations (ICLR), 2023.
- Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation, pp. 1689–1696, 2010.
- COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software, 36:114–144, 2021. URL http://numbbo.github.io/coco/. (accessed: 08.12.2023).
- Anytime performance assessment in blackbox optimization benchmarking. IEEE Transactions on Evolutionary Computation, 26(6):1293–1305, 2022.
- Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- A probabilistic evaluation framework for preference aggregation reflecting group homogeneity. Mathematical Social Sciences, (96):49–62, 2018a.
- Concepts for decision making under severe uncertainty with partial ordinal and partial cardinal preferences. International Journal of Approximate Reasoning, 98:112–131, 2018b.
- Statistical comparisons of classifiers by generalized stochastic dominance. Journal of Machine Learning Research, 24(231):1–37, 2023a.
- Robust statistical comparison of random variables with locally varying scale of measurement. In Robin J. Evans and Ilya Shpitser (eds.), Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 941–952. PMLR, 31 Jul–04 Aug 2023b.
- Preference ranking: An axiomatic approach. mathematical. Mathematical Models in Social Science., pp. 9–23, 1962.
- The number of finite topologies. Proceedings of the American Mathematical Society, 25(2):276, 1970.
- Regina Liu. On a notion of data depth based on random simplices. The Annals of Statistics, 18:405–414, 1990.
- Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the genetic and evolutionary computation conference, pp. 419–427, 2019.
- MLPerf training benchmark. In Proceedings of Machine Learning and Systems, volume 2, pp. 336–349, 2020.
- Benchmarking evolutionary multiobjective optimization algorithms. In IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE, 2010.
- Analyzing the bbob results by means of benchmarking concepts. Evolutionary Computation, 23(1):161–185, 2015.
- Choosing among notions of multivariate depth statistics. Statistical Science, 37:348–368, 2022.
- Deepobs: A deep learning optimizer benchmark suite. In International Conference on Learning Representations (ICLR), 2019. URL https://deepobs.github.io/. (accessed: 08.12.2023).
- Optimizer benchmarking needs to account for hyperparameter tuning. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 9036–9045. PMLR, 2020.
- A dynamic multi-objective optimization method based on classification strategies. Scientific Reports, 13(1):15221, 2023.
- Mihalis Yannakakis. The complexity of the partial order dimension problem. SIAM Journal on Algebraic Discrete Methods, 3(3):351–358, 1982.
- Beyond one-preference-for-all: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023.
- Scaling pareto-efficient decision making via offline multi-objective RL. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
- General notions of statistical depth function. Annals of statistics, pp. 461–482, 2000.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
