Ranking a Set of Objects using Heterogeneous Workers: QUITE an Easy Problem (2310.02016v1)
Abstract: We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers, each worker being characterized by a specific degree of reliability, which reflects her ability to rank pairs of objects. More specifically, we assume that objects are endowed with intrinsic qualities and that the probability with which an object is preferred to another depends both on the difference between the qualities of the two competitors and on the reliability of the worker. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects. Performance of QUITE is compared in different scenarios against previously proposed algorithms. Finally, we show how QUITE can be naturally made adaptive.
- M. Richardson, E. Dominowska, and R. Ragno, “Predicting clicks: Estimating the click-through rate for new ads,” in Proceedings of the 16th International Conference on World Wide Web, ser. WWW ’07. New York, NY, USA: ACM, 2007, pp. 521–530.
- R. Herbrich, T. Minka, and T. Graepel, “Trueskill™: A bayesian skill rating system,” in Advances in Neural Information Processing Systems 19, B. Schölkopf, J. C. Platt, and T. Hoffman, Eds. MIT Press, 2007, pp. 569–576.
- N. B. Shah and M. J. Wainwright, “Simple, robust and optimal ranking from pairwise comparisons,” The Journal of Machine Learning Research, vol. 18, no. 1, pp. 7246–7283, 2017.
- A. d’Aspremont, M. Cucuringu, and H. Tyagi, “Ranking and synchronization from pairwise measurements via svd,” arXiv preprint arXiv:1906.02746, 2019.
- M. Falahatgar, Y. Hao, A. Orlitsky, V. Pichapati, and V. Ravindrakumar, “Maxing and ranking with few assumptions,” in Advances in Neural Information Processing Systems, 2017, pp. 7060–7070.
- E. Christoforou, A. Nordio, A. Tarable, and E. Leonardi, “Ranking a set of objects: A graph based least-square approach,” IEEE Transactions on Network Science and Engineering, vol. 8, no. 1, pp. 803–813, 2021.
- T. Jin, P. Xu, Q. Gu, and F. Farnoud, “Rank aggregation via heterogeneous thurstone preference models,” in thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
- R. L. Plackett, “The analysis of permutations,” Applied Statistics, pp. 193–202, 1975.
- R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952.
- L. L. Thurstone, “The method of paired comparisons for social values.” The Journal of Abnormal and Social Psychology, vol. 21, no. 4, p. 384, 1927.
- B. Szörényi, R. Busa-Fekete, A. Paul, and E. Hüllermeier, “Online rank elicitation for plackett-luce: A dueling bandits approach,” in Advances in Neural Information Processing Systems, 2015, pp. 604–612.
- M. Falahatgar, A. Orlitsky, V. Pichapati, and A. T. Suresh, “Maximum selection and ranking under noisy comparisons,” arXiv preprint arXiv:1705.05366, 2017.
- M. Falahatgar, A. Jain, A. Orlitsky, V. Pichapati, and V. Ravindrakumar, “The limits of maxing, ranking, and preference learning,” in International Conference on Machine Learning, 2018, pp. 1426–1435.
- Y. Wu, T. Jin, H. Lou, P. Xu, F. Farnoud, and Q. Gu, “Adaptive sampling for heterogeneous rank aggregation from noisy pairwise comparisons,” in Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, G. Camps-Valls, F. J. R. Ruiz, and I. Valera, Eds., vol. 151. PMLR, 28–30 Mar 2022, pp. 11 014–11 036. [Online]. Available: https://proceedings.mlr.press/v151/wu22f.html
- L. W. Tu, “Bump functions and partitions of unity,” in An Introduction to Manifolds. Springer, 2008, pp. 127–134.