Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

A tutorial on learning from preferences and choices with Gaussian Processes (2403.11782v4)

Published 18 Mar 2024 in cs.LG and stat.ML

Abstract: Preference modelling lies at the intersection of economics, decision theory, machine learning and statistics. By understanding individuals' preferences and how they make choices, we can build products that closely match their expectations, paving the way for more efficient and personalised applications across a wide range of domains. The objective of this tutorial is to present a cohesive and comprehensive framework for preference learning with Gaussian Processes (GPs), demonstrating how to seamlessly incorporate rationality principles (from economics and decision theory) into the learning process. By suitably tailoring the likelihood function, this framework enables the construction of preference learning models that encompass random utility models, limits of discernment, and scenarios with multiple conflicting utilities for both object- and label-preference. This tutorial builds upon established research while simultaneously introducing some novel GP-based models to address specific gaps in the existing literature.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (127)
  1. Accelerated spectral ranking. In International Conference on Machine Learning, pages 70–79. PMLR, 2018.
  2. Charu C. Aggarwal. Recommender Systems. Springer International Publishing, Cham, 2016. ISBN 978-3-319-29657-9. doi: 10.1007/978-3-319-29659-3.
  3. Learning preferences for multiclass problems. Advances in neural information processing systems, 17, 2004.
  4. General theory of best variants choice: Some aspects. IEEE Transactions on Automatic Control, 26(5):1030–1040, 1981.
  5. MT Alodat and Mohammed K Shakhatreh. Gaussian process regression with skewed errors. Journal of Computational and Applied Mathematics, 370:112665, 2020.
  6. Mayer Alvo and LH Philip. Statistical methods for ranking data, volume 1341. Springer, 2014.
  7. Bayesian conjugacy in probit, tobit, multinomial probit and extensions: A review and new results. Journal of the American Statistical Association, 118(542):1451–1469, 2023.
  8. On the unification of families of skew-normal distributions. Scandinavian Journal of Statistics, 33(3):561–574, 2006.
  9. Pareto rationalizability by two single-peaked preferences. Mathematical Social Sciences, 118:1–11, 2022.
  10. Kenneth J Arrow. Social choice and individual values. Yale university press, 1963.
  11. Random utility theory for social choice. Advances in Neural Information Processing Systems, 25, 2012.
  12. Generalized method-of-moments for rank aggregation. Advances in Neural Information Processing Systems, 26, 2013.
  13. Adelchi Azzalini. The skew-normal and related families, volume 3. Cambridge University Press, 2013.
  14. Understanding probabilistic sparse Gaussian process approximations. In Advances in neural information processing systems, pages 1533–1541, 2016.
  15. prefGP: A python package for preference and choice learning with gaussian processes. https://github.com/benavoli/prefGP, 2024.
  16. Skew Gaussian Processes for Classification. Machine Learning, 109:1877–1902, 2020. doi: 10.1007/s10994-020-05906-3. URL https://arxiv.org/abs/2005.12987.
  17. A unified framework for closed-form nonparametric regression, classification, preference and mixed problems with Skew Gaussian Processes. Machine Learning, pages 1–39, 2021a. doi: 10.1007/s10994-021-06039-x. URL https://link.springer.com/article/10.1007/s10994-021-06039-x.
  18. Preferential Bayesian optimisation with Skew Gaussian Processes. In 2021 Genetic and Evolutionary Computation Conference Companion (GECCO ’21 Companion), July 10–14, 2021, Lille, France, New York, NY, USA, 2021b. ACM.
  19. Preferential bayesian optimisation with skew gaussian processes. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1842–1850, 2021c.
  20. Learning choice functions with gaussian processes. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 141–151. PMLR, 2023a. URL https://proceedings.mlr.press/v216/benavoli23a.html.
  21. Bayesian optimization for choice data. In 2023 Genetic and Evolutionary Computation Conference Companion (GECCO ’23 Companion), July 15–19, 2023, Lisbon, 2023b. doi: 10.1145/3583133.3596324.
  22. A discrete choice model for subset selection. In Proceedings of the eleventh ACM international conference on web search and data mining, pages 37–45, 2018.
  23. Michel Bierlaire. Pandasbiogeme: a short introduction. Report TRANSP-OR: Lausanne, Switzerland, 181219, 2018.
  24. Zdravko I Botev. The normal law under linear restrictions: simulation and estimation via minimax tilting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(1):125–148, 2017.
  25. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
  26. Asymmetric, closed-form, finite-parameter models of multinomial choice. Journal of choice modelling, 29:78–112, 2018.
  27. A unifying framework for gaussian process pseudo-point approximations using power expectation propagation. The Journal of Machine Learning Research, 18(1):3649–3720, 2017.
  28. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96, 2005.
  29. Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010.
  30. When do noisy votes reveal the truth? ACM Transactions on Economics and Computation, 4(3):1–30, Jun 2016. ISSN 2167-8375, 2167-8383. doi: 10.1145/2892565.
  31. Efficient bayesian inference for generalized bradley–terry models. Journal of Computational and Graphical Statistics, 21(1):174–196, 2012.
  32. Herman Chernoff. Rational selection of decision functions. Econometrica: journal of the Econometric Society, pages 422–443, 1954.
  33. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  34. Preference Learning with Gaussian Processes. In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, page 137–144, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10.1145/1102351.1102369.
  35. Learning to order things. Advances in neural information processing systems, 10, 1997.
  36. Paired comparison, triple comparison, and ranking experiments as generalized linear models, and their implementation on glim. Psychometrika, 56(3):517–533, 1991.
  37. Yves Croissant. Estimation of random utility models in r: the mlogit package. Journal of Statistical Software, 95:1–41, 2020.
  38. Gerard Debreu. Representation of a preference ordering by a numerical function. Decision processes, 3:159–165, 1954.
  39. Torch-choice: A pytorch package for large-scale choice modelling with python. arXiv preprint arXiv:2304.01906, 2023.
  40. Daniele Durante. Conjugate Bayes for probit regression via unified skew-normal distributions. Biometrika, 106(4):765–779, 08 2019.
  41. Indifference or indecisiveness? choice-theoretic foundations of incomplete preferences. Games and economic behavior, 56(1):61–86, 2006.
  42. Active preference learning with discrete choice data. Advances in neural information processing systems, 20, 2007.
  43. Generalized robust conjoint estimation. Marketing Science, 24(3):415–429, 2005.
  44. Learning subjective functions with large margins. In ICML, pages 287–294, 2000.
  45. Peter C. Fishburn. Nonlinear preference and utility theory. Johns Hopkins series in the mathematical sciences. Johns Hopkins University Press, Baltimore, 1988. ISBN 978-0-8018-3598-8.
  46. Posterior probabilities for a consensus ordering. Psychometrika, 55:53–63, 1990.
  47. A rank-ordered logit model with unobserved heterogeneity in ranking capabilities. Journal of applied econometrics, 27(5):831–846, 2012.
  48. Application and interpretation of nested logit models of intercity mode choice. Transportation research record, (1413), 1993.
  49. Preference learning. Springer, 2010.
  50. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems, 2018-Decem(NeurIPS):7576–7586, 2018. ISSN 10495258. arXiv: 1809.11165.
  51. Integrals over Gaussians under Linear Domain Constraints. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 2764–2774. PMLR, 26–28 Aug 2020.
  52. Variational Gaussian process classifiers. IEEE Transactions on Neural Networks, 11(6):1458–1464, 2000.
  53. Preferential Bayesian Optimization. In Proceedings of the 34th International Conference on Machine Learning, pages 1282–1291. PMLR, 2017. URL https://proceedings.mlr.press/v70/gonzalez17a.html.
  54. Bayesian inference for plackett-luce ranking models. In proceedings of the 26th annual international conference on machine learning, pages 377–384, 2009.
  55. Constraint classification: A new approach to multiclass classification. In International conference on algorithmic learning theory, pages 365–379. Springer, 2002.
  56. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems, 5(4):19:1–19:19, 2015. ISSN 2160-6455. doi: 10.1145/2827872.
  57. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI’13, pages 282–290, Arlington, Virginia, USA, 2013. AUAI Press.
  58. Scalable Variational Gaussian Process Classification. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, page 351–360. PMLR, 2015. URL https://proceedings.mlr.press/v38/hensman15.html.
  59. Learning preference relations for information retrieval. In ICML-98 Workshop: text categorization and machine learning, pages 80–84, 1998.
  60. Scalable gaussian process classification via expectation propagation. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, page 168–176. PMLR, 2016. URL https://proceedings.mlr.press/v51/hernandez-lobato16.html.
  61. Apollo: A flexible, powerful and customisable freeware package for choice model estimation and application. Journal of choice modelling, 32:100170, 2019.
  62. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.
  63. Label ranking by learning pairwise preferences. Artificial Intelligence, 172(16-17):1897–1916, 2008.
  64. David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1):384–406, 2004.
  65. Gibbs sampling approach for generation of truncated multivariate gaussian random variables. In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. 99CH36258), volume 3, pages 1757–1760. IEEE, 1999.
  66. David M Kreps et al. A course in microeconomic theory. Princeton university press, 1990.
  67. Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms. Journal of Choice Modelling, 38:100226, 2021. ISSN 1755-5345.
  68. Antithetic and monte carlo kernel estimators for partial rankings. Statistics and Computing, 29:1127–1147, 2019.
  69. R Duncan Luce. Semiorders and a theory of utility discrimination. Econometrica, Journal of the Econometric Society, pages 178–191, 1956.
  70. R Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. Wiley, New York, 1959.
  71. Learning Inconsistent Preferences with Gaussian Processes. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 2266–2281. PMLR, 2022. URL https://proceedings.mlr.press/v151/lun-chau22a.html.
  72. David JC MacKay. Bayesian methods for backpropagation networks. In Models of neural networks III, pages 211–254. Springer, 1996.
  73. Advanced conjoint analysis using feature selection via support vector machines. European Journal of Operational Research, 241(2):564–574, 2015.
  74. Colin L Mallows. Non-null ranking models. i. Biometrika, 44(1/2):114–130, 1957.
  75. John I Marden. Analyzing and modeling rank data. CRC Press, 1996.
  76. Fast and accurate inference of plackett–luce models. Advances in neural information processing systems, 28, 2015.
  77. Daniel McFadden. The measurement of urban travel demand. Journal of Public Economics, 3(4):303–328, 1974.
  78. Daniel McFadden. Modeling the choice of residential location. In Handbook of Regional and Urban Economics, volume 2, pages 753–803. 1978.
  79. Consensus ranking under the exponential model. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, UAI’07, page 285–294, Arlington, Virginia, USA, 2007. AUAI Press. ISBN 978-0-9749039-3-4. URL http://arxiv.org/abs/1206.5265. arXiv:1206.5265 [cs, stat].
  80. Thomas Peter Minka. A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, 2001.
  81. An experimental measurement of utility. Journal of Political Economy, 59(5):371–404, 1951.
  82. Hervé Moulin. Choice functions over a finite set: a summary. Social Choice and Welfare, 2(2):147–160, 1985.
  83. Iterative ranking from pair-wise comparisons. Advances in neural information processing systems, 25, 2012.
  84. Top-k ranking bayesian optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1010):9135–9143, 2021. ISSN 2374-3468. doi: 10.1609/aaai.v35i10.17103.
  85. Anthony O’Hagan. Curve fitting and optimal design for prediction. Journal of the Royal Statistical Society: Series B (Methodological), 40(1):1–24, 1978.
  86. The variational gaussian approximation revisited. Neural computation, 21(3):786–792, 2009.
  87. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  88. Learning intransitive reciprocal relations with kernel methods. European Journal of Operational Research, 206(3):676–685, 2010.
  89. Exact hamiltonian monte carlo for truncated multivariate gaussians. Journal of Computational and Graphical Statistics, 23(2):518–542, 2014.
  90. Learning Choice Functions via Pareto-Embeddings. In German Conference on Artificial Intelligence (Künstliche Intelligenz), pages 327–333. Springer, 2020.
  91. Learning context-dependent choice functions. International Journal of Approximate Reasoning, 140:116–155, 2022.
  92. Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975.
  93. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research, 6(Dec):1939–1959, 2005.
  94. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, page 53728–53741. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf.
  95. Pairwise choice markov chains. Advances in neural information processing systems, 29, 2016.
  96. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
  97. The diffusion decision model: Theory and data for two-choice decision tasks. Neural computation, 20(4):873–922, 2008. ISSN 0899-7667. doi: 10.1162/neco.2008.12-06-420.
  98. Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI ’09, pages 452–461, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 978-0-9749039-5-8.
  99. Recommender systems: introduction and challenges. Recommender systems handbook, pages 1–34, 2015.
  100. Paul A Samuelson. A note on the pure theory of consumer’s behaviour. Economica, 5(17):61–71, 1938.
  101. Recursive estimation for sparse Gaussian process regression. Automatica, 120:109–127, 2020. ISSN 0005-1098.
  102. Correlated product of experts for sparse gaussian process regression. Machine Learning, 2023. doi: 10.1007/s10994-022-06297-3.
  103. Thomas Schwartz. Chapter 1. Choice Functions, pages 11–23. Columbia University Press, New York Chichester, West Sussex, 1986. ISBN 9780231895309. doi: doi:10.7312/schw93758-004. URL https://doi.org/10.7312/schw93758-004.
  104. Amartya K Sen. Choice functions and revealed preference. The Review of Economic Studies, 38(3):307–317, 1971.
  105. Wayne J Shafer. The nontransitive consumer. Econometrica: Journal of the Econometric Society, pages 913–919, 1974.
  106. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. In Artificial intelligence and statistics, pages 856–865. PMLR, 2015.
  107. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015.
  108. Enhancing discrete choice models with representation learning. Transportation Research Part B: Methodological, 140:236–261, 2020. ISSN 0191-2615.
  109. Preferential Batch Bayesian Optimization. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2021.
  110. B Babington Smith. Discussion of professor ross’s paper. Journal of the Royal Statistical Society B, 12(1):41–59, 1950.
  111. Sparse Gaussian processes using pseudo-inputs. In Advances in neural information processing systems, pages 1257–1264, 2006.
  112. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  113. Towards practical preferential bayesian optimization with skew gaussian processes. In Proceedings of the 40th International Conference on Machine Learning, page 33516–33533. PMLR, July 2023. URL https://proceedings.mlr.press/v202/takeno23b.html.
  114. J Taylor and Y Benjamini. Restrictedmvn: multivariate normal restricted by affine constraints. R package version, 1, 2016.
  115. Louis L. Thurstone. A law of comparative judgment. Psychological Review, 34(4):273–286, Jul 1927. ISSN 1939-1471, 0033-295X. doi: 10.1037/h0070288.
  116. Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In David van Dyk and Max Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR.
  117. Kenneth E Train. Discrete choice methods with simulation. Cambridge university press, 2009.
  118. Context-dependent preferences. Management science, 39(10):1179–1189, 1993.
  119. Choice modelling in the age of machine learning-discussion paper. Journal of Choice Modelling, 42:100340, 2022.
  120. Parameter estimation for generalized thurstone choice models. In International Conference on Machine Learning, pages 498–506. PMLR, 2016.
  121. Christopher Williams. Computing with infinite networks. Advances in neural information processing systems, 9, 1996.
  122. Bayesian classification with gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1342–1351, 1998.
  123. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46, 2017.
  124. A bi-partite generative model framework for analyzing and simulating large scale multiple discrete-continuous travel behaviour data. Transportation Research Part C: Emerging Technologies, 110:247–268, 2020.
  125. Principled preferential bayesian optimization, 2024.
  126. John I Yellott Jr. The relationship between luce’s choice axiom, thurstone’s theory of comparative judgment, and the double exponential distribution. Journal of Mathematical Psychology, 15(2):109–144, 1977.
  127. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning, pages 10–18. PMLR, 2014.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube