Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exact and general decoupled solutions of the LMC Multitask Gaussian Process model (2310.12032v2)

Published 18 Oct 2023 in cs.LG and stat.ML

Abstract: The Linear Model of Co-regionalization (LMC) is a very general model of multitask gaussian process for regression or classification. While its expressivity and conceptual simplicity are appealing, naive implementations have cubic complexity in the number of datapoints and number of tasks, making approximations mandatory for most applications. However, recent work has shown that under some conditions the latent processes of the model can be decoupled, leading to a complexity that is only linear in the number of said processes. We here extend these results, showing from the most general assumptions that the only condition necessary to an efficient exact computation of the LMC is a mild hypothesis on the noise model. We introduce a full parametrization of the resulting \emph{projected LMC} model, and an expression of the marginal likelihood enabling efficient optimization. We perform a parametric study on synthetic data to show the excellent performance of our approach, compared to an unrestricted exact LMC and approximations of the latter. Overall, the projected LMC appears as a credible and simpler alternative to state-of-the art models, which greatly facilitates some computations such as leave-one-out cross-validation and fantasization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Kernels for vector-valued functions: A review. Foundations and Trends in Machine Learning, 4(3):195–266, 2012. ISSN 1935-8237. doi: 10.1561/2200000036. URL http://dx.doi.org/10.1561/2200000036.
  2. Multi-output separable gaussian process: Towards an efficient, fully bayesian paradigm for uncertainty quantification. Journal of Computational Physics, 241:212–239, 2013. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2013.01.011. URL https://www.sciencedirect.com/science/article/pii/S0021999113000417.
  3. Christopher M. Bishop. Pattern recognition and machine learning. Information science and statistics. Springer, 2006. ISBN 978-0-387-31073-2.
  4. Multi-task gaussian process prediction. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/file/66368270ffd51418ec58bd793f2d9b1b-Paper.pdf.
  5. Generic inference in latent gaussian process models. J. Mach. Learn. Res., 20:117–1, 2019.
  6. Convex optimization. Cambridge university press, 2004.
  7. Scalable exact inference in multi-output gaussian processes. In Hal Daume and Aarti Singh, editors, ICML’20: Proceedings of the 37th International Conference on Machine Learning, 2020.
  8. Machine learning approaches for improving condition-based maintenance of naval propulsion plants. Journal of Engineering for the Maritime Environment, –(–):–, 2014.
  9. Grouped gaussian processes for solar power prediction. Machine Learning, 108(8-9):1287–1306, 2019.
  10. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
  11. Generalized probabilistic principal component analysis of correlated data. The Journal of Machine Learning Research, 21(1):428–468, 2020.
  12. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 351–360, San Diego, California, USA, 09–12 May 2015. PMLR. URL https://proceedings.mlr.press/v38/hensman15.html.
  13. Collaborative gaussian processes for preference learning. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf.
  14. Efficient nonmyopic bayesian optimization via one-shot multi-step trees. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18039–18049. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/d1d5923fc822531bbfd9d87d4760914b-Paper.pdf.
  15. Mario Lezcano Casado. Trivializations for gradient-based optimization on manifolds. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/1b33d16fc562464579b7199ca3114982-Paper.pdf.
  16. Remarks on multi-output gaussian process regression. Knowledge-Based Systems, 144:102–121, 2018. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2017.12.034. URL https://www.sciencedirect.com/science/article/pii/S0950705117306123.
  17. When gaussian process meets big data: A review of scalable gps. IEEE Transactions on Neural Networks and Learning Systems, 31(11):4405–4423, 2020. doi: 10.1109/TNNLS.2019.2957109.
  18. Scalable multi-task gaussian processes with neural embedding of coregionalization. Knowledge-Based Systems, 247:108775, 2022. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2022.108775. URL https://www.sciencedirect.com/science/article/pii/S0950705122003641.
  19. Learning multitask gaussian process over heterogeneous input domains. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 53(10):6232–6244, 2023. doi: 10.1109/TSMC.2023.3281973.
  20. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  21. Bayesian optimization with high-dimensional outputs. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19274–19287. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/a0d3973ad100ad83a64c304bb58677dd-Paper.pdf.
  22. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf.
  23. The matrix cookbook, nov 2012. URL http://www2.compute.dtu.dk/pubdb/pubs/3274-full.html. Version 20121115.
  24. It is all in the noise: Efficient multi-task gaussian process inference with structured residuals. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/59c33016884a62116be975a9bb8257e3-Paper.pdf.
  25. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006. ISBN 978-0-262-18253-9. OCLC: ocm61285753.
  26. Esteban Alejandro Szames. Few group cross section modeling by machine learning for nuclear reactor. PhD thesis, 2020. URL http://www.theses.fr/2020UPASS134.
  27. Multi-output gaussian process-based data augmentation for multi-building and multi-floor indoor localization. 2022 IEEE International Conference on Communications Workshops (ICC Workshops), pages 361–366, 2022. doi: 10.1109/ICCWorkshops53468.2022.9814616.
  28. Semiparametric latent factor models. In Robert G. Cowell and Zoubin Ghahramani, editors, Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, volume R5 of Proceedings of Machine Learning Research, pages 333–340. PMLR, 06–08 Jan 2005. URL https://proceedings.mlr.press/r5/teh05a.html. Reissued by PMLR on 30 March 2021.
  29. Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In David van Dyk and Max Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR. URL https://proceedings.mlr.press/v5/titsias09a.html.
  30. Locally weighted projection regression: An o (n) algorithm for incremental real time learning in high dimensional space. In Proceedings of the seventeenth international conference on machine learning (ICML 2000), volume 1, pages 288–293. Morgan Kaufmann, 2000.
  31. Modeling and analysis of permanent magnet spherical motors by a multitask gaussian process method and finite element method for output torque. IEEE Transactions on Industrial Electronics, 68(9):8540–8549, 2021. doi: 10.1109/TIE.2020.3018078.
  32. Gaussian process kernels for pattern discovery and extrapolation. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1067–1075, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR. URL https://proceedings.mlr.press/v28/wilson13.html.
  33. Inverse method of centrifugal pump blade based on gaussian process regression. Mathematical Problems in Engineering, 2020:1–10, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets