Papers
Topics
Authors
Recent
Search
2000 character limit reached

Duality in Multi-View Restricted Kernel Machines

Published 26 May 2023 in cs.LG | (2305.17251v2)

Abstract: We propose a unifying setting that combines existing restricted kernel machine methods into a single primal-dual multi-view framework for kernel principal component analysis in both supervised and unsupervised settings. We derive the primal and dual representations of the framework and relate different training and inference algorithms from a theoretical perspective. We show how to achieve full equivalence in primal and dual formulations by rescaling primal variables. Finally, we experimentally validate the equivalence and provide insight into the relationships between different methods on a number of time series data sets by recursively forecasting unseen test data and visualizing the learned features.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Semi-Supervised Classification with Graph Convolutional Kernel Machines, 2023. arXiv:2301.13764 [cs].
  2. Convex formulation for Kernel PCA and its use in semisupervised learning. IEEE Transactions on Neural Networks and Learning Systems, 29(8):3863–3869, 2018. doi: 10.1109/TNNLS.2017.2709838.
  3. Convex Optimization. Cambridge University Press, New York, NY, USA, 2004. ISBN 0521833787.
  4. Projection-free kernel principal component analysis for denoising. Neurocomputing, 357:163–176, 2019. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2019.04.042.
  5. A Survey on Deep Learning for Multimodal Data Fusion. Neural Computation, 32(5):829–864, 05 2020. ISSN 0899-7667. doi: 10.1162/neco˙a˙01273. URL https://doi.org/10.1162/neco_a_01273.
  6. Multiple kernel learning algorithms. The Journal of Machine Learning Research, 12:2211–2268, 2011.
  7. Multiple kernel learning based multi-view spectral clustering. In 2014 22nd International Conference on Pattern Recognition, pp.  3774–3779, 2014. doi: 10.1109/ICPR.2014.648.
  8. Hinton, G. E. What kind of a graphical model is the brain? In Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI’05, pp.  1765–1775, San Francisco, CA, USA, 2005. Morgan Kaufmann Publishers Inc.
  9. Preimage problem in kernel-based machine learning. IEEE Signal Processing Magazine, 28(2):77–88, March 2011. ISSN 1053-5888.
  10. Tensor-based restricted kernel machines for multi-view classification. Information Fusion, 68:54–66, 2021. ISSN 1566-2535. doi: https://doi.org/10.1016/j.inffus.2020.10.022.
  11. The pre-image problem in kernel methods. IEEE Transactions on Neural Networks, 15:1517–1525, 2003.
  12. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
  13. Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform. In ICLR, 2019.
  14. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468), pp. 41–48, 1999a. doi: 10.1109/NNSP.1999.788121.
  15. Kernel pca and de-noising in feature spaces. In Advances in Neural Information Processing Systems, pp. 536–542. MIT Press, 1999b.
  16. Generative restricted kernel machines: A framework for multi-view generation and disentangled feature learning. Neural Networks, 135:177–191, 2021.
  17. Recurrent Restricted Kernel Machines for Time-series Forecasting. In ESANN 2022 proceedings, pp.  399–404, Bruges (Belgium) and online event, 2022a. ISBN 978-2-87587-084-1. doi: 10.14428/esann/2022.ES2022-120.
  18. Disentangled representation learning and generation with manifold optimization. Neural Computation, 34(10):2009–2036, 09 2022b. ISSN 0899-7667. doi: 10.1162/neco˙a˙01528.
  19. Multi-view Kernel PCA for Time series Forecasting, January 2023. arXiv:2301.09811 [cs].
  20. Random features for large-scale kernel machines. In Platt, J., Koller, D., Singer, Y., and Roweis, S. (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
  21. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X.
  22. Rockafellar, R. T. Conjugate Duality and Optimization. Society for Industrial and Applied Mathematics, 1974. doi: 10.1137/1.9781611970524.
  23. Deep Boltzmann Machines. In van Dyk, D. and Welling, M. (eds.), Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp.  448–455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR.
  24. Restricted Boltzmann Machines for collaborative filtering. In ICML ’07, pp.  791–798, Corvalis, Oregon, 2007. ACM Press.
  25. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, USA, 2001. ISBN 0262194759.
  26. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Computation, 10(5):1299–1319, 1998. doi: 10.1162/089976698300017467.
  27. Generative Kernel PCA. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning , pp.  129–134, 2018.
  28. A support vector machine formulation to pca analysis and its kernel version. IEEE Transactions on Neural Networks, 14(2):447–450, 2003. doi: 10.1109/TNN.2003.809414.
  29. Suykens, J. A. K. Deep restricted kernel machines using conjugate feature duality. Neural Computation, 29(8):2123–2163, August 2017. ISSN 0899-7667, 1530-888X.
  30. Least Squares Support Vector Machines. World Scientific, River Edge, NJ, January 2002. ISBN 978-981-238-151-4.
  31. Unsupervised energy-based out-of-distribution detection using stiefel-restricted kernel machine. In 2021 International Joint Conference on Neural Networks (IJCNN), pp.  1–8, 2021a. doi: 10.1109/IJCNN52387.2021.9533706.
  32. Unsupervised learning of disentangled representations in deep restricted kernel machines with orthogonality constraints. Neural Networks, 142:661–679, 2021b. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2021.07.023.
  33. Deep kernel principal component analysis for multi-level feature learning, 2023.
  34. Vapnik, V. N. The nature of statistical learning theory. Springer-Verlag New York, Inc., 1995. ISBN 0-387-94559-8.
  35. Time Series Prediction: Forecasting the Future and Understanding the Past. Addison-Wesley, 1993.
  36. Learning to find pre-images. In Advances in Neural Information Processing Systems, volume 16, 2003.
  37. Using the Nyström method to speed up kernel machines. In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
  38. Latent space exploration using generative kernel PCA. In Bogaerts, B., Bontempi, G., Geurts, P., Harley, N., Lebichot, B., Lenaerts, T., and Louppe, G. (eds.), Artificial Intelligence and Machine Learning, pp.  70–82, Cham, 2020. Springer International Publishing.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up