Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Revisiting Deep Generalized Canonical Correlation Analysis (2312.13455v1)

Published 20 Dec 2023 in cs.LG, eess.SP, and stat.ML

Abstract: Canonical correlation analysis (CCA) is a classic statistical method for discovering latent co-variation that underpins two or more observed random vectors. Several extensions and variations of CCA have been proposed that have strengthened our capabilities in terms of revealing common random factors from multiview datasets. In this work, we first revisit the most recent deterministic extensions of deep CCA and highlight the strengths and limitations of these state-of-the-art methods. Some methods allow trivial solutions, while others can miss weak common factors. Others overload the problem by also seeking to reveal what is not common among the views -- i.e., the private components that are needed to fully reconstruct each view. The latter tends to overload the problem and its computational and sample complexities. Aiming to improve upon these limitations, we design a novel and efficient formulation that alleviates some of the current restrictions. The main idea is to model the private components as conditionally independent given the common ones, which enables the proposed compact formulation. In addition, we also provide a sufficient condition for identifying the common random factors. Judicious experiments with synthetic and real datasets showcase the validity of our claims and the effectiveness of the proposed approach.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. S. M. Kakade and D. P. Foster, “Multi-view regression via canonical correlation analysis,” in International Conference on Computational Learning Theory.   Springer, 2007, pp. 82–96.
  2. D. P. Foster, S. M. Kakade, and T. Zhang, “Multi-view dimensionality reduction via canonical correlation analysis,” 2008.
  3. K. Chaudhuri, S. M. Kakade, K. Livescu, and K. Sridharan, “Multi-view clustering via canonical correlation analysis,” in Proceedings of the 26th annual international conference on machine learning, 2009, pp. 129–136.
  4. H. Hotelling, “Relations between two sets of variates,” Biometrika, vol. 28, no. 3-4, pp. 321–377, 1936.
  5. R. Arora and K. Livescu, “Kernel cca for multi-view learning of acoustic features using articulatory measurements,” in Symposium on machine learning in speech and language processing, 2012.
  6. W. Wang, R. Arora, and K. Livescu, “Reconstruction of articulatory measurements with smoothed low-rank matrix completion,” in 2014 IEEE Spoken Language Technology Workshop (SLT).   IEEE, 2014, pp. 54–59.
  7. W. Wang, R. Arora, K. Livescu, and J. A. Bilmes, “Unsupervised learning of acoustic features via deep canonical correlation analysis,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2015, pp. 4590–4594.
  8. M. S. Ibrahim and N. D. Sidiropoulos, “Cell-edge interferometry: Reliable detection of unknown cell-edge users via canonical correlation analysis,” in 2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC).   IEEE, 2019, pp. 1–5.
  9. M. S. Ibrahim, P. A. Karakasis, and N. D. Sidiropoulos, “A simple and practical underlay scheme for short-range secondary communication,” IEEE Transactions on Wireless Communications, 2022.
  10. Y.-O. Li, T. Adali, W. Wang, and V. D. Calhoun, “Joint blind source separation by multiset canonical correlation analysis,” IEEE Transactions on Signal Processing, vol. 57, no. 10, pp. 3918–3929, 2009.
  11. N. M. Correa, T. Adali, Y.-O. Li, and V. D. Calhoun, “Canonical correlation analysis for data fusion and group inferences,” IEEE signal processing magazine, vol. 27, no. 4, pp. 39–50, 2010.
  12. J. R. Katthi and S. Ganapathy, “Deep correlation analysis for audio-eeg decoding,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, pp. 2742–2753, 2021.
  13. P. A. Karakasis, A. P. Liavas, N. D. Sidiropoulos, P. G. Simos, and E. Papadaki, “Multi-subject task-related fmri data processing via a two-stage generalized canonical correlation analysis,” IEEE Transactions on Image Processing, 2022.
  14. P. L. Lai and C. Fyfe, “Kernel and nonlinear canonical correlation analysis,” International Journal of Neural Systems, vol. 10, no. 05, pp. 365–377, 2000.
  15. W. Wang, X. Yan, H. Lee, and K. Livescu, “Deep variational canonical correlation analysis,” arXiv preprint arXiv:1610.03454, 2016.
  16. T. Michaeli, W. Wang, and K. Livescu, “Nonparametric canonical correlation analysis,” in International conference on machine learning.   PMLR, 2016, pp. 1967–1976.
  17. G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in International conference on machine learning.   PMLR, 2013, pp. 1247–1255.
  18. W. Wang, R. Arora, K. Livescu, and J. Bilmes, “On deep multi-view representation learning,” in International conference on machine learning.   PMLR, 2015, pp. 1083–1092.
  19. A. Benton, H. Khayrallah, B. Gujral, D. A. Reisinger, S. Zhang, and R. Arora, “Deep generalized canonical correlation analysis,” arXiv preprint arXiv:1702.02519, 2017.
  20. Q. Lyu and X. Fu, “Nonlinear multiview analysis: Identifiability and neural network-assisted implementation,” IEEE Transactions on Signal Processing, vol. 68, pp. 2697–2712, 2020.
  21. Q. Lyu, X. Fu, W. Wang, and S. Lu, “Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective,” in International Conference on Learning Representations, 2021.
  22. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
  23. J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in International Conference on Machine Learning.   PMLR, 2021, pp. 12 310–12 320.
  24. X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 750–15 758.
  25. P. Horst, “Generalized canonical correlations and their applications to experimental data,” Journal of Clinical Psychology, vol. 17, no. 4, pp. 331–347, 1961.
  26. J. R. Kettenring, “Canonical analysis of several sets of variables,” Biometrika, vol. 58, no. 3, pp. 433–451, 1971.
  27. N. A. Asendorf, “Informative data fusion: Beyond canonical correlation analysis,” Ph.D. dissertation, 2015.
  28. M. Sørensen, C. I. Kanatsoulis, and N. D. Sidiropoulos, “Generalized canonical correlation analysis: A subspace intersection approach,” IEEE Transactions on Signal Processing, vol. 69, pp. 2452–2467, 2021.
  29. M. Borga, “Canonical correlation: a tutorial,” On line tutorial http://people. imt. liu. se/magnus/cca, vol. 4, no. 5, 2001.
  30. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, and L. Bottou, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” Journal of machine learning research, vol. 11, no. 12, 2010.
  31. A. Hyvarinen, H. Sasaki, and R. Turner, “Nonlinear ica using auxiliary variables and generalized contrastive learning,” in The 22nd International Conference on Artificial Intelligence and Statistics.   PMLR, 2019, pp. 859–868.
  32. J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello, “Self-supervised learning with data augmentations provably isolates content from style,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  33. D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” Advances in neural information processing systems, vol. 31, 2018.
  34. P. H. Schönemann, “A generalized solution of the orthogonal procrustes problem,” Psychometrika, vol. 31, no. 1, pp. 1–10, 1966.
  35. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
  36. K. Y. Yeung and W. L. Ruzzo, “Details of the adjusted rand index and clustering algorithms, supplement to the paper an empirical study on principal component analysis for clustering gene expression data,” Bioinformatics, vol. 17, no. 9, pp. 763–774, 2001.
  37. J. R. Westbury, G. Turner, and J. Dembowski, “X-ray microbeam speech production database user’s handbook,” University of Wisconsin, 1994.
  38. R. Arora and K. Livescu, “Multi-view learning with supervision for transformed bottleneck features,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2014, pp. 2499–2503.
  39. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  40. L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now