Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Identifiable Feature Learning for Spatial Data with Nonlinear ICA (2311.16849v1)

Published 28 Nov 2023 in stat.ML and cs.LG

Abstract: Recently, nonlinear ICA has surfaced as a popular alternative to the many heuristic models used in deep representation learning and disentanglement. An advantage of nonlinear ICA is that a sophisticated identifiability theory has been developed; in particular, it has been proven that the original components can be recovered under sufficiently strong latent dependencies. Despite this general theory, practical nonlinear ICA algorithms have so far been mainly limited to data with one-dimensional latent dependencies, especially time-series data. In this paper, we introduce a new nonlinear ICA framework that employs $t$-process (TP) latent components which apply naturally to data with higher-dimensional dependency structures, such as spatial and spatio-temporal data. In particular, we develop a new learning and inference algorithm that extends variational inference methods to handle the combination of a deep neural network mixing function with the TP prior, and employs the method of inducing points for computational efficacy. On the theoretical side, we show that such TP independent components are identifiable under very general conditions. Further, Gaussian Process (GP) nonlinear ICA is established as a limit of the TP Nonlinear ICA model, and we prove that the identifiability of the latent components at this GP limit is more restricted. Namely, those components are identifiable if and only if they have distinctly different covariance kernels. Our algorithm and identifiability theorems are explored on simulated spatial data and real world spatio-temporal data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Sparse gaussian process variational autoencoders. arXiv preprint arXiv:2010.10177.
  2. Uncovering the structure of clinical EEG signals with self-supervised learning. J. Neural Engineering, 18(046020).
  3. A blind source separation technique based on second order statistics. IEEE Trans. on Signal Processing, 45(2):434–444.
  4. Gaussian process prior variational autoencoders. Advances in neural information processing systems, 31.
  5. Multivariate student-t regression models: Pitfalls and inference. Biometrika, 86(1):153–167.
  6. Hidden Markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Proc. 36th Conf.  on Uncertainty in Artificial Intelligence (UAI2020), Toronto, Canada (virtual).
  7. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34:1624–1633.
  8. Gaussian processes for big data. In Uncertainty in Artificial Intelligence, page 282. Citeseer.
  9. Scalable variational gaussian process classification. In Artificial Intelligence and Statistics, pages 351–360. PMLR.
  10. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media.
  11. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. arXiv preprint arXiv:2303.16535.
  12. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems (NIPS2016), Barcelona, Spain.
  13. Nonlinear ICA of temporally dependent stationary sources. In Proc. Artificial Intelligence and Statistics (AISTATS2017), Fort Lauderdale, Florida.
  14. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429–439.
  15. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In Proc. Artificial Intelligence and Statistics (AISTATS2019), Okinawa, Japan.
  16. Scalable gaussian process variational autoencoders. In International Conference on Artificial Intelligence and Statistics, pages 3511–3519. PMLR.
  17. Composing graphical models with neural networks for structured representations and fast inference. Advances in neural information processing systems, 29.
  18. Variational autoencoders and nonlinear ICA: A unifying framework. In Proc. Artificial Intelligence and Statistics (AISTATS2020).
  19. ICE-BeeM: Identifiable conditional energy-based deep models based on nonlinear ICA. In Advances in Neural Information Processing Systems (NeurIPS2020), Virtual.
  20. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
  21. Towards nonlinear disentanglement in natural data with temporal sparse coding. In 9th International Conference on Learning Representations, ICLR 2021.
  22. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research.
  23. Causal discovery with general non-linear relationships using non-linear ICA. In Proc. 35th Conf.  on Uncertainty in Artificial Intelligence (UAI2019), Tel Aviv, Israel.
  24. Pearce, M. (2020). The gaussian process prior vae for interpretable latent dynamics from pixels. In Symposium on advances in approximate bayesian inference, pages 1–12. PMLR.
  25. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research, 6(65):1939–1959.
  26. RadiantEarthFoundation (2020). CV4A competition Kenya crop type dataset. https://mlhub.earth/data/ref_african_crops_kenya_02.
  27. Black Box Variational Inference. In Kaski, S. and Corander, J., editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 814–822, Reykjavik, Iceland. PMLR.
  28. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR.
  29. Nonlinear independent component analysis for discrete-time and continuous-time signals. Annals of Statistics. In press.
  30. Student-t Processes as Alternatives to Gaussian Processes. In Kaski, S. and Corander, J., editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Research, pages 877–885, Reykjavik, Iceland. PMLR.
  31. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems, volume 18. MIT Press.
  32. Variational inference for student-t models: Robust bayesian interpolation and generalised component analysis. Neurocomputing, 69(1-3):123–141.
  33. Titsias, M. (2009). Variational learning of inducing variables in sparse gaussian processes. In van Dyk, D. and Welling, M., editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567–574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA. PMLR.
  34. Causal mosaic: Cause-effect inference via nonlinear ica and ensemble method. In International Conference on Artificial Intelligence and Statistics, pages 1157–1167. PMLR.
  35. Robust multi-task learning with t -processes. In Proceedings of the 24th International Conference on Machine Learning - ICML ’07, pages 1103–1110, Corvalis, Oregon. ACM Press.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now