2000 character limit reached
From latent dynamics to meaningful representations (2209.00905v4)
Published 2 Sep 2022 in cs.LG, cond-mat.dis-nn, cond-mat.stat-mech, physics.chem-ph, and physics.comp-ph
Abstract: While representation learning has been central to the rise of machine learning and artificial intelligence, a key problem remains in making the learned representations meaningful. For this, the typical approach is to regularize the learned representation through prior probability distributions. However, such priors are usually unavailable or are ad hoc. To deal with this, recent efforts have shifted towards leveraging the insights from physical principles to guide the learning process. In this spirit, we propose a purely dynamics-constrained representation learning framework. Instead of relying on predefined probabilities, we restrict the latent representation to follow overdamped Langevin dynamics with a learnable transition density - a prior driven by statistical mechanics. We show this is a more natural constraint for representation learning in stochastic dynamical systems, with the crucial ability to uniquely identify the ground truth representation. We validate our framework for different systems including a real-world fluorescent DNA movie dataset. We show that our algorithm can uniquely identify orthogonal, isometric and meaningful latent representations.
- Schölkopf, B.; Janzing, D.; Peters, J.; Sgouritsa, E.; Zhang, K.; Mooij, J. On causal and anticausal learning. arXiv preprint arXiv:1206.6471 2012,
- Peters, J.; Janzing, D.; Schölkopf, B. Elements of causal inference: foundations and learning algorithms; The MIT Press, 2017
- Kalinin, S. V.; Zhang, S.; others Exploring particle dynamics during self-organization processes via rotationally invariant latent representations. arXiv preprint arXiv:2009.00783 2020,
- Kaufman, B.; Williams, E. C.; Underkoffler, C.; Pederson, R.; Mardirossian, N.; Watson, I.; Parkhill, J. COATI: Multimodal Contrastive Pretraining for Representing and Traversing Chemical Space. Journal of Chemical Information and Modeling 2023,
- Mehdi, S.; Smith, Z.; Herron, L.; Zou, Z.; Tiwary, P. Enhanced Sampling with Machine Learning. Annual Review of Physical Chemistry 2024, 75
- Tschannen, M.; Bachem, O.; Lucic, M. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069 2018,
- Kingma, D. P.; Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013,
- Locatello, F.; Bauer, S.; Lucic, M.; Gelly, S.; Schölkopf, B.; Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811.12359 2019,
- Greydanus, S.; Dzamba, M.; Yosinski, J. Hamiltonian neural networks. Advances in neural information processing systems 2019, 32
- Lutter, M.; Ritter, C.; Peters, J. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv preprint arXiv:1907.04490 2019,
- Cranmer, M.; Greydanus, S.; Hoyer, S.; Battaglia, P.; Spergel, D.; Ho, S. Lagrangian neural networks. arXiv preprint arXiv:2003.04630 2020,
- Toth, P.; Rezende, D. J.; Jaegle, A.; Racanière, S.; Botev, A.; Higgins, I. Hamiltonian generative networks. arXiv preprint arXiv:1909.13789 2019,
- Saemundsson, S.; Terenin, A.; Hofmann, K.; Deisenroth, M. Variational integrator networks for physically structured embeddings. International Conference on Artificial Intelligence and Statistics. 2020; pp 3078–3087
- Kaltenbach, S.; Koutsourelakis, P.-S. Physics-aware, probabilistic model order reduction with guaranteed stability. arXiv preprint arXiv:2101.05834 2021,
- Chung, J.; Kastner, K.; Dinh, L.; Goel, K.; Courville, A. C.; Bengio, Y. A recurrent latent variable model for sequential data. Advances in neural information processing systems 2015, 28
- Hsu, W.-N.; Zhang, Y.; Glass, J. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems 2017, 30
- Li, Y.; Mandt, S. Disentangled sequential autoencoder. International Conference on Machine Learning. 2018; pp 5670–5679
- Girin, L.; Leglaive, S.; Bie, X.; Diard, J.; Hueber, T.; Alameda-Pineda, X. Dynamical variational autoencoders: A comprehensive review. arXiv preprint arXiv:2008.12595 2020,
- Hasan, A.; Pereira, J. M.; Farsiu, S.; Tarokh, V. Identifying Latent Stochastic Differential Equations. IEEE Transactions on Signal Processing 2021,
- Hyvarinen, A.; Morioka, H. Nonlinear ICA of temporally dependent stationary sources. Artificial Intelligence and Statistics. 2017; pp 460–469
- Hälvä, H.; Hyvarinen, A. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. Conference on Uncertainty in Artificial Intelligence. 2020; pp 939–948
- Morioka, H.; Hälvä, H.; Hyvarinen, A. Independent innovation analysis for nonlinear vector autoregressive process. International Conference on Artificial Intelligence and Statistics. 2021; pp 1549–1557
- Khemakhem, I.; Kingma, D.; Monti, R.; Hyvarinen, A. Variational autoencoders and nonlinear ica: A unifying framework. International Conference on Artificial Intelligence and Statistics. 2020; pp 2207–2217
- Chen, X.; Kingma, D. P.; Salimans, T.; Duan, Y.; Dhariwal, P.; Schulman, J.; Sutskever, I.; Abbeel, P. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731 2016,
- Tolstikhin, I.; Bousquet, O.; Gelly, S.; Schoelkopf, B. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 2017,
- Dilokthanakul, N.; Mediano, P. A.; Garnelo, M.; Lee, M. C.; Salimbeni, H.; Arulkumaran, K.; Shanahan, M. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:.02648 2016,
- Tomczak, J. M.; Welling, M. VAE with a VampPrior. arXiv preprint arXiv:1705.07120 2017,
- Hoffman, M. D.; Johnson, M. J. Elbo surgery: yet another way to carve up the variational evidence lower bound. Workshop in Advances in Approximate Bayesian Inference, NIPS. 2016
- Rosca, M.; Lakshminarayanan, B.; Mohamed, S. Distribution matching in variational inference. arXiv preprint arXiv:1802.06847 2018,
- Kolouri, S.; Pope, P. E.; Martin, C. E.; Rohde, G. K. Sliced Wasserstein auto-encoders. International Conference on Learning Representations. 2018
- Kolouri, S.; Nadjahi, K.; Simsekli, U.; Badeau, R.; Rohde, G. Generalized sliced wasserstein distances. Advances in neural information processing systems 2019, 32
- Tishby, N.; Pereira, F. C.; Bialek, W. The information bottleneck method. arXiv preprint physics/0004057 2000,
- Alemi, A. A.; Fischer, I.; Dillon, J. V.; Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016,
- Rey, L. A. P.; Menkovski, V.; Portegies, J. W. Diffusion variational autoencoders. arXiv preprint arXiv:1901.08991 2019,
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.