Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning Low-dimensional Latent Dynamics from High-dimensional Observations: Non-asymptotics and Lower Bounds (2405.06089v3)

Published 9 May 2024 in eess.SY, cs.IT, cs.LG, cs.SY, and math.IT

Abstract: In this paper, we focus on learning a linear time-invariant (LTI) model with low-dimensional latent variables but high-dimensional observations. We provide an algorithm that recovers the high-dimensional features, i.e. column space of the observer, embeds the data into low dimensions and learns the low-dimensional model parameters. Our algorithm enjoys a sample complexity guarantee of order $\tilde{\mathcal{O}}(n/\epsilon2)$, where $n$ is the observation dimension. We further establish a fundamental lower bound indicating this complexity bound is optimal up to logarithmic factors and dimension-independent constants. We show that this inevitable linear factor of $n$ is due to the learning error of the observer's column space in the presence of high-dimensional noises. Extending our results, we consider a meta-learning problem inspired by various real-world applications, where the observer column space can be collectively learned from datasets of multiple LTI systems. An end-to-end algorithm is then proposed, facilitating learning LTI systems from a meta-dataset which breaks the sample complexity lower bound in certain scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Improved Algorithms for Linear Stochastic Bandits. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://papers.nips.cc/paper_files/paper/2011/hash/e1d5be1c7f2f456670de3d53c7b54f4a-Abstract.html.
  2. Linear System Challenges of Dynamic Factor Models. Econometrics, 10(4):35, December 2022. ISSN 2225-1146. doi: 10.3390/econometrics10040035. URL https://www.mdpi.com/2225-1146/10/4/35. Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.
  3. A New Approach to Learning Linear Dynamical Systems. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC 2023, pp.  335–348, New York, NY, USA, June 2023. Association for Computing Machinery. ISBN 978-1-4503-9913-5. doi: 10.1145/3564246.3585247. URL https://dl.acm.org/doi/10.1145/3564246.3585247.
  4. Streaming PCA and Subspace Tracking: The Missing Data Case. Proceedings of the IEEE, 106(8):1293–1310, August 2018. ISSN 1558-2256. doi: 10.1109/JPROC.2018.2847041. URL https://ieeexplore.ieee.org/document/8417980. Conference Name: Proceedings of the IEEE.
  5. Statistical properties of kernel principal component analysis. Machine Learning, 66(2):259–294, March 2007. ISSN 1573-0565. doi: 10.1007/s10994-006-6895-9. URL https://doi.org/10.1007/s10994-006-6895-9.
  6. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford, New York, March 2013. ISBN 978-0-19-953525-5.
  7. Dynamic factor models. Allgemeines Statistisches Archiv, 90(1):27–42, March 2006. ISSN 1614-0176. doi: 10.1007/s10182-006-0219-z. URL https://doi.org/10.1007/s10182-006-0219-z.
  8. Model Reduction for Large-Scale Systems with High-Dimensional Parametric Input Space. SIAM Journal on Scientific Computing, 30(6):3270–3288, January 2008. ISSN 1064-8275. doi: 10.1137/070694855. URL https://epubs.siam.org/doi/abs/10.1137/070694855. Publisher: Society for Industrial and Applied Mathematics.
  9. Robust principal component analysis? Journal of the ACM, 58(3):11:1–11:37, June 2011. ISSN 0004-5411. doi: 10.1145/1970392.1970395. URL https://dl.acm.org/doi/10.1145/1970392.1970395.
  10. Consistent Estimation of Low-Dimensional Latent Structure in High-Dimensional Data, October 2015. URL http://arxiv.org/abs/1510.03497. arXiv:1510.03497 [stat].
  11. Neural population dynamics during reaching. Nature, 487(7405):51–56, July 2012. ISSN 1476-4687. doi: 10.1038/nature11129. URL https://www.nature.com/articles/nature11129. Number: 7405 Publisher: Nature Publishing Group.
  12. The Rotation of Eigenvectors by a Perturbation. III. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970. ISSN 0036-1429. URL https://www.jstor.org/stable/2949580. Publisher: Society for Industrial and Applied Mathematics.
  13. Invariant subspace learning for time series data based on dynamic time warping distance. Pattern Recognition, 102:107210, June 2020. ISSN 0031-3203. doi: 10.1016/j.patcog.2020.107210. URL https://www.sciencedirect.com/science/article/pii/S0031320320300169.
  14. Non asymptotic estimation lower bounds for LTI state space models with Cram\’er-Rao and van Trees, September 2021. URL http://arxiv.org/abs/2109.08582. arXiv:2109.08582 [math, stat].
  15. Efficient learning of hidden state LTI state space models of unknown order, February 2022. URL http://arxiv.org/abs/2202.01625. arXiv:2202.01625 [math, stat].
  16. Extracting a low-dimensional predictable time series. Optimization and Engineering, 23(2):1189–1214, June 2022. ISSN 1389-4420, 1573-2924. doi: 10.1007/s11081-021-09643-x. URL https://link.springer.com/10.1007/s11081-021-09643-x.
  17. Fattahi, S. Learning Partially Observed Linear Dynamical Systems from Logarithmic Number of Samples. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, pp.  60–72. PMLR, May 2021. URL https://proceedings.mlr.press/v144/fattahi21a.html. ISSN: 2640-3498.
  18. Long-term stability of cortical population dynamics underlying consistent behavior. Nature Neuroscience, 23(2):260–270, February 2020. ISSN 1546-1726. doi: 10.1038/s41593-019-0555-4. URL https://www.nature.com/articles/s41593-019-0555-4. Number: 2 Publisher: Nature Publishing Group.
  19. Sample Complexity of Dictionary Learning and Other Matrix Factorizations. IEEE Transactions on Information Theory, 6(61):3469–3486, 2015. ISSN 0018-9448, 1557-9654. doi: 10.1109/TIT.2015.2424238. URL https://www.infona.pl//resource/bwmeta1.element.ieee-art-000007088631.
  20. Continuous multiplexed population representations of task context in the mouse primary visual cortex. Nature Communications, 14(1):6687, October 2023. ISSN 2041-1723. doi: 10.1038/s41467-023-42441-w. URL https://www.nature.com/articles/s41467-023-42441-w. Number: 1 Publisher: Nature Publishing Group.
  21. Factor models for high-dimensional functional time series I: Representation results. Journal of Time Series Analysis, 44(5-6):578–600, 2023. ISSN 1467-9892. doi: 10.1111/jtsa.12676. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/jtsa.12676. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jtsa.12676.
  22. Hespanha, J. P. Linear Systems Theory: Second Edition. Princeton University Press, February 2018. ISBN 978-1-4008-9008-8.
  23. Finite-Time Identification of Linear Systems: Fundamental Limits and Optimal Algorithms. IEEE Transactions on Automatic Control, 68(5):2805–2820, May 2023. ISSN 1558-2523. doi: 10.1109/TAC.2022.3221705. URL https://ieeexplore.ieee.org/document/9946382. Conference Name: IEEE Transactions on Automatic Control.
  24. Lee, H. Improved rates for prediction and identification of partially observed linear dynamical systems, March 2022. URL http://arxiv.org/abs/2011.10006. arXiv:2011.10006 [cs, eess, math, stat].
  25. Li, R.-C. Relative Perturbation Theory: II. Eigenspace and Singular Subspace Variations. SIAM Journal on Matrix Analysis and Applications, 20(2):471–492, June 1998. ISSN 0895-4798. doi: 10.1137/S0895479896298506. URL https://epubs.siam.org/doi/abs/10.1137/S0895479896298506. Publisher: Society for Industrial and Applied Mathematics.
  26. Merging simulation and projection approaches to solve high-dimensional problems with an application to a new Keynesian model. Quantitative Economics, 6(1):1–47, 2015. ISSN 1759-7331. doi: 10.3982/QE364. URL https://onlinelibrary.wiley.com/doi/abs/10.3982/QE364. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.3982/QE364.
  27. Finite-Time Model Inference From A Single Noisy Trajectory, January 2021. URL http://arxiv.org/abs/2010.06616. arXiv:2010.06616 [cs, eess].
  28. Stimulus-dependent representational drift in primary visual cortex. Nature Communications, 12(1):5169, August 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-25436-3. URL https://www.nature.com/articles/s41467-021-25436-3. Number: 1 Publisher: Nature Publishing Group.
  29. Machine learning advances for time series forecasting. Journal of Economic Surveys, 37(1):76–111, 2023. ISSN 1467-6419. doi: 10.1111/joes.12429. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/joes.12429. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/joes.12429.
  30. Recurrent Neural Networks: Design and Applications. CRC Press, December 1999. ISBN 978-1-4200-4917-6. Google-Books-ID: ME1SAkN0PyMC.
  31. Time-series forecasting of Bitcoin prices using high-dimensional features: a machine learning approach. Neural Computing and Applications, July 2020. ISSN 1433-3058. doi: 10.1007/s00521-020-05129-6. URL https://doi.org/10.1007/s00521-020-05129-6.
  32. Non-asymptotic Identification of LTI Systems from a Single Trajectory, February 2019. URL http://arxiv.org/abs/1806.05722. arXiv:1806.05722 [cs, math, stat].
  33. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods, 15(10):805–815, October 2018. ISSN 1548-7105. doi: 10.1038/s41592-018-0109-9. URL https://www.nature.com/articles/s41592-018-0109-9. Number: 10 Publisher: Nature Publishing Group.
  34. Closed-form results for vector moving average models with a univariate estimation approach. Econometrics and Statistics, 10:27–52, April 2019. ISSN 2452-3062. doi: 10.1016/j.ecosta.2018.06.003. URL https://www.sciencedirect.com/science/article/pii/S2452306218300327.
  35. Qin, S. J. Latent vector autoregressive modeling and feature analysis of high dimensional and noisy data from dynamic systems. AIChE Journal, 68:e17703, June 2022. ISSN 0001-1541. doi: 10.1002/aic.17703. URL https://ui.adsabs.harvard.edu/abs/2022AIChE..68E7703Q. ADS Bibcode: 2022AIChE..68E7703Q.
  36. Rudelson, M. Random Vectors in the Isotropic Position. Journal of Functional Analysis, 164(1):60–72, May 1999. ISSN 0022-1236. doi: 10.1006/jfan.1998.3384. URL https://www.sciencedirect.com/science/article/pii/S0022123698933845.
  37. Near optimal finite time identification of arbitrary linear dynamical systems. In Proceedings of the 36th International Conference on Machine Learning, pp.  5610–5618. PMLR, May 2019. URL https://proceedings.mlr.press/v97/sarkar19a.html. ISSN: 2640-3498.
  38. Finite time LTI system identification. The Journal of Machine Learning Research, 22(1):26:1186–26:1246, January 2021. ISSN 1532-4435.
  39. An Introduction to Locally Linear Embedding.
  40. Schutter, B. D. Minimal state-space realization in linear system theory: an overview. Journal of Computational and Applied Mathematics, 121(1):331–354, September 2000. ISSN 0377-0427. doi: 10.1016/S0377-0427(00)00341-1. URL https://www.sciencedirect.com/science/article/pii/S0377042700003411.
  41. Finite time identification in unstable linear systems. Automatica, 96:342–353, October 2018. ISSN 0005-1098. doi: 10.1016/j.automatica.2018.07.008. URL https://www.sciencedirect.com/science/article/pii/S0005109818303546.
  42. Linear time-invariant system reduction using a mixed methods approach. Applied Mathematical Modelling, 39(16):4848–4858, August 2015. ISSN 0307-904X. doi: 10.1016/j.apm.2015.04.014. URL https://www.sciencedirect.com/science/article/pii/S0307904X15002504.
  43. Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification. In Proceedings of the 31st Conference On Learning Theory, pp.  439–473. PMLR, July 2018. URL https://proceedings.mlr.press/v75/simchowitz18a.html. ISSN: 2640-3498.
  44. Stewart, G. W. G. W. Matrix perturbation theory. Computer science and scientific computing. Academic Press, Boston, 1990. ISBN 0-12-670230-6.
  45. High-dimensional geometry of population responses in visual cortex. Nature, 571(7765):361–365, July 2019. ISSN 1476-4687. doi: 10.1038/s41586-019-1346-5.
  46. Finite Time Performance Analysis of MIMO Systems Identification, October 2023. URL http://arxiv.org/abs/2310.11790. arXiv:2310.11790 [cs, eess].
  47. Finite Sample Identification of Low-Order LTI Systems via Nuclear Norm Regularization. IEEE Open Journal of Control Systems, 1:237–254, 2022. ISSN 2694-085X. doi: 10.1109/OJCSYS.2022.3200015. URL https://ieeexplore.ieee.org/document/9870857. Conference Name: IEEE Open Journal of Control Systems.
  48. Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks. Neural Computation, 25(3):626–649, March 2013. ISSN 0899-7667. doi: 10.1162/NECO˙a˙00409. URL https://doi.org/10.1162/NECO_a_00409.
  49. Data-driven Optimal Filtering for Linear Systems with Unknown Noise Covariances, October 2023. URL http://arxiv.org/abs/2305.17836. arXiv:2305.17836 [cs, eess, math].
  50. Toward Understanding Latent Model Learning in MuZero: A Case Study in Linear Quadratic Gaussian Control. July 2023. URL https://openreview.net/forum?id=r9YZ357Trz.
  51. Provable Meta-Learning of Linear Representations. In Proceedings of the 38th International Conference on Machine Learning, pp.  10434–10443. PMLR, July 2021. URL https://proceedings.mlr.press/v139/tripuraneni21a.html. ISSN: 2640-3498.
  52. Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recovery. IEEE Signal Processing Magazine, 35:32–55, July 2018. ISSN 1053-5888. doi: 10.1109/MSP.2018.2826566. URL https://ui.adsabs.harvard.edu/abs/2018ISPM...35d..32V. ADS Bibcode: 2018ISPM…35d..32V.
  53. Vershynin, R. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018. ISBN 978-1-108-41519-4. doi: 10.1017/9781108231596. URL https://www.cambridge.org/core/books/highdimensional-probability/797C466DA29743D2C8213493BD2D2102.
  54. Stable representation of a naturalistic movie emerges from episodic activity with gain variability. Nature Communications, 12(1):5170, August 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-25437-2. URL https://www.nature.com/articles/s41467-021-25437-2. Number: 1 Publisher: Nature Publishing Group.
  55. Analysis of different RNN autoencoder variants for time series classification and machine prognostics. Mechanical Systems and Signal Processing, 149:107322, February 2021. ISSN 0888-3270. doi: 10.1016/j.ymssp.2020.107322. URL https://www.sciencedirect.com/science/article/pii/S0888327020307081.
  56. Meta-Learning Operators to Optimality from Multi-Task Non-IID Data, August 2023. URL http://arxiv.org/abs/2308.04428. arXiv:2308.04428 [cs, eess, stat].
  57. Non-Asymptotic Identification of Linear Dynamical Systems Using Multiple Trajectories. IEEE Control Systems Letters, 5(5):1693–1698, November 2021. ISSN 2475-1456. doi: 10.1109/LCSYS.2020.3042924. URL https://ieeexplore.ieee.org/document/9284539. Conference Name: IEEE Control Systems Letters.

Summary

We haven't generated a summary for this paper yet.

X Twitter Logo Streamline Icon: https://streamlinehq.com