Papers
Topics
Authors
Recent
Search
2000 character limit reached

PAC bounds of continuous Linear Parameter-Varying systems related to neural ODEs

Published 7 Jul 2023 in cs.LG | (2307.03630v1)

Abstract: We consider the problem of learning Neural Ordinary Differential Equations (neural ODEs) within the context of Linear Parameter-Varying (LPV) systems in continuous-time. LPV systems contain bilinear systems which are known to be universal approximators for non-linear systems. Moreover, a large class of neural ODEs can be embedded into LPV systems. As our main contribution we provide Probably Approximately Correct (PAC) bounds under stability for LPV systems related to neural ODEs. The resulting bounds have the advantage that they do not depend on the integration interval.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Lpv modeling of nonlinear systems: A multi-path feedback linearization approach. International Journal of Robust and Nonlinear Control, 31(18):9436–9465, 2021. doi: https://doi.org/10.1002/rnc.5799. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/rnc.5799.
  2. Antoulas, A. Approximation of Large-Scale Dynamical Systems. Advances in Design and Control. 2009. ISBN 9780898716580. URL https://books.google.fr/books?id=HntlsoAR0NoC.
  3. Fast and exact enumeration of deep networks partitions regions. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1–5, 2023. doi: 10.1109/ICASSP49357.2023.10095698.
  4. Finite sample properties of system identification methods. IEEE Transactions on Automatic Control, 47(8):1329–1334, 2002.
  5. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  6. Framing rnn as a kernel method: A neural ode approach. Advances in Neural Information Processing Systems, 34:3121–3134, 2021.
  7. Logarithmic regret for adversarial online control. In Proceedings of the 37th ICML, volume 119 of PMLR, pp.  3211–3221. PMLR, 7 2020.
  8. Reduced-order modeling of lpv systems in the loewner framework. In 2021 60th IEEE Conference on Decision and Control (CDC), pp.  3299–3305. IEEE, 2021.
  9. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019.
  10. Learning recurrent neural net models of nonlinear systems. In Learning for Dynamics and Control, pp.  425–435. PMLR, 2021.
  11. Adversarial robustness of stabilized neural ode might be from obfuscated gradients. In Bruna, J., Hesthaven, J., and Zdeborova, L. (eds.), Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pp.  497–515. PMLR, 16–19 Aug 2022. URL https://proceedings.mlr.press/v145/huang22a.html.
  12. Isidori, A. Nonlinear control systems: an introduction. Springer, 1985.
  13. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  14925–14937. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/7d5430cf85f78c4b7aa09813b14bce0d-Paper.pdf.
  14. Kidger, P. On neural differential equations. arXiv preprint arXiv:2202.02435, 2022.
  15. Koelewijn, P. Analysis and Control of Nonlinear Systems with Stability and Performance Guarantees: A Linear Parameter-Varying Approach. PhD thesis, Electrical Engineering, February 2023. Proefschrift.
  16. Vapnik-chervonenkis dimension of recurrent neural networks. Discrete Applied Mathematics, 86(1):63–79, 1998.
  17. Krener, A. Bilinear and nonlinear realization of input-output maps. SIAM Journal on Control, 13(4), 1974.
  18. Learning complexity dimensions for a continuous-time control system. SIAM journal on control and optimization, 43(3):872–898, 2004.
  19. Logarithmic regret bound in partially observable linear dynamical systems. Advances in Neural Information Processing Systems, 33:20876–20888, 2020.
  20. Marion, P. Generalization bounds for neural ordinary differential equations and deep residual networks. arXiv preprint arXiv:2305.06648, 2023.
  21. Dissecting neural odes. Advances in Neural Information Processing Systems, 33:3952–3963, 2020.
  22. Control of Linear Parameter Varying Systems with Applications. Springer, 2012.
  23. Revisiting ho–kalman-based system identification: Robustness and finite-sample analysis. IEEE Transactions on Automatic Control, 67(4):1914–1928, 4 2022.
  24. Lyanet: A lyapunov framework for training neural odes. In International Conference on Machine Learning, pp. 18687–18703. PMLR, 2022.
  25. Do residual neural networks discretize neural ordinary differential equations? Advances in Neural Information Processing Systems, 35:36520–36532, 2022.
  26. Finite time LTI system identification. J. Mach. Learn. Res., 22:26:1–26:61, 2021.
  27. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
  28. Learning linear dynamical systems with semi-parametric least squares. In Conference on Learning Theory, pp.  2714–2802. PMLR, 2019.
  29. Sontag, E. D. A learning result for continuous-time recurrent neural networks. Systems & control letters, 34(3):151–158, 1998.
  30. Tóth, R. Modeling and identification of linear parameter-varying systems, volume 403. Springer, 2010.
  31. Finite sample analysis of stochastic system identification. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp.  3648–3654, 2019.
  32. Learning stable and robust linear parameter-varying state-space models, 2023.
  33. A learning theory approach to system identification and stochastic adaptive control. Probabilistic and randomized methods for design under uncertainty, pp.  265–302, 2006.
  34. Weinan, E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 1(5):1–11, 2017.
  35. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pp. 5286–5295. PMLR, 2018.
  36. Stability analysis using quadratic constraints for systems with neural network controllers. IEEE Transactions on Automatic Control, 67(4):1980–1987, 2022. doi: 10.1109/TAC.2021.3069388.
  37. On robust classification using contractive hamiltonian neural odes. arXiv preprint arXiv:2203.11805, 2022.
  38. On ℋ2subscriptℋ2\mathcal{H}_{2}caligraphic_H start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT model reduction of bilinear systems. Automatica, 38:205–216, 2002.

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up