Liquid Resistance Liquid Capacitance Networks (2403.08791v3)
Abstract: We introduce liquid-resistance liquid-capacitance neural networks (LRCs), a neural-ODE model which considerably improve the generalization, accuracy, and biological plausibility of electrical equivalent circuits (EECs), liquid time-constant networks (LTCs), and saturated liquid time-constant networks (STCs), respectively. We also introduce LRC units (LRCUs), as a very efficient and accurate gated RNN-model, which results from solving LRCs with an explicit Euler scheme using just one unfolding. We empirically show and formally prove that the liquid capacitance of LRCs considerably dampens the oscillations of LTCs and STCs, while at the same time dramatically increasing accuracy even for cheap solvers. We experimentally demonstrate that LRCs are a highly competitive alternative to popular neural ODEs and gated RNNs in terms of accuracy, efficiency, and interpretability, on classic time-series benchmarks and a complex autonomous-driving lane-keeping task.
- Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
- Learning with chemical versus electrical synapses – does it make a difference? arXiv preprint arXiv:2401.08602, 2023.
- Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):2222–2232, 2016.
- Liquid time-constant networks. In AAAI Conference on Artificial Intelligence, 2020.
- Long short-term memory. Neural Comput., 9(8):1735–1780, nov 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL https://doi.org/10.1162/neco.1997.9.8.1735.
- An empirical exploration of recurrent network architectures. In International Conference on Machine Learning, 2015. URL https://api.semanticscholar.org/CorpusID:9668607.
- Principles of neural science, volume 4. McGraw-hill New York, 2000.
- Adam: A method for stochastic optimization, 2017.
- A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.
- Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418, 2020.
- Designing worm-inspired neural networks for interpretable robotic control. In 2019 International Conference on Robotics and Automation (ICRA), pp. 87–94, 2019. doi: 10.1109/ICRA.2019.8793840.
- Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2:642–652, 2020.
- Are all vision models created equal? a study of the open-loop to closed-loop causality gap. arXiv preprint arXiv:2210.04303, 2022.
- Fixing weight decay regularization in adam. ArXiv, abs/1711.05101, 2017. URL https://api.semanticscholar.org/CorpusID:3312944.
- Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-1015.
- A simple step size selection algorithm for ode codes. Journal of Computational and Applied Mathematics, 58(3):345–354, 1995. ISSN 0377-0427. doi: https://doi.org/10.1016/0377-0427(94)00007-N. URL https://www.sciencedirect.com/science/article/pii/037704279400007N.
- Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. Technical report, 2017.
- Localization Data for Person Activity. UCI Machine Learning Repository, 2010. DOI: https://doi.org/10.24432/C57G8X.
- A dynamic network simulation of the nematode tap withdrawal circuit: predictions concerning synaptic function using behavioral criteria. Journal of Neuroscience, 16(12):4017–4031, 1996.
- Minimal gated unit for recurrent neural networks. International Journal of Automation and Computing, 13(3):226–234, 2016.