Fading memory and the convolution theorem (2408.07386v3)
Abstract: Several topological and analytical notions of continuity and fading memory for causal and time-invariant filters are introduced, and the relations between them are analyzed. A significant generalization of the convolution theorem that establishes the equivalence between the fading memory property and the availability of convolution representations of linear filters is proved. This result extends a previous similar characterization to a complete array of weighted norms in the definition of the fading memory property. Additionally, the main theorem shows that the availability of convolution representations can be characterized, at least when the codomain is finite-dimensional, not only by the fading memory property but also by the reunion of two purely topological notions that are called minimal continuity and minimal fading memory property. Finally, when the input space and the codomain of a linear functional are Hilbert spaces, it is shown that minimal continuity and the minimal fading memory property guarantee the existence of interesting embeddings of the associated reproducing kernel Hilbert spaces.
- Borys, A. Relationships between two definitions of fading memory for discrete-time systems. International Journal of Electronics and Telecommunications 61, 4 (2015), 377–380.
- Fading memory and the problem of approximating nonlinear operators with Volterra series. IEEE Transactions on Circuits and Systems 32, 11 (1985), 1150–1161.
- Support Vector Machines. Springer, 2008.
- On the general theory of fading memory. Archive for Rational Mechanics and Analysis 29, 1 (jan 1968), 18–31.
- Conway, J. B. A Course in Functional Analysis, 2 ed., vol. 96 of Graduate Texts in Mathematics. Springer, New York, NY, 2007.
- Reservoir kernels and Volterra series. arXiv:2212.14641v1 (2022).
- Approximation error estimates for random neural networks and reservoir systems. The Annals of Applied Probability 33, 1 (2023), 28–69.
- Infinite-dimensional reservoir computing. Neural Networks 179 (2024), 106486.
- Fading memory echo state networks are universal. Neural Networks 138 (2021), 10–13.
- Echo state networks are universal. Neural Networks 108 (2018), 495–508.
- Differentiable reservoir computing. Journal of Machine Learning Research 20, 179 (2019), 1–62.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752v2 (2023).
- Jaeger, H. The “echo state” approach to analysing and training recurrent neural networks – with an Erratum note. Tech. Rep. GMD Report 148, German National Research Center for Information Technology, 2010.
- A Brief Survey on the Approximation Theory for Sequence Modelling. Journal of Machine Learning 2, 1 (2023), 1–30.
- Introduction to Banach Spaces: Analysis and Probability Volume 1, vol. 166 of Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2017. Translated by Danièle Gibbons and Greg Gibbons.
- On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis. In International Conference on Learning Representations (2021).
- Dimension reduction in recurrent networks by canonicalization. Journal of Geometric Mechanics 13, 4 (2021), 647–677.
- Computational aspects of feedback in neural circuits. PLoS Computational Biology 3, 1 (2007), e165.
- Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation 14 (2002), 2531–2560.
- Fading memory and kernel properties of generic cortical microcircuit models. Journal of Physiology Paris 98, 4-6 SPEC. ISS. (2004), 315–330.
- Neural Systems as Nonlinear Filters. Neural Computation 12, 8 (aug 2000), 1743–1772.
- Matthews, M. B. On the Uniform Approximation of Nonlinear Discrete-Time Fading-Memory Systems Using Neural Network Models. PhD thesis, ETH Zurich, 1992.
- Matthews, M. B. Approximating nonlinear fading-memory operators using neural network models. Circuits, Systems, and Signal Processing 12, 2 (jun 1993), 279–307.
- Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges. arXiv:2404.16112v1 (2024).
- SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series. arXiv:2403.15360v2 (2024).
- Perryman, P. C. Approximation Theory for Deterministic and Stochastic Nonlinear Systems. PhD thesis, University of California, Irvine, 1996.
- Rugh, W. J. Nonlinear System Theory. The Volterra/Wiener Approach. The Johns Hopkins University Press, 1981.
- Sandberg, I. W. Approximation theorems for discrete-time systems. IEEE Transactions on Circuits and Systems 38, 5 (1991), 564–566.
- Sandberg, I. W. Structure theorems for nonlinear systems. Multidimensional Systems and Signal Processing 2 (1991), 267–286.
- Sandberg, I. W. Z+ fading memory and extensions of input-output maps. International Journal of Circuit Theory and Applications 29, 11 (2001), 381–388.
- Sandberg, I. W. Notes of fading-memory conditions. Circuits, Systems, and Signal Processing 22, 1 (2003), 43–55.
- A Generalized Representer Theorem. In Computational Learning Theory (2001), D. Helmbold and B. Williamson, Eds., Springer Berlin Heidelberg, pp. 416–426.
- Current state of system approximation for deterministic and stochastic systems. In Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers (1997), vol. 1, IEEE Comput. Soc. Press, pp. 141–145.
- Attention is All you Need. In Advances in Neural Information Processing Systems (2017), I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30, Curran Associates, Inc.
- Volterra, V. Theory of Functionals and of Integral and Integro-Differential Equations. Dover, 1959.
- Wahba, G. Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, 1990.
- Wiener, N. Nonlinear Problems in Random Theory. The Technology Press of MIT, 1958.
- Fading memory and stability. Journal of the Franklin Institute 340, 6-7 (2004), 489–502.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.