An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling

Abstract: We establish a mathematical correspondence between state space models, a state-of-the-art architecture for capturing long-range dependencies in data, and an exactly solvable nonlinear oscillator network. As a specific example of this general correspondence, we analyze the diagonal linear time-invariant implementation of the Structured State Space Sequence model (S4). The correspondence embeds S4D, a specific implementation of S4, into a ring network topology, in which recent inputs are encoded, as waves of activity traveling over the one-dimensional spatial layout of the network. We then derive an exact operator expression for the full forward pass of S4D, yielding an analytical characterization of its complete input-output map. This expression reveals that the nonlinear decoder in the system induces interactions between these information-carrying waves that enable classifying real-world sequences. These results generalize across modern SSM architectures, and show that they admit an exact mathematical description with a clear physical interpretation. These insights enable a new level of interpretability for these systems in terms of nonlinear oscillator networks.

• The paper introduces an explicit operator-level expression for the full forward pass in diagonal SSMs, linking nonlinear activations to traveling wave dynamics.
• It employs Carleman embeddings and spectral techniques to quantitatively explain how nonlinear mixtures of modal energies contribute to classification performance.
• This framework enables principled architectural design and enhanced interpretability of sequence models by mapping SSM topologies to oscillator network dynamics.

Operator-Theoretic Foundations for Computation in Diagonal State Space Models

Introduction: Bridging SSMs and Nonlinear Oscillator Networks

This work establishes a rigorous mathematical correspondence between diagonal state-space models (SSMs)—notably the S4D architecture—and a specific class of exactly solvable nonlinear oscillator networks. Through this correspondence, SSMs are recast as networks exhibiting traveling wave dynamics on a ring topology, providing explicit modal and operator-theoretic characterizations of their end-to-end input-output computation. The analysis leverages spectral and modal techniques from nonlinear dynamical systems theory and applies Carleman embeddings to derive closed-form operator expressions explaining the forward pass, including both linear recurrence and nonlinear activation stages.

Mathematical Framework: From Oscillator Network to SSM

The analysis begins with a network of $N$ nodes on a 1D ring, with each node governed by nonlinear oscillator dynamics involving distance-dependent coupling and phase lags. By a specific variable substitution, these nonlinear equations admit exact solutions in terms of matrix exponentiation and are equivalently interpreted as the repeated action of a discrete propagator on a state vector.

Figure 1: The oscillator network produces distinct spatiotemporal phenomena such as traveling waves and chimeras, mapping onto the computational repertoire observed in SSMs.

Diagonalization of circulant connectivity matrices reveals that the propagator acts in a discrete Fourier basis, where each eigenmode corresponds to a spatial wave. The authors show that the diagonal state-recurrence operator in S4D-Lin is mathematically isomorphic to such a system: the eigenvalue spectrum imparts systematically structured decay rates and frequencies, resulting in coherent wave propagation on the ring topology.

Figure 2: The operator correspondence maps S4D’s recurrent layer onto a ring topology with structured connectivity and modal spectrum, supporting traveling waves distinguished by their eigenfrequency and decay profile.

Modal Decomposition and Traveling Wave Computation

Through explicit decomposition, the state at each timestep in the SSM can be represented as a superposition of traveling wave modes with complex amplitudes, each evolving independently under the diagonal propagator. The input signal is linearly projected into these modes, producing dynamic patterns that encode temporal features of the sequence.

The paper demonstrates that simple input classes (e.g., sinusoids of different frequencies plus noise) excite specific resonant modes of the system. Classification becomes linearly separable in the space of modal energies, with only a subset of modes providing high discriminatory power.

Figure 3: Amplitude dynamics of resonant modes (eigenfrequencies) sharply differentiate input classes, substantiating the claim that S4D's recurrent state acts as a spectral extractor of temporally encoded features.

End-to-End Operator Formalism and Nonlinear Effects

The core contribution is an explicit operator-level expression for the entire forward pass in S4D, from input encoding through linear recurrence and nonlinear activation to class prediction. The use of Carleman embeddings provides an analytical expansion of the effect of the nonlinearity (e.g., GELU) in the modal coordinate system. Each term in this expansion corresponds to increasing orders of interaction among traveling waves in the network.

The truncation of the Carleman series at low order (quadratic) already explains 94% of classification performance on a real-world signal processing benchmark, as compared to 83% for the purely linear (first-order) contribution. Full modeling of the nonlinearity yields perfect reproduction of the network’s observed classification output.

Figure 4: The operator expression reveals the contributions of individual modal and nonlinear interaction terms to classification accuracy; second-order (quadratic) Carleman expansion nearly saturates the network’s actual performance.

S4D Operators and Network Topology

The mathematical analysis extends to other diagonal SSM parameterizations, whose topological and spectral structures are visualized in terms of induced network connectivity. S4D-Lin’s spectrum supports uniform decay; alternative parameterizations (e.g., S4D-Inv, S4D-DFouT) generate distinct connectivity and phase structures, but remain amenable to the same modal and operator analysis.

Figure 5: Network topology and induced adjacency/phase structure for several diagonal SSMs are compared, illustrating the dependence of physical network realization on eigenvalue allocation.

Implications and Future Directions

This operator-theoretic framework yields significant implications:

• Interpretability: The forward computation is now mathematically transparent—the model output can be expressed analytically as a function of the input sequence and trained parameters, including how contributions are distributed across modes and nonlinear interaction orders.
• Architectural Design: Understanding the induced topology and modal structure allows principled design of SSMs with targeted spectral and temporal properties, facilitating extension to models with custom memory or input integration properties.
• Controllability & Safety: The explicit mapping from input to output in eigenbasis coordinates enables prospective output engineering and potential runtime intervention, supporting robust and verifiable deployments.
• Generalization Across Architectures: The correspondence applies broadly to diagonalizable SSMs, and the modal-Carleman approach accommodates varying nonlinearities and spectral allocations.

The findings establish that the core computational substrate of diagonal SSMs is the selective excitation and interaction of traveling wave modes, not high-dimensional abstract state vectors. Nonlinear activations serve to couple and modulate these modes, realizing complex temporal discrimination in sequence modeling.

Conclusion

This paper delivers a detailed operator-level mechanistic account of computation in SSMs for sequence and language modeling. Through the lens of nonlinear oscillator networks, the work makes explicit how recurrent wave propagation, modal selectivity, and nonlinear interactions combine to yield effective and interpretable sequence classifiers. These results lay the foundation for mathematically guided design and analysis of efficient, transparent neural sequence models, with direct pathways toward enhanced interpretability and controllability in future AI systems.

(2604.20595)