Toeplitz Fully-Connected Layer

• The paper introduces Toeplitz-FC layers, where weights are constant along diagonals, reducing parameters from n² to 2n-1 for efficient network design.
• It establishes the equivalence between Toeplitz-FC layers and 1D convolutions, allowing the use of optimized convolution routines for both forward and backward passes.
• The work demonstrates that stacking these layers can achieve universal approximation, effectively modeling time-series and dynamical systems.

A Toeplitz Fully-Connected (Toeplitz-FC) Layer is a linear neural network layer in which the weight matrix is constrained to exhibit Toeplitz structure, i.e., its entries are constant along each diagonal. This construction generalizes classical fully-connected (FC) layers and provides an explicit connection to discrete convolutional operations, allowing for efficient implementation and a dramatic reduction in learnable parameters. Toeplitz-FC layers have found utility in both theoretical and practical contexts, including universal function approximation with depth, learning linear operators for time-series modeling, and as primitives in deep networks for dynamical systems analysis (Ma et al., 2017, Hashimoto et al., 2024).

1. Mathematical Definition and Structure

An $n \times n$ real Toeplitz matrix $T$ is defined such that $T_{ij} = t_{i-j}$ for $-(n-1) \leq i-j \leq n-1$, parameterized by the vector $t \in \mathbb{R}^{2n-1}$. In code, this is typically stored using an offset to index from $t_0$ (main diagonal) out to $t_{\pm(n-1)}$ (extremal diagonals), so $T_{i,j} = t[(i-j) + (n-1)]$ (Hashimoto et al., 2024). This parameterization reduces the number of model parameters from $n^2$ to $2n-1$, providing a highly parameter-efficient alternative to standard dense FC layers.

The core forward computation is a linear transformation $y = T x$, with

$$y_i = \sum_{j=1}^n T_{ij} x_j = \sum_{j=1}^n t_{i-j} \, x_j.$$

This is mathematically equivalent to a 1D discrete convolution between $t$ and $x$ with appropriate zero-padding, where $t$ acts as the convolution kernel.

2. Link to Convolutional Layers and Equivalence

Toeplitz-FC layers are mathematically equivalent to 1D convolutional layers. In the valid convolution setting, for input $x \in \mathbb{R}^n$ and kernel $k$ of size $m$, the linear operation $y_j = \sum_{i=0}^{m-1} k_i\, x_{j+i}$ can be formulated as a matrix multiplication $y = W x$, where $W \in \mathbb{R}^{n-m+1 \times n}$ is banded Toeplitz: $W_{j, j+i} = k_i$ for $i = 0, \dots, m-1$ and zeros elsewhere (Ma et al., 2017).

Boundary handling is determined by the convolution type:

• Valid convolution: $W \in \mathbb{R}^{n-m+1 \times n}$, no padding.
• Same convolution: Input is padded with $P = \lfloor m/2 \rfloor$ zeros, $W \in \mathbb{R}^{n \times (n+2P)}$.
• Full convolution: Padding with $m-1$ zeros on each side, $W \in \mathbb{R}^{(n+m-1) \times (n+2(m-1))}$.

Many neural network frameworks (e.g., via im2col + GEMM) internally convert convolution to this Toeplitz matrix-multiplication form for computational efficiency (Ma et al., 2017).

3. Parameter Efficiency and Expressive Capacity

The Toeplitz-FC layer requires $2n-1$ parameters for $n$-dimensional input/output, versus $n^2$ for dense FC. Although this ties weights along each diagonal and limits representable linear maps to those expressible as a Toeplitz structure, recent theoretical advances establish that arbitrary linear operators can be approximated arbitrarily well by compositions (products or exponentials) of Toeplitz matrices.

Specifically, the following universality property holds: for any $B \in \mathbb{C}^{N \times N}$ and any $\epsilon > 0$, there exist $J$ and Toeplitz-derived matrices $L_1, \dots, L_J$ such that

$B \approx e^{L_1} e^{L_2} \cdots e^{L_J}$

within operator norm $\epsilon$ (Hashimoto et al., 2024). Key results supporting this include: any $B \in \mathbb{C}^{N \times N}$ can be factored into at most $|N| + 1$ Toeplitz matrices; the Lie algebra generated by Toeplitz and diagonal matrices spans the full space; and every invertible matrix can be written as a product of exponentials within this algebra.

Thus, stacking Toeplitz-FC layers (potentially with nonlinearities) recovers the expressive universality of dense FC networks.

4. Backpropagation and Efficient Computation

Given a forward pass $y = T x$ and loss $\mathcal{L}(y)$, the gradient with respect to the Toeplitz parameter $t_k$ is

$$\frac{\partial \mathcal{L}}{\partial t_k} = \sum_{i, j : i-j = k} \frac{\partial \mathcal{L}}{\partial y_i}\, x_j.$$

This can be computed as a cross-correlation between the loss gradients and input. With batch processing, this becomes a convolution operation between the reversed input and the per-sample gradients, paralleling efficient gradient computations in standard convolutional layers (Hashimoto et al., 2024).

The gradient with respect to $x$ is another convolution:

$$\frac{\partial \mathcal{L}}{\partial x_j} = \sum_{i=1}^n \frac{\partial \mathcal{L}}{\partial y_i} \, t_{i-j}.$$

These structures allow leveraging deep learning library primitives (e.g., PyTorch’s conv1d) for both the forward and backward passes, yielding computational complexity $O(n \log n)$ (FFT-based) or $O(nm)$ (direct) for input of length $n$ and kernel of size $m$ (Hashimoto et al., 2024).

5. Universal Approximation and Theoretical Foundations

The universality theorem for Toeplitz-FC layers states that the set of matrix exponentials of Toeplitz matrices is dense in the set of linear operators over $L_0^2(\mathbb{T}^d)$. Specifically, for any finite-dimensional subspace (Fourier-index set $N$), and any target $f$, there exist layered Toeplitz operators $L_j$ such that the propagated vector approximates $f$ to within any prescribed $\epsilon$. The proof relies on results about Lie algebra generation, factorization into Toeplitz and diagonal components, and density of products of exponentials in the invertible group. This establishes that although a single Toeplitz-FC layer is not universal, arbitrary-depth architectures can approximate any linear map (Hashimoto et al., 2024).

6. Practical Implementation and Fast Algorithms

Toeplitz-FC layers can be implemented in PyTorch or TensorFlow by parameterizing the $2n-1$ diagonals, reconstructing the convolution kernel, and performing the forward pass as a 1D convolution with suitable padding. Model instantiation, forward computation, and gradient backpropagation all leverage optimized convolution routines.

For settings that require exponentials of Toeplitz matrices, as in deep Koopman-layered models, fast computation of $y = \exp(L_j) u$ is achieved via Krylov subspace methods (Arnoldi, Lanczos). Each Krylov step involves Toeplitz mat-vec products, which reduce to $O(n \log n)$ convolution operations. In practice, only $m \ll n$ Krylov iterations are necessary, allowing orders-of-magnitude speedup compared to naive approaches (Hashimoto et al., 2024).

7. Applications and Advanced Architectures

Toeplitz-FC layers provide a flexible yet structured modeling mechanism in time-series prediction, system identification, and analysis of nonautonomous dynamical systems. “Deep Koopman-layered” models use stacked, learnable Toeplitz-FC (and exponentials thereof) layers to capture complex, time-varying transitions; empirical results demonstrate competitive or superior performance for tasks such as Koopman operator eigenvalue estimation in nonautonomous settings (Hashimoto et al., 2024). The parameter tying and structural efficiency of Toeplitz-FC layers are leveraged for both model compression and theoretical analysis.

Table 1: Parameter Comparison

Layer Type	Parameter Count	Forward Structure
Dense FC	$n^2$	General matrix multiply
Toeplitz FC	$2n-1$	Convolution (Toeplitz)
Conv1D	$m$ (per filter/kernel)	Sliding window convolution

This comparison illustrates the parameter-efficiency gain of Toeplitz-FC layers compared to their dense fully-connected counterparts, while connecting their mechanism directly to convolutional operations (Ma et al., 2017, Hashimoto et al., 2024).