---
title: Toeplitz Fully-Connected Layer
url: https://www.emergentmind.com/topics/toeplitz-fully-connected-layer-eedde8da-1547-47aa-ba72-76debccec0e5
type: topic
---

# Toeplitz Fully-Connected Layer

A Toeplitz Fully-Connected (Toeplitz-FC) Layer is a linear neural network layer in which the weight matrix is constrained to exhibit Toeplitz structure, i.e., its entries are constant along each diagonal. This construction generalizes classical fully-connected (FC) layers and provides an explicit connection to discrete convolutional operations, allowing for efficient implementation and a dramatic reduction in learnable parameters. Toeplitz-FC layers have found utility in both theoretical and practical contexts, including universal function approximation with depth, learning linear operators for time-series modeling, and as primitives in deep networks for dynamical systems analysis [1712.01252, 2410.02199].

## 1. Mathematical Definition and Structure

An $n \times n$ real Toeplitz matrix $T$ is defined such that $T_{ij} = t_{i-j}$ for $-(n-1) \leq i-j \leq n-1$, parameterized by the vector $t \in \mathbb{R}^{2n-1}$. In code, this is typically stored using an offset to index from $t_0$ (main diagonal) out to $t_{\pm(n-1)}$ (extremal diagonals), so $T_{i,j} = t[(i-j) + (n-1)]$ [2410.02199]. This parameterization reduces the number of model parameters from $n^2$ to $2n-1$, providing a highly parameter-efficient alternative to standard dense FC layers.

The core forward computation is a linear transformation $y = T x$, with
$$
y_i = \sum_{j=1}^n T_{ij} x_j = \sum_{j=1}^n t_{i-j} \, x_j.
$$
This is mathematically equivalent to a 1D discrete convolution between $t$ and $x$ with appropriate zero-padding, where $t$ acts as the convolution kernel.

## 2. Link to Convolutional Layers and Equivalence

Toeplitz-FC layers are mathematically equivalent to 1D convolutional layers. In the *valid* convolution setting, for input $x \in \mathbb{R}^n$ and kernel $k$ of size $m$, the linear operation $y_j = \sum_{i=0}^{m-1} k_i\, x_{j+i}$ can be formulated as a matrix multiplication $y = W x$, where $W \in \mathbb{R}^{n-m+1 \times n}$ is banded Toeplitz: $W_{j, j+i} = k_i$ for $i = 0, \dots, m-1$ and zeros elsewhere [1712.01252].

Boundary handling is determined by the convolution type:
- *Valid* convolution: $W \in \mathbb{R}^{n-m+1 \times n}$, no padding.
- *Same* convolution: Input is padded with $P = \lfloor m/2 \rfloor$ zeros, $W \in \mathbb{R}^{n \times (n+2P)}$.
- *Full* convolution: Padding with $m-1$ zeros on each side, $W \in \mathbb{R}^{(n+m-1) \times (n+2(m-1))}$.

Many neural network frameworks (e.g., via im2col + GEMM) internally convert convolution to this Toeplitz matrix-multiplication form for computational efficiency [1712.01252].

## 3. Parameter Efficiency and Expressive Capacity

The Toeplitz-FC layer requires $2n-1$ parameters for $n$-dimensional input/output, versus $n^2$ for dense FC. Although this ties weights along each diagonal and limits representable linear maps to those expressible as a Toeplitz structure, recent theoretical advances establish that arbitrary linear operators can be approximated arbitrarily well by compositions (products or exponentials) of Toeplitz matrices.

Specifically, the following universality property holds: for any $B \in \mathbb{C}^{N \times N}$ and any $\epsilon > 0$, there exist $J$ and Toeplitz-derived matrices $L_1, \dots, L_J$ such that
$$
B \approx e^{L_1} e^{L_2} \cdots e^{L_J}
$$
within operator norm $\epsilon$ [2410.02199]. Key results supporting this include: any $B \in \mathbb{C}^{N \times N}$ can be factored into at most $|N| + 1$ Toeplitz matrices; the Lie algebra generated by Toeplitz and diagonal matrices spans the full space; and every invertible matrix can be written as a product of exponentials within this algebra.

Thus, stacking Toeplitz-FC layers (potentially with nonlinearities) recovers the expressive universality of dense FC networks.

## 4. Backpropagation and Efficient Computation

Given a forward pass $y = T x$ and loss $\mathcal{L}(y)$, the gradient with respect to the Toeplitz parameter $t_k$ is
$$
\frac{\partial \mathcal{L}}{\partial t_k} = \sum_{i, j : i-j = k} \frac{\partial \mathcal{L}}{\partial y_i}\, x_j.
$$
This can be computed as a cross-correlation between the loss gradients and input. With batch processing, this becomes a convolution operation between the reversed input and the per-sample gradients, paralleling efficient gradient computations in standard convolutional layers [2410.02199].

The gradient with respect to $x$ is another convolution:
$$
\frac{\partial \mathcal{L}}{\partial x_j} = \sum_{i=1}^n \frac{\partial \mathcal{L}}{\partial y_i} \, t_{i-j}.
$$

These structures allow leveraging deep learning library primitives (e.g., PyTorch’s `conv1d`) for both the forward and backward passes, yielding computational complexity $O(n \log n)$ (FFT-based) or $O(nm)$ (direct) for input of length $n$ and kernel of size $m$ [2410.02199].

## 5. Universal Approximation and Theoretical Foundations

The universality theorem for Toeplitz-FC layers states that the set of matrix exponentials of Toeplitz matrices is dense in the set of linear operators over $L_0^2(\mathbb{T}^d)$. Specifically, for any finite-dimensional subspace (Fourier-index set $N$), and any target $f$, there exist layered Toeplitz operators $L_j$ such that the propagated vector approximates $f$ to within any prescribed $\epsilon$. The proof relies on results about Lie algebra generation, factorization into Toeplitz and diagonal components, and density of products of exponentials in the invertible group. This establishes that although a single Toeplitz-FC layer is not universal, arbitrary-depth architectures can approximate any linear map [2410.02199].

## 6. Practical Implementation and Fast Algorithms

Toeplitz-FC layers can be implemented in PyTorch or TensorFlow by parameterizing the $2n-1$ diagonals, reconstructing the convolution kernel, and performing the forward pass as a 1D convolution with suitable padding. Model instantiation, forward computation, and gradient backpropagation all leverage optimized convolution routines.

For settings that require exponentials of Toeplitz matrices, as in deep Koopman-layered models, fast computation of $y = \exp(L_j) u$ is achieved via Krylov subspace methods (Arnoldi, Lanczos). Each Krylov step involves Toeplitz mat-vec products, which reduce to $O(n \log n)$ convolution operations. In practice, only $m \ll n$ Krylov iterations are necessary, allowing orders-of-magnitude speedup compared to naive approaches [2410.02199].

## 7. Applications and Advanced Architectures

Toeplitz-FC layers provide a flexible yet structured modeling mechanism in time-series prediction, system identification, and analysis of nonautonomous dynamical systems. “Deep Koopman-layered” models use stacked, learnable Toeplitz-FC (and exponentials thereof) layers to capture complex, time-varying transitions; empirical results demonstrate competitive or superior performance for tasks such as Koopman operator eigenvalue estimation in nonautonomous settings [2410.02199]. The parameter tying and structural efficiency of Toeplitz-FC layers are leveraged for both model compression and theoretical analysis.

### Table 1: Parameter Comparison

| Layer Type       | Parameter Count         | Forward Structure          |
|------------------|------------------------|---------------------------|
| Dense FC         | $n^2$                  | General matrix multiply   |
| Toeplitz FC      | $2n-1$                 | Convolution (Toeplitz)    |
| Conv1D           | $m$ (per filter/kernel)| Sliding window convolution|

This comparison illustrates the parameter-efficiency gain of Toeplitz-FC layers compared to their dense fully-connected counterparts, while connecting their mechanism directly to convolutional operations [1712.01252, 2410.02199].

Source: https://www.emergentmind.com/topics/toeplitz-fully-connected-layer-eedde8da-1547-47aa-ba72-76debccec0e5