Toeplitz Fully-connected Layer

• Toeplitz fully-connected layers are neural network components whose weight matrices have constant diagonals, ensuring translational invariance and reduced parameter count.
• They utilize fast Fourier transforms to perform convolution-like operations, significantly lowering computational complexity from O(n²) to O(n) while preserving approximation capabilities.
• These layers are ideal for tasks with shift-invariant data, such as time-series analysis and image processing, offering enhanced efficiency and implicit regularization.

A Toeplitz fully-connected layer is a neural network component in which the weight matrix is constrained to have a Toeplitz or block Toeplitz structure; that is, each diagonal (or block diagonal) contains constant elements. This architecture imposes strong weight-sharing and translational invariance properties analogous to those found in convolutional layers, leading to significant reductions in parameter count and computational complexity. Toeplitz (and general block Toeplitz) matrices enable fast computation of matrix–vector products, particularly by leveraging the Fast Fourier Transform (FFT), and possess universal approximation capabilities in certain deep operator learning frameworks. These properties make Toeplitz fully-connected layers of interest in neural architectures where memory and computational resources are constrained, or where data exhibit specific structural symmetries.

1. Definition and Mathematical Foundation

A Toeplitz matrix $T \in \mathbb{R}^{n \times n}$ satisfies $T_{i,j}=t_{i-j}$ for some sequence $\{t_k\}$. In the context of a fully-connected (dense) neural network layer, replacing the general dense weight matrix with a Toeplitz-structured one restricts the space of possible linear mappings to those with shift-invariant properties: $(Tx)_k = \sum_{l=0}^{n-1} t_{k-l}x_l,$ essentially implementing a discrete convolution. In higher-dimensional settings or when acting on tensors, the weight matrix can adopt a block Toeplitz with Toeplitz blocks (BTTB) structure, generalizing translational invariance along multiple dimensions.

This structure arises naturally when representing convolutional operations as matrix multiplications via the "im2col" or similar stretching procedures (Ma et al., 2017). In such cases, the so-called Toeplitz fully-connected layer is mathematically equivalent to imposing the "convolutional" pattern on the linear mapping, transforming the universal dense matrix into a highly sparse and structured object.

2. Toeplitz Structure in Neural Network Operations

The canonical example of Toeplitz structure within neural network computation stems from the equivalence between convolution and matrix multiplication. For a one-dimensional convolution, the resulting linear operator is Toeplitz, which allows the convolution to be realized as a multiplication between a structured matrix and input vector (Ma et al., 2017). In two dimensions, the corresponding matrix is block Toeplitz with Toeplitz blocks.

This realization underpins several important practical strategies:

Layer Type	Matrix Structure	Parameter Count
Fully-connected	Dense ($n^2$ independent values)	$O(n^2)$
Toeplitz	Toeplitz (1D; $2n - 1$ values)	$O(n)$
Block Toeplitz	BTTB (2D; see note)	$O(nd)$

(For block Toeplitz, $d$ is the block size; actual count depends on block configuration.)

The imposition of Toeplitz structure means the weight sharing is maximized, reducing over-parameterization and enhancing data efficiency for problems with symmetries. When applied, for example, in sequence modeling tasks or physical systems with translational invariance, this structure embeds an inductive bias aligned with the domain.

3. Parameterization, Computation, and Efficiency

The Toeplitz constraint dramatically reduces the parameter space, from $O(n^2)$ to $O(n)$. This reduction has substantial computational and storage implications:

• Fast Multiplication: Products between a Toeplitz matrix and a vector can be efficiently computed as a discrete convolution, allowing for $O(n\log n)$ complexity using FFT, a method directly inspired by related advances in electromagnetic array simulation (Åkerstedt et al., 5 Jun 2025).
• Block Toeplitz Generalization: For multi-dimensional or grouped inputs, block Toeplitz (potentially multilevel as in BTTB) layers offer a balance between expressivity and efficiency; memory allocation can scale as $O(N_x N_y)$ instead of $O(N_x^2 N_y^2)$.
• Implementation: Instead of forming an explicit dense matrix, Toeplitz layers typically store a generator vector (or vectors for blocks), with computation delegated to convolution or FFT routines in the forward and backward passes.

The structured sparsity introduced by Toeplitz weight matrices reduces overfitting risk and improves generalization, analogous to the role of convolutional kernels.

4. Universal Approximation Properties

The universal property of Toeplitz matrices states that any linear operator (within a finite-dimensional space) can be represented as a product of Toeplitz matrices. Specifically, for any $B \in \mathbb{C}^{N \times N}$, there exist Toeplitz matrices $A_1, \dots, A_R$ (with $R = N+1$) such that $B = A_1 A_2 \cdots A_R$ (Hashimoto et al., 3 Oct 2024). In the deep Koopman-layered model, each layer's parameterization via products of Toeplitz matrices enables the model to approximate arbitrarily general linear operators within the chosen basis (e.g., Fourier basis), granting the architecture universal approximation power for operator learning tasks.

This property is not limited to operator learning: deeply stacking Toeplitz (or block Toeplitz) layers can, in principle, approximate a wide family of linear mappings, provided the product sequence is sufficiently expressive and the domain structure is well-matched to the translational invariance constraint.

5. Applications and Empirical Results

Toeplitz fully-connected layers are applied primarily in contexts where data symmetries (e.g., time/space invariance) justify the constraints. Notable domains include:

• Convolution Equivalents: Any convolutional layer can be seen as a particular Toeplitz or block Toeplitz fully-connected layer (Ma et al., 2017).
• Deep Koopman-layered Models: For time-series analysis, especially in approximating Koopman operators in dynamical systems modeling. Here, Toeplitz-parameterized layers have demonstrated superior empirical accuracy in eigenvalue estimation tasks for both measure-preserving nonautonomous systems (accurately locating eigenvalues on the unit circle) and systems with damping/external forces (where time-dependent spectral shifts are revealed) (Hashimoto et al., 3 Oct 2024).
• Large-Scale Linear Systems: Adaptations from electromagnetic array analysis demonstrate orders-of-magnitude reduction in memory and computation for models involving very large matrices, which can be mapped to deep learning layers for efficient training and inference (Åkerstedt et al., 5 Jun 2025).
• Regularization: The weight sharing and sparsity serve as an implicit regularizer, potentially improving generalization on appropriate tasks.

6. Computational and Implementation Considerations

While Toeplitz layers offer large gains in memory and computation, several caveats are critical:

• Expressive Power Limitation: If underlying data are not translationally invariant, a Toeplitz layer may underfit. The restricted parameter set, while beneficial for regularization and efficiency, reduces the layer's capacity compared to full connectivity.
• Hardware and Software Integration: Efficient backpropagation through FFT-based layers, support for block structures, and compatibility with existing frameworks (such as PyTorch or TensorFlow) require nontrivial engineering (Åkerstedt et al., 5 Jun 2025).
• Training Algorithms: Krylov subspace methods may be required when the operator is constructed as a product or exponential of large Toeplitz matrices, allowing for efficient evaluation of expressions of the form $e^{\mathbb{L}_j}u$ during model training (Hashimoto et al., 3 Oct 2024).

A plausible implication is that, by exploiting these techniques, large networks could scale to problems previously bottlenecked by O($n^2$) computation or memory.

7. Limitations and Prospective Developments

Several limitations temper the universal adoption of Toeplitz fully-connected layers:

• Domain Matching: These layers are effective when the data or task naturally exhibits shift invariance; otherwise, enforcing this structure can be detrimental.
• Architecture Flexibility: General mappings and heterogeneous datasets may require hybrid architectures incorporating both Toeplitz and unstructured layers.
• Implementation Complexity: Ensuring numerically stable, hardware-accelerated, and differentiable FFT-based routines in end-to-end training pipelines remains nontrivial.

Emerging research explores adaptive parameterizations (e.g., mixtures of Toeplitz and low-rank updates), block-wise variants, and deeper theoretical understanding of tradeoffs between structure-imposed constraints and model capacity.

In summary, the Toeplitz fully-connected layer embodies a shift-invariant linear operator with significant advantages in efficiency and regularization. Its theoretical grounding, computational properties, and practical benefits in domains such as operator learning and large linear system modeling demonstrate the utility of structured linear mappings in modern deep learning architectures (Ma et al., 2017, Hashimoto et al., 3 Oct 2024, Åkerstedt et al., 5 Jun 2025).