Tensorial Convolutions in Neural and Geometric Domains

• Tensorial Convolutions are an advanced framework for generalizing classical convolution to high-order tensors, enabling matrix-like operations through t-products and related algebraic constructions.
• They significantly enhance neural network design by reducing parameter counts and computational cost via fast Fourier-based methods and low-rank approximations.
• The approach extends to geometric domains and free probability, facilitating invariant filtering on spheres and efficient spectral analyses in complex tensor systems.

Tensorial convolution is a broad algebraic and computational framework for generalizing classical convolution operations to tensors of arbitrary order, supporting sophisticated multilinear signal processing, neural network design, and geometric analysis. This paradigm encompasses circulant-based tensor products, spherical harmonic generalizations, free probability analogs, and fast low-rank approximations, while connecting with practical computational algorithms in deep learning and physics.

1. Algebraic Foundations: The t-product and Multilinear Extensions

The t-product is a central construction for tensorial convolution in the context of third-order tensors. For a tensor $$\mathcal{A} \in \mathbb{R}^{\ell \times m \times n}$$, its t-product with $\mathcal{B} \in \mathbb{R}^{m \times p \times n}$ is defined by first forming a block-circulant matrix $\operatorname{bcirc}(\mathcal{A})$, unfolding $\mathcal{B}$, performing matrix multiplication, and folding the result. Specifically,

$$\mathcal{C} = \operatorname{fold}\left(\operatorname{bcirc}(\mathcal{A}) \cdot \operatorname{unfold}(\mathcal{B})\right)$$

Each frontal slice of the output is a circular convolution along the third mode:

$C^{(k)} = \sum_{i=1}^n A^{(i)} B^{(k-i+1 \mod n)}$

This operation renders tensors as $t$-linear operators, endowing them with matrix-like properties including compositional algebra, invertibility (where applicable), and spectral theory via block-circulant structures (Newman et al., 2018). The t-product is extensible via the $M$-product, encompassing any invertible transform $M$ (DFT, DCT, etc.) along the third mode, thus realizing different generalized convolutions.

In higher-order cases, the circular convolution product is defined for N-way arrays (t-scalars), forming a commutative ring under addition and convolution, serving as the basis for t-matrix algebra and tensor decompositions such as t-SVD and higher-order SVD variants (Liao et al., 2020).

2. Tensorial Convolutions in Neural Networks and Computational Acceleration

Tensorial convolution layers directly replace matrix-based linear transforms in neural networks. A typical layer operates on tensor-valued data $$\mathcal{X} \in \mathbb{R}^{\ell_j \times m \times n}$$ via

$$\mathcal{Y} = \sigma\left(\mathcal{W} * \mathcal{X} + \mathcal{B}\right)$$

with the t-product encoding correlations along the chosen tensor mode, dramatically compacting the parameter space and exposing multidimensional correlations otherwise occluded in flat representations (Newman et al., 2018).

Empirically, tensorial layers (e.g., t-NNs) require $O(n^3)$ parameters compared to $O(n^4)$ for standard fully-connected layers when operating on $n \times n$ images, and admit fast evaluation via independent per-slice matrix multiplications in Fourier space, cost $O(n^3 + n^2 \log n)$ with FFT. On MNIST and CIFAR-10, tensorial leapfrog networks attain competitive or superior accuracy with dramatically fewer parameters and layers.

For deep convolutional models, tensor-based techniques—especially low-rank CP decompositions—can dramatically lower computation, replacing an order-$O(N' M' H W L)$ convolution with $O(N' M' R (H + W + L))$ where filter rank $R \ll \min\{H, W, L\}$. This yields up to $4.5\times$–$7\times$ speedups with minimal accuracy degradation (Parkhomenko et al., 2017). Practical frameworks such as conv_einsum provide optimal scheduling of tensorial convolutions and contractions, achieving additional FLOP and memory savings via automated sequence parsing and dynamic-programming-based path selection, for a wide range of architectures and decompositions (Rabbani et al., 7 Jan 2024).

3. Generalized Convolution in Spherical and Geometric Domains

Tensorial convolution extends naturally to vector and tensor fields on curved manifolds, notably the sphere $S^2$. Scalar convolution is defined via zonal kernels $G:[-1,1] \to \mathbb{R}$,

$(G * f)(x) = \int_{S^2} G(x \cdot y) f(y) dS(y)$

with eigenstructure given by the spherical harmonics. For vector and rank-2 tensor fields, convolution must preserve tangent/normal structure and commute with differential operators. This is achieved by a spectral shift prescription on each Edmonds harmonic branch, yielding a block-diagonal convolution that acts as scalar convolution on appropriately shifted degrees (Aluie, 2018). The commutativity with surface-gradients, divergences, and curls enables exact coarse-graining of geophysical, astrophysical, or fusion-relevant PDEs, with Helmholtz decomposition providing an explicit scalar-potential filtering algorithm.

On homogeneous spaces and compact Lie groups, tensorial convolution is defined using right- and left-invariant parallel transport, supporting covariant operations amongst tensor fields, crucial for gauge-theoretic constructions such as "gauge $\times$ gauge $=$ gravity" mappings (Borsten et al., 2021).

4. Tensorial Convolutions in Free Probability and Random Tensor Theory

The framework of tensorial free convolution generalizes classical free probability to high-order tensors. On order-$p$ tensors, a notion of freeness is cast via vanishing mixed cumulants defined on $p$-regular combinatorial maps. The tensorial free additive convolution of compactly supported measures $\mu \oplus_p \nu$ satisfies

$$\kappa_n(\mu \oplus_p \nu) = \kappa_n(\mu) + \kappa_n(\nu)$$

for each order $n$, and has an $R$-transform additive in its argument. High-order semicircular and free Poisson laws (with moments given by Fuss–Catalan and Fuss–Narayana numbers, respectively) govern the limiting spectral distributions of Wigner and Wishart tensors, supporting central limit and Marčenko–Pastur-type theorems in the tensor regime (Bonnin, 3 Dec 2024).

5. Diagrammatic and Computational Representations

Graphical calculus provides a rigorous yet intuitive framework for tensorial convolution as index contraction. The fundamental rank-3 convolution tensor

$$\chi_{ijk}^{(\sigma_i \sigma_j \sigma_k)} = \begin{cases} 1 & \sigma_i i + \sigma_j j + \sigma_k k \equiv 0 \pmod{D} \ 0 & \text{otherwise} \end{cases}$$

encodes all standard convolution types (circular, cross-correlation, etc.), and supports generalized convolution theorems under Fourier transforms (Miatto, 2019). Implementation employs einsum-like code snippets, efficiently realizing contractions and convolutions over arbitrary index sets, enabling seamless translation between algebraic diagrams and high-performance computational kernels.

6. Practical Algorithmics and Applications

Tensorial convolution frameworks are now integral to efficient neural network design, multidimensional signal processing, geometric analysis, and random tensor theory. Algorithmic highlights include:

• Automated scheduling and execution of tensorized convolutional layers via meta-algorithms (conv_einsum), yielding 2–6× runtime and memory gains (Rabbani et al., 7 Jan 2024).
• Low-rank tensor factorization for convolutional filters (CP, Tucker, etc.), balancing accuracy trade-offs against computational gain (Parkhomenko et al., 2017).
• Spherical tensorial convolution supporting invariant filtering and operator commutation for spherical-domain PDEs (Aluie, 2018).
• Tensorial algebra facilitating generalized decompositions (t-SVD, t-PCA, HOSVD) for image analysis, compressive representation, and classification (Liao et al., 2020).
• Covariant tensor convolution constructions for gauge-theoretic gravity on group manifolds and homogeneous spaces (Borsten et al., 2021).
• Analytical tools for high-order tensor spectra, elucidating free convolution and non-commutative random matrix analogs (Bonnin, 3 Dec 2024).

The field continues to expand, with new connections to optimal computational graph scheduling, geometric deep learning, and advanced harmonic analysis.