Custom Approximate Convolution Layer

• Custom Approximate Convolution Layer is a module that computes multidimensional convolutions approximately by leveraging low-rank tensor decompositions and advanced sampling strategies.
• It employs Fourier transforms and cross approximation techniques to reduce computational complexity while preserving accuracy in the frequency domain.
• The method demonstrates practical benefits in computational physics and chemistry, offering scalable performance for grid-based simulations and high-dimensional problems.

A custom approximate convolution layer is a computational module that computes multidimensional convolutions approximately, rather than exactly, by leveraging low-rank tensor formats and advanced sampling strategies to reduce computational complexity. The “cross-conv” algorithm, as introduced in the context of multidimensional tensor computations, exemplifies such a layer by combining Fourier domain transforms, cross approximation methods, and low-rank tensor decompositions to achieve efficient, scalable, and accurate approximate convolution, especially in high-dimensional settings (Rakhuba et al., 2014).

1. Cross-Conv Algorithm: Frequency Domain and Cross Approximation

The cross-conv approach replaces direct convolution—costly for high-dimensional tensors—with a sequence of transformations and approximations that both lower resource requirements and control output error. The classical spatial convolution

$(f * g)(x) = \int_\mathbb{R}^d f(y)g(x-y)dy$

is, after discretization, recast by first embedding the tensors into circulant form and then applying the multidimensional discrete Fourier transform (FFT), yielding

$$\tilde{w} = \mathcal{F}^{-1}\left( \mathcal{F}(c_g) \circ \mathcal{F}(q_f)\right),$$

where $c_g$ is the circulant extension of the kernel $g$, $q_f$ is the zero-padded signal (both as tensors), $\mathcal{F}$ is the FFT and $\circ$ indicates elementwise multiplication.

A key innovation is the use of cross approximation directly on the elementwise (Hadamard) product in the frequency domain. Instead of evaluating the entire product that would lead to a “rank explosion” in any low-rank tensor representation, only a carefully chosen subset of tensor entries is computed, sufficient to recover an approximate low-rank structure within a user-specified tolerance. This mechanism yields a compressed representation of the product, avoiding full-rank computations.

This procedure stands in contrast to conventional FFT-based convolution (complexity $O(n^d\log n)$) and to other low-rank approaches that form full products followed by rank truncation—both of which are costlier in terms of operations and memory footprint.

2. Low-Rank Tensor Formats and Preservation in FFT

The custom approximate convolution layer operates intrinsically in several SVD-based low-rank tensor formats, notably:

• Tucker format: $$A(i_1, ..., i_d) = \sum_{\alpha_1,...,\alpha_d} G(\alpha_1, ..., \alpha_d) U_1(i_1, \alpha_1)\cdots U_d(i_d, \alpha_d)$$, where $G$ is a core tensor and $U_k$ are factor matrices.
• Tensor Train (TT) format: $$A(i_1,...,i_d) = \sum_{\alpha_0,...,\alpha_d} G_1(\alpha_0, i_1, \alpha_1)\cdots G_d(\alpha_{d-1}, i_d, \alpha_d)$$, with boundary conditions $r_0 = r_d = 1$.
• Hierarchical Tucker (HT) format: A binary-tree based generalization, efficient for very high-dimensional problems.

Fourier transforms applied separately to each mode (tensor factor) preserve the tensor rank, which is critical: the structural advantages of low-rank representations are not lost under frequency-domain operations. Thus, after the FFT step, the tensor’s compressed format remains intact, and cross approximation can be directly applied to the frequency-domain product without expansion in intermediate ranks.

3. Computational Complexity and Resource Analysis

The cross-conv algorithm offers superior rank-dependence in its computational complexity compared with prior methods:

• Skeleton decomposition (2D matrices): $O(nr^2 + nr\log n)$,
• Tucker format in 3D (with Schur–Cross3D variant): $O(nr^2 + r^4 + nr\log n)$,
• TT-format (d-dimensional tensors): $O(dnr^3 + nr^2\log n)$,
• HT or extended TT: $O(dnr^2 + dr^4 + nr\log n)$.

Unlike elementwise multiplication in low-rank tensor algebra, which would “square” the rank and significantly increase the number of tensor parameters, the cross approximation selectively samples only so many entries as there are effective tensor parameters; i.e., the sample complexity is linear in the number of SVD parameters. This is a fundamental advantage for large-scale, high-dimensional settings and is particularly beneficial for moderate tensor ranks, where alternatives such as QTT (quantized TT) can be less practical due to larger constant factors in asymptotic scaling.

4. Error Control and Approximation Guarantees

Approximation error control in the cross-conv method is achieved via direct control of the backward error in the frequency domain:

$$\frac{\| \Delta\Theta \|}{\| \Theta \|} = \delta \implies \frac{\| \Delta\tilde{w} \|}{\| \tilde{w} \|} = \delta$$

where $\Theta$ is the tensor in the frequency domain. The invariance of the FFT under unitary transformation ensures that the error introduced by cross approximation in the frequency domain maps directly (with the same norm) to the spatial domain convolution result. This link enables precise setting of the allowed approximation tolerance ($\delta$), ensuring that prescribed accuracy targets are met.

5. Applications in Computational Physics and Chemistry

The custom approximate convolution layer is designed for settings where multidimensional convolution is central, particularly when both the signal and the kernel admit accurate low-rank representations:

• Three-dimensional Newton potential: $V(x) = \int f(y)/\|x-y\| dy$, ubiquitous in electronic structure computation.
• Hartree–Fock (HF) and Kohn–Sham (KS) equations: Repeated convolutions with Newton and Yukawa kernels are required for Coulomb and exchange potentials; the cross-conv approach, when used as a grid-based “black-box” subroutine, circumvents basis-set errors inherent to classical quantum chemistry solvers.

Practical experiments reported for grids with $n\sim 10^3-10^4$ points indicate that the cross-conv method achieves faster run times than matrix-by-vector or QTT-based alternatives, with observed Tucker ranks remaining moderate (e.g., low 20s to below 100 per mode) even for large molecular systems. Empirically, the method handled grids of size $n^3=5121^3$ efficiently, with convolution execution times in the range of 1–20 seconds, and provided solutions to chemical accuracy in grid-based HF simulations ($\sim 10^{-6}$ error, $n$ up to 4096 per mode).

Beyond physics and chemistry, the cross-conv framework is applicable to any multidimensional convolution with low-rank structure, including in kinetic equations (Smoluchowski, population balance), signal processing, and quantitative finance.

6. Key Mathematical Formulations

The main mathematical constructs include:

• Discrete convolution on an $n^d$ grid:

$$w_{\mathbf{j}} = \sum_{\mathbf{i}} f_{\mathbf{i}}\cdot g_{\mathbf{i}-\mathbf{j}}$$

• Fourier-based embedding:

$$\tilde{w} = \mathcal{F}^{-1}(\mathcal{F}(c_g)\circ \mathcal{F}(q_f))$$

• Tucker format tensor representation:

$$A(i_1,\ldots,i_d) = \sum_{\alpha_1,\ldots,\alpha_d} G(\alpha_1,\ldots,\alpha_d) U_1(i_1,\alpha_1)\cdots U_d(i_d,\alpha_d)$$

• Cross approximation tolerance control:

$$\|\Delta\Theta\|/\|\Theta\| = \delta \implies \|\Delta\tilde{w}\|/\|\tilde{w}\| = \delta$$

The above formulas underpin the sampling, compression, and error management strategies that define the custom approximate convolution layer’s computational workflow.

7. Numerical Validation and Empirical Observations

Empirical results from extensive numerical experiments validate the efficiency and reliability of the cross-conv approximate convolution layer:

• For 3D Newton potential calculations on grids up to $n\sim2^{18}$, with prescribed error $\varepsilon$ as low as $10^{-9}$, the cross-conv scheme was faster than alternative low-rank methods.
• In electronic structure benchmarks (methane, ethane, ethanol, nitroethane), Tucker rank compression enabled grid sizes as large as $n^3=5121^3$, with per-convolution times ranging from 1 to 20 seconds and preserved accuracy.
• Solution of Hartree–Fock equations with the cross-conv procedure achieved grid-based HF-limits within $10^{-6}$ error, using up to 4096 grid points per mode.

The ability to meet high-accuracy requirements and deliver reductions in computational time and storage, particularly for grid-based solvers, demonstrates the practical significance of the custom approximate convolution layer in scientific computing environments.

---

In conclusion, the custom approximate convolution layer—exemplified by the cross-conv algorithm—delivers efficient, scalable computation for multidimensional convolutions by integrating low-rank tensor techniques, frequency domain processing, and cross approximation. Its ability to adjust computational cost with respect to tensor rank while directly managing approximation error, paired with empirical performance on high-dimensional tasks, establishes it as a robust tool for applications that require repeated high-accuracy convolution on large grids or in high-dimensional tensor-product spaces (Rakhuba et al., 2014).