Implicit Padding in FFT Convolution

• Implicit padding in spectral convolution is a technique that integrates boundary conditions within FFT computations to avoid explicit zero-padding in memory.
• It leverages subtransform decomposition and residue indexing to achieve dealiased convolution while preserving operator norms and enhancing spectral accuracy.
• Implementations such as TurboFNO report up to 2.5× reduction in memory usage and 150% speedup, demonstrating practical efficiency for deep learning optimizations.

Implicit padding in spectral convolution refers to strategies for incorporating zero-padding, domain extensions, or signal boundary conditions directly into the fast Fourier transform (FFT) workflow without explicitly materializing the padded values in memory. This approach contrasts with explicit padding, where arrays are formally enlarged with appended zeros. Implicit padding is essential for dealiased convolution, precise operator norm estimation, and efficient large-scale implementations of spectral neural layers, especially in GPU contexts. The distinction between “circular” (FFT-based, wrap-around) and zero-padded (Toeplitz) convolution is central to understanding effects on spectral properties and implementation efficiency.

1. Theoretical Foundations: Circular Versus Linear Convolution

In FFT-based convolution, the most direct route—convolving two sequences $f$ and $g$ using the convolution theorem—naturally computes a circular convolution:

$$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$

This circular convolution arises from the periodic extension assumed by the DFT. Linear convolution, the standard in spatial or signal-processing settings (often with explicit zero-padding), is defined by:

$$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$

To compute linear convolution via FFT, one must ensure that wrap-around effects (aliasing) are precluded, typically by padding both $f$ and $g$ with sufficient zeros to $M \geq N_1 + N_2 - 1$. Failure to do so results in aliasing artifacts (Murasko et al., 2023, Bowman et al., 2010).

In higher dimensions (e.g. for images), FFT-based convolution similarly induces circular boundary conditions—implementing what is called implicit padding: the operation algebraically assumes that out-of-bounds references wrap around to the opposite border. In contrast, explicit spatial zero-padding extends the domain such that out-of-bounds values are literal zeros, leading to a block-Toeplitz operator (spatially) rather than the block-circulant matrix realized by the circular (FFT) variant (Delattre et al., 2024).

2. Implicit Padding: Algorithms and Practical Formulation

Classical Algorithmic Distinction

• Explicit zero-padding: Arrays are directly enlarged by appending zeros, and the FFT is performed on this extended array. This requires extra memory bandwidth and computational work—especially problematic in GPU or memory-constrained environments (Wu et al., 16 Apr 2025).
• Implicit padding: The DFT (and its inverse) is reorganized so that the structure of the transform inherently ignores or bypasses the regions known to be zero, obviating the need to materialize the zeros in memory. This is accomplished by decomposing the FFT into smaller subtransforms operating only on the nontrivial domain, and recombining results via Cooley–Tukey or residue-based indexing (Bowman et al., 2010, Murasko et al., 2023).

Algorithmic Realization

Suppose the input has length $L$, desired output/padded length $M$, and subtransform block size $m$. For 1D:

• Partition input and output indices $g$0, $g$1.
• Compute inner DFTs on each residue block without accessing zero-padded data.
• In higher dimensions, the implicit strategy is applied recursively along each axis (Murasko et al., 2023).

In advanced neural operator implementations, such as TurboFNO, the implicit padding (and mode pruning) is realized not by copying but by conditional reads and writes inside a single GPU kernel: memory accesses are masked so that only in-range elements of the original array are accessed, and zeroes are injected on-the-fly for padded indices (Wu et al., 16 Apr 2025).

3. Effects of Implicit Padding on Operator Norms and Spectral Analysis

FFT-based convolution, due to its circular wrap-around, affects the operator norm and spectral characteristics of the layer. The singular value spectrum of the block-circulant (circular) convolution operator differs from that of the block-Toeplitz (zero-padded) operator, especially at higher frequencies near $g$2.

A rigorous connection between the zero-padded spectral norm $g$3 and the circular spectral norm $g$4 can be established. The main result is a closed-form upper bound (Delattre et al., 2024):

$g$5

where $g$6 and $g$7 is the $g$8-th Gram iterate of the Fourier block $g$9 of $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$0. As $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$1 increases, the bound tightens and converges to $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$2. For typical image sizes with $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$3, the corrective factor $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$4 approaches unity, rendering “implicit padding” error negligible in practical settings.

For computational efficiency, Gram-iteration in the frequency domain provides a deterministic, differentiable upper-bound for the spectral norm—enabling spectral rescaling layers for robustness in deep learning frameworks (Delattre et al., 2024).

4. Implicit versus Explicit Padding: Memory, Performance, and Implementation

Explicit padding increases memory usage and computation, especially in high-dimensional convolution (2D, 3D) (Murasko et al., 2023):

• Explicit: Padding and FFTs scale with the size of the zero-padded domain.
• Implicit: Only nontrivial data are transformed, and memory for the padded regions is avoided; hybrid schemes blend the advantages by explicit paddings up to block size and implicit extension thereafter.

Benchmarks (Murasko et al., 2023, Bowman et al., 2010):

• For 2D complex convolution, hybrid and implicit dealiasing achieve speedups of $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$5 over explicit padding for large input sizes, and reduce memory usage by up to $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$6 in $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$7 dimensions.
• In GPU architectures, implicit padding and mode-pruning (as in TurboFNO) allow FFT, GEMM, and iFFT steps to be fused in a single kernel—eliminating intermediate memory traffic and achieving up to 150% speedup versus standard library-based implementations (e.g., PyTorch’s FNO reference) (Wu et al., 16 Apr 2025).

Approach	Memory Usage	Speedup vs Explicit	Comments
Explicit padding	Maximal	Baseline	All zeros materialized
Implicit padding	Minimal	Up to $$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$8	Nontrivial data only
Hybrid dealiasing	Intermediate	$$(f\circledast g)_n = \sum_{m=0}^{M-1} f_m\,g_{(n-m)\bmod M}, \quad n=0,\dots,M-1.$$9	Block-level explicit+implicit

5. Gram Iteration and Certified Norm Bounds under Implicit Padding

The Gram iteration provides a scheme for certifiably bounding the operator norm for both circular (implicit) and zero-padded (explicit) convolutions (Delattre et al., 2024):

• For a matrix $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$0, iterate $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$1.
• The sequence $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$2 converges quadratically to $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$3 and furnishes an explicit upper bound at every step.
• In the frequency domain (circular case), Gram iteration is applied blockwise to $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$4 FFT blocks, and the maximum over blocks gives the spectral norm estimate. For Toeplitz (zero-padded) convolution, Gram iteration proceeds as recursive spatial-domain self-convolutions.

Pseudocode (dense version, as provided): $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$6 This routine, and its frequency-domain parallelization, are GPU-friendly and deterministically ensure that spectral norm estimates for zero-padded convolution can be obtained without bias, corrected for the small “implicit padding” gap via the $$(f * g)_n = \sum_{m=0}^{N_1-1} f_m\,g_{n-m}, \quad 0 \leq n < N_1 + N_2 - 1.$$5 factor.

6. Applications: Spectral Neural Operators and High-performance Implementations

In neural operator models such as Fourier Neural Operators (FNOs), the convolution “Fourier layer” requires zero-padding, DFT, truncation in spectral space, and an inverse DFT. Traditionally implemented as distinct memory stages, these can be fused—using implicit padding—with mode pruning and matrix multiplications for high-performance inference and training.

TurboFNO achieves this by integrating implicit zero-padding and Fourier mode truncation into FFT/GEMM/iFFT stages within a single GPU kernel (Wu et al., 16 Apr 2025):

• Only in-range (unpadded) indices are loaded from data.
• FFTs are computed with pruning so that only active low-frequency modes are written or further operated upon.
• GEMM, iFFT, and final cropping are performed with shared-memory swizzling to guarantee bank-conflict-free access.

Reported results include up to 2.5× reduction in global memory traffic and end-to-end speedup of up to 150%, with full equivalence to explicit padded results.

7. Hybrid and Multidimensional Implicit Schemes

In multidimensional convolutions, hybrid dealiasing merges block-level explicit padding with implicit extension, optimizing for FFT plan size and memory re-use (Murasko et al., 2023):

• Each axis’ transform is performed with a tunable block size; explicit padding is applied to the smallest convenient block, then implicit techniques apply to the whole padded domain.
• Recursive subtransform decomposition enables further memory and computational savings.
• This approach generalizes across 1D–3D convolution, and is implemented efficiently in FFTW++.

Grid Size	Explicit (in-place)	Implicit (1/2)	Hybrid
512×512	100 ms	85 ms	55 ms
1024×1024	420 ms	350 ms	210 ms

Hybrid schemes are especially advantageous when the input dimensions are large or when optimal performance on a given FFT library (e.g., best-case “magic” sizes) is desired.

---

In conclusion, implicit padding in spectral (FFT-based) convolution generalizes the notion of boundary extension and dealiasing by eliminating explicit memory allocation for zero regions and tailoring computation to active data and modes. It is rigorously connected to the operator norms of convolutional layers, supports efficient upper-bounds for robustness via Gram iteration with minimal correction, and underpins state-of-the-art implementations in both classical signal processing and neural operators. Its practical significance spans from reduced memory footprint and higher throughput to strict certified robustness in deep learning applications (Delattre et al., 2024, Murasko et al., 2023, Bowman et al., 2010, Wu et al., 16 Apr 2025).