---
title: Zero-Memory-Overhead Direct Convolutions
url: https://www.emergentmind.com/topics/zero-memory-overhead-direct-convolutions
type: topic
---

# Zero-Memory-Overhead Direct Convolutions

Zero-memory-overhead direct convolution refers to a class of algorithms for evaluating convolution operations—central to deep neural networks—without the extra memory footprint characteristic of traditional “indirect” approaches such as im2col+GEMM. These direct strategies aim to maximize arithmetic intensity, exploit vectorization, and efficiently utilize hierarchical cache while entirely eliminating large intermediate packing buffers. The resulting performance superiority is especially pronounced in memory-constrained or bandwidth-limited environments such as embedded CPUs and for large-scale inference on general-purpose architectures [2411.15659, 2303.04739, 1809.10170, 1808.05567].

## 1. Motivation and Limitations of Indirect Convolutions

Indirect convolution methods, most famously im2col+GEMM, transform the convolutional operation into a matrix multiplication by “lowering” patches of input tensors into columns, yielding a buffer of shape $(c_{in} \cdot k_h \cdot k_w) \times (h' \cdot w')$ for input of shape $(c_{in} \times h \times w)$ and kernel of $(c_{out} \times c_{in} \times k_h \times k_w)$, where $h' = h - k_h + 1$, $w' = w - k_w + 1$. This duplication of overlapping regions inflates memory requirements by a factor of $c_{in} k_h k_w$, significantly exceeding the original input tensor size and often exceeding available L2/L3 cache capacity [2411.15659].

Cache inefficiency emerges from the irregular aspect ratio (“skinny” matrices) of the im2col buffer, which leads to under-utilization of cache lines and poor prefetcher performance. Each matrix column is read by GEMM once, whereas output storage is random in cache, further reducing locality and prefetch efficiency. These factors contribute to bandwidth-bound performance, particularly on CPU architectures [2411.15659, 1809.10170, 2303.04739].

## 2. Fundamentals of Zero-Memory-Overhead Direct Convolutions

Direct convolution approaches sidestep the im2col buffer by computing output elements directly from input and filter tensors, maintaining strict locality and avoiding any large intermediate allocations. Typical strategies include reordering loop nests, smart register/caching blocking, vectorized microkernels, and “on-demand” packing of small tiles, always restricted to cache-resident working sets [2411.15659, 2303.04739, 1809.10170].

### Mathematical Formulation

For input tensor $X \in \mathbb{R}^{c_{in} \times h \times w}$ and kernel $W \in \mathbb{R}^{c_{out} \times c_{in} \times k_h \times k_w}$, the output is:
\[
Y_{c_{out},i,j} = \sum_{c_{in}=1}^{c_{in}} \sum_{u=1}^{k_h} \sum_{v=1}^{k_w} X_{c_{in},\,i+u-1,\,j+v-1} \times W_{c_{out},\,c_{in},\,u,\,v}
\]
A direct approach preserves this structure—emphasizing strategic buffering (registers, per-tile cache) rather than global packing [2411.15659, 1809.10170].

## 3. Algorithmic Techniques and Implementation

Several concrete algorithmic paradigms have emerged, with differences in how they block, vectorize, and tile the convolutional computation:

### SMM-Conv: Scalar Matrix Multiplication with Zero Packing

In “SMM-Conv,” the key innovation involves decomposing the convolution into a sequence of scalar–matrix multiplications over contiguous “slices” of the input tensor. Rather than packing all patches, a small buffer $B \in \mathbb{R}^{h \times w'}$ holds each sliced region. For each vertical and horizontal kernel offset, valid $h' \times w'$ submatrices are multiply-accumulated, requiring only pointer offsetting and exploiting fast SIMD FMA instructions for the inner width loop. Overall, the buffer $B$ is reused $k_h k_w$ times before refilling, guaranteeing extremely low memory overhead (essentially $O(hw')$) [2411.15659].

**Key steps:**
- For each input channel and horizontal offset, extract $T^c_v = X_{c,1:h,v:v+w'-1}$ into $B$.
- For each vertical offset $u$, pointer advance into $B$ yields $S^c_{u,v}$.
- Each $S^c_{u,v}$ is multiplied by the scalar kernel weight $W_{c_{out},c,u,v}$ and accumulated into $Y$.

### CSA/CSO (SConv) and Vector-Based Packing

“SConv” introduces a code-generation and tiling analysis pipeline: Convolution Slicing Analysis (CSA) systematically determines the maximal tile sizes fitting in L1 (and recursively, L2/L3) cache. Its Slicing Optimization (CSO) stage emits a deeply nested loop macro-kernel with explicit packing for only those input/filter tiles about to be used, always under the L1 threshold ($\leq 28$ KB typical). Vector-Based Packing (VBP) leverages hardware shift/permutation (e.g. AVX-512, POWER10 VSX) to exploit overlap between neighboring windows for stride-1 convolutions, further minimizing redundant data movement and eliminating the need for large packed matrices [2303.04739].

### Loop/Blocking/Raster Strategies

Traditional direct convolution algorithms optimize via:
- Spatial blocking: Output tiles (height $\times$ width) that fit in L1/L2 [1809.10170].
- Register blocking: Output channel bursts $\approx$ SIMD width.
- Depth-first kernel application: Inner loops over kernel/channel are unrolled for maximum FMA utilization.
- Data layout: “Channels-last” and blocked layouts enable unit stride for memory-efficient vectorization [1808.05567].

Implementation is accomplished either manually (hand-written SIMD microkernels) or via dynamic code generation (JIT), with inner-most loops engineered to reside in registers or L1 at all times [1808.05567].

## 4. Memory Complexity, Arithmetic Intensity, and Roofline Analysis

### Memory Footprint

The memory requirement is restricted to the input, kernel, and output tensors, plus (at most) cache-resident working tiles:
- im2col+GEMM: $O(c_{in} k_h k_w h'w' + c_{out} h'w')$
- Zero-overhead direct: $O(h w' + c_{out} h' w')$ (SMM-Conv) or $O$(small L1-sized buffers + outputs) (SConv)

For $h' \approx h$, the im2col memory exceeds the zero-overhead direct approach by a factor of $c_{in} k_h k_w$ or more, with typical savings exceeding $10\times$ on modern convolutional layers [2411.15659, 2303.04739].

### Arithmetic Intensity

Direct methods maximize reuse:
\[
I = \frac{2 c_{out} c_{in} k_h k_w h' w'}{ \text{Bytes loaded/stored} }
\]
Direct methods approach roofline limits, being either compute- or bandwidth-bound depending on architectural ratios. Empirical intensity in, e.g., ResNet-50 3$\times$3 convolution can reach $I \approx 20$ FLOP/byte [1808.05567].

## 5. Experimental Results and Performance

The elimination of memory overhead translates into demonstrable performance gains:

| Model   | im2col+GEMM | SMM-Conv | SConv (mean) | Direct SIMD JIT | Observed Speedup      |
|---------|-------------|----------|--------------|-----------------|-----------------------|
| AlexNet | 0.4608s     | 0.1348s  | N/A          | N/A             | $3.4\times$ [2411.15659] |
| VGG     | 2.3670s     | 1.3535s  | N/A          | N/A             | $2.1\times$ [2411.15659] |
| YoloV3  | 0.4478s     | 0.2889s  | N/A          | N/A             | $2.0\times$ [2411.15659] |
| ONNXNet (x86 mean) | N/A  | N/A      | $9$–$25\%$ faster | N/A             | $2.0$–$3.9\times$ packing [2303.04739] |
| Skylake-SP | 1.2 TF/s (im2col) | N/A | N/A  | 3.1 TF/s (direct) | $2.6\times$ [1808.05567] |

Direct convolution consistently reduces both intermediate memory footprint and DRAM traffic, delivering $2$–$3\times$ end-to-end speedups for inference tasks and maintaining high scaling even as thread count increases [2411.15659, 1809.10170, 2303.04739, 1808.05567]. Removal of the im2col stage increases the relative benefit as FMA throughput grows (e.g., on POWER10 MMA) [2303.04739].

## 6. Practical Considerations and Hardware Aspects

### Cache Optimizations

Zero-memory-overhead approaches structure loops and tile sizes to guarantee that transient working sets reside in L1 or L2. The compact buffer or register allocations are reused $k_h k_w$ (or more) times before refill, maximizing locality [2411.15659].

### Multi-threading

Task parallelism is exposed in the output-channel, batch, or spatial tile dimensions. Thread-private buffers avoid false sharing, and no locks are required since output slices are disjoint. Scaling remains nearly linear until core-saturation [2411.15659, 1809.10170].

### ISA Extensions and Vectorization

Advanced SIMD instructions (e.g., FMA, vector shifts) are leveraged for inner-loop fusion and input-tile packing, with special handling for stride-1 when hardware supports vector-register serial shifting (e.g., POWER10 VSX, x86 AVX-512). Code generation can elide all loop-boundary checks and branch overhead [2303.04739, 1808.05567].

## 7. Limitations, Extensions, and Open Problems

Zero-memory-overhead direct convolution is most effective under:
- Inference scenarios (filter repacking is trivial at compile time; training requires on-the-fly tiling) [2303.04739].
- Moderate to large output channel counts (for sufficient register and FMA utilization) [2411.15659].
- Stride-1 or small strides (for optimal VBP use); stride$>$1 requires scalar loads or microtiles [2303.04739].

Current SMM-Conv and SConv algorithms are written for “valid” convolution; extension to “same” padding involves lightweight zero-insertion during extraction. Hybridization with Winograd or FFT may be optimal for large kernels or highly compute-bound layers. Generalization to GPU architectures, backward-pass convolutions, and autotuning of blocking parameters remain active areas of investigation [2411.15659, 2303.04739, 1809.10170].

## References

- "SMM-Conv: Scalar Matrix Multiplication with Zero Packing for Accelerated Convolution" [2411.15659]
- "Advancing Direct Convolution using Convolution Slicing Optimization and ISA Extensions" [2303.04739]
- "High Performance Zero-Memory Overhead Direct Convolutions" [1809.10170]
- "Anatomy Of High-Performance Deep Learning Convolutions On SIMD Architectures" [1808.05567]

Source: https://www.emergentmind.com/topics/zero-memory-overhead-direct-convolutions