ShortConv2D: Efficient Convolution Modules

• ShortConv2D layers are efficient convolution modules that compress kernel tensors via low-rank factorization to reduce parameters and FLOPs while preserving performance.
• Implementation methods like LightConv2D and BasisConv integrate as drop-in replacements in standard CNNs, using sequential convolutions with fixed basis or learnable projections.
• Empirical evaluations show up to 18× parameter reduction and 5× FLOP savings with minimal accuracy loss, demonstrating practical benefits for resource-constrained settings.

ShortConv2D layers constitute a family of parameter- and compute-efficient convolutional modules designed to replace standard Conv2D operations in deep neural networks (DNNs), particularly in settings demanding low memory footprint and fast execution. Methods branded as “ShortConv2D” share the core principle of compressing or factorizing the convolutional kernel tensor, thereby reducing the number of trainable parameters, storage, and floating-point operations (FLOPs), while retaining much of the original representational power. Prominent implementations include low-rank kernel factorizations (e.g., LightConv2D (Jha et al., 2021)), fixed-basis + learnable-projection architectures (BasisConv (Tayyab et al., 2019)), and efficient einsum-based contractions for small “short” kernels in tensor-network frameworks (Dangel, 2023). These approaches also have direct implications for theoretical analysis, backpropagation, and large-scale optimization.

1. Mathematical Foundations of ShortConv2D

ShortConv2D layers share a unifying formalism based on structurally reducing the dimensionality or learnable parameter space of convolutional kernels.

Let $$W \in \mathbb{R}^{k_h \times k_w \times C_{\text{in}} \times C_{\text{out}}}$$ denote the weight tensor of a standard Conv2D with spatial size $(k_h,k_w)$ and channel dimensions $(C_{\text{in}},C_{\text{out}})$. The conventional parameter count is $$N_{\text{std}} = k_h \cdot k_w \cdot C_{\text{in}} \cdot C_{\text{out}}$$.

1.1 LightConv2D Factorization

In LightConv2D (Jha et al., 2021), the weight tensor at each $(i,j)$ spatial offset is factorized into:

$$W(i,j) \approx W^1(i,j) \cdot W^2(i,j), \quad W^1(i,j) \in \mathbb{R}^{C_{\text{in}} \times k}, \quad W^2(i,j) \in \mathbb{R}^{k \times C_{\text{out}}}$$

Globally, two 4D tensors $W^1, W^2$ of rank $k$ replace the original, reducing the trainable parameter count to:

$$N_{\text{light}} = k_h k_w k (C_{\text{in}} + C_{\text{out}})$$

1.2 BasisConv/Basis-Projection ShortConv2D

In “BasisConv”/ShortConv2D (Tayyab et al., 2019), the kernel $W$ is first flattened and subjected to an explicit low-rank factorization $W' \approx B A$, with $B \in \mathbb{R}^{(k^2 C_{\text{in}})\times r}$ (the basis, typically fixed) and $A \in \mathbb{R}^{r\times C_{\text{out}}}$ (the learnable projection). This topology is implemented by a fixed $k \times k$ convolution with $r$ output channels, followed by a learnable $1 \times 1$ convolution.

The resulting parameter count is:

$$N_{\text{short}} = k^2 C_{\text{in}} r + r C_{\text{out}}$$

Compression ratios up to $18\times$ in parameter count and $5\times$ in FLOPs are observed with minimal accuracy loss (<3%) on challenging image classification tasks such as CIFAR-100 (Tayyab et al., 2019).

1.3 Tensor Network/Einsum View

From the tensor network (TN) perspective (Dangel, 2023), ShortConv2D is especially effective for small ($k_h$, $k_w$)—so-called “short” kernels—because sparse, structured contraction can be realized as a single efficient einsum operation, minimizing intermediate memory usage and maximizing compute reuse.

2. ShortConv2D Layer Construction and Integration

ShortConv2D layers are drop-in replacements for Conv2D within standard CNN architectures.

2.1 Layer Construction

• LightConv2D: Two Conv2D layers in sequence—first, Conv2D mapping $C_{\text{in}}$ to $k$ with given kernel size; second, Conv2D from $k$ to $C_{\text{out}}$ with identical kernel. Each is typically followed by BatchNorm and ReLU activation.
• BasisConv: First, a $k\times k$ convolution with $r$ filters and fixed weights (basis); second, a $1 \times 1$ convolution from $r$ to $C_{\text{out}}$ channels, which is fully learnable.

2.2 Practical Replacement

The replacement is architecture-agnostic and requires no custom layers or non-standard operations. This general protocol applies equivalently for pre-trained models (where basis is extracted via truncated eigendecompositions) and for models trained from scratch (where basis is random orthonormal, fixed).

2.3 Hyperparameters

• Rank $k$ (LightConv2D) or rank $r$ (BasisConv) governs the compression-accuracy trade-off.
• Empirical selection of $k=2$–$5$ balances parameter reduction and performance, with higher ranks advised for complex datasets or deep architectures (Jha et al., 2021, Tayyab et al., 2019).

3. Computational Complexity and Storage Benefits

ShortConv2D induces both parameter and FLOP savings.

Model	Parameters	FLOPs	Training Memory
Conv2D	$k_hk_wC_{\text{in}}C_{\text{out}}$	$2H_oW_ok_hk_wC_{\text{in}}C_{\text{out}}$	Proportional to parameter/gradient size
LightConv2D	$k_h k_w k (C_{\text{in}}+C_{\text{out}})$	$2H_oW_ok_h k_w k (C_{\text{in}}+C_{\text{out}})$	Reduced by factor of compression
BasisConv	$k^2 C_{\text{in}} r + r C_{\text{out}}$	$H_o W_o (C_{\text{in}} k^2 r + r C_{\text{out}})$	Reduced commensurately

Hardware implications are nuanced; for $k \ll C_{\text{in}}, C_{\text{out}}$, speed-ups are substantial, though decomposed operations may incur kernel-launch overhead on some hardware (Jha et al., 2021, Tayyab et al., 2019).

4. Optimization, Backpropagation, and Analytical Tractability

ShortConv2D layers offer additional advantages for optimization and theoretical analysis.

• Einsum-based forward and backward passes: Tensor network approaches unify the forward, input-gradient, and weight-gradient contractions into compact einsum expressions, efficiently supporting arbitrary stride, padding, and channel grouping (Dangel, 2023).
• Second-order (KFAC/GGN) curvature approximations: Methods such as KFAC become tractable due to the reduced dimensionality of parameter spaces in ShortConv2D. With short kernels, Kronecker factors are efficiently computed and inverted, yielding up to $4.5\times$ speed-up and less peak memory vs. im2col (Dangel, 2023).

5. Empirical Performance and Trade-offs

ShortConv2D layers have been validated extensively on standard image classification tasks. The following table summarizes the empirical accuracy and parameter count across several architectures and datasets (Jha et al., 2021, Tayyab et al., 2019):

Dataset	Method	Params	Test Acc.
MNIST	Conv2D	18,818	0.9887
	LightLayers ($k$=3)	6,135	0.9775
CIFAR-10	Conv2D	76,794	0.6882
	LightLayers ($k$=5)	20,557	0.5576
CIFAR-100	Conv2D	82,644	0.3262
	LightLayers ($k$=5)	21,367	0.0589
CIFAR-100	BasisConv/VGG16	18$\times$ fewer	$<2\%$ loss

On MNIST and Fashion-MNIST, compression of $2\times$–$3\times$ is achieved with only 1–3% reduction in accuracy. On CIFAR-10 and CIFAR-100, parameter and compute reductions remain (3–18$\times$), though higher rank or architectural depth increasingly becomes necessary to avoid greater accuracy loss.

6. Implementation Guidelines

ShortConv2D is compatible with mainstream deep learning frameworks. Example implementations:

TensorFlow/Keras (LightConv2D) (Jha et al., 2021):

import tensorflow as tf from tensorflow.keras import layers class LightConv2D(tf.keras.layers.Layer): def __init__(self, out_channels, kernel_size, rank, padding='same', use_bias=False): super().__init__() self.conv1 = layers.Conv2D(filters=rank, kernel_size=kernel_size, padding=padding, use_bias=use_bias) self.bn1 = layers.BatchNormalization() self.act1 = layers.ReLU() self.conv2 = layers.Conv2D(filters=out_channels, kernel_size=kernel_size, padding=padding, use_bias=use_bias) self.bn2 = layers.BatchNormalization() self.act2 = layers.ReLU() def call(self, x, training=False): x = self.conv1(x) x = self.bn1(x, training=training) x = self.act1(x) x = self.conv2(x) x = self.bn2(x, training=training) return self.act2(x)

PyTorch (BasisConv/ShortConv2D) (Tayyab et al., 2019):

import torch import torch.nn as nn class ShortConv2D(nn.Module): def __init__(self, C_in, C_out, kernel_size, rank_r, basis_weights=None, proj_weights=None, bias=None): super().__init__() self.basis = nn.Conv2d(C_in, rank_r, kernel_size=kernel_size, stride=1, padding=kernel_size//2, bias=False) if basis_weights is not None: self.basis.weight.data.copy_(basis_weights) for p in self.basis.parameters(): p.requires_grad = False self.proj = nn.Conv2d(rank_r, C_out, kernel_size=1, stride=1, padding=0, bias=(bias is not None)) if proj_weights is not None: self.proj.weight.data.copy_(proj_weights) if bias is not None: self.proj.bias.data.copy_(bias) def forward(self, x): z = self.basis(x) # stage 1: fixed basis y = self.proj(z) # stage 2: learnable projection return y

Initialization differs between pre-trained (truncated eigendecomposition for the basis) and scratch (random orthogonal basis) regimes; both attain low overhead and quick integration.

7. Theoretical Analysis and Future Directions

ShortConv2D layers facilitate both practical compression and theoretical analysis. In the TN/einsum framework, all major forward and backward contractions (including higher-order curvature approximations) reduce to a sequence of memory- and FLOP-efficient operations for small kernel sizes, making high-fidelity analytic work tractable (Dangel, 2023). A plausible implication is that continued advancements in tensor algebraic methods and structured parameter sharing could further improve the efficiency, generalization, and theoretical transparency of future convolutional architectures.

• LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification (Jha et al., 2021)
• BasisConv: A method for compressed representation and learning in CNNs (Tayyab et al., 2019)
• Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (Dangel, 2023)