Asymmetric Convolution Block (ACB)

• ACB is a CNN module employing parallel square and 1D convolutions to explicitly strengthen the kernel skeleton.
• It improves performance in tasks like image classification and super-resolution by enhancing feature extraction and robustness.
• After training, ACBs are fused into standard convolutions, offering enhanced representation with no additional inference overhead.

The Asymmetric Convolution Block (ACB) is a convolutional neural network (CNN) module designed to enhance the representational power of CNNs by explicitly strengthening the “skeleton” of square convolution kernels through the use of multiple branches with asymmetric (1D) convolutions. ACBs are architecture-neutral building blocks that introduce horizontal and vertical filter branches in parallel with traditional square kernels, enabling improved accuracy, robustness to spatial distortions, and efficient deployment with no inference-time overhead. ACBs have demonstrated empirical success across standard classification benchmarks and super-resolution tasks, providing performance improvements in scenarios ranging from image classification on ImageNet to single-image super-resolution.

1. Motivation and Architectural Principles

Standard CNN architectures typically employ square kernels (e.g., $3 \times 3$), which emphasize the central skeleton—comprising the middle row and column—more than the corners. Most architectural advances in recent years (e.g., ResNet, DenseNet, Inception) have focused on macro-level connectivity or channel attention, not on the microstructure of kernel weights. ACB addresses this gap by incorporating additional parallel branches with “horizontal” $(1 \times d)$ and “vertical” $(d \times 1)$ kernels, thereby explicitly enhancing the central skeleton during training without incurring additional inference cost. This strategy is strictly architecture-neutral, requiring no changes to the model’s global structure or the introduction of new hyperparameters or loss terms (Ding et al., 2019).

In low-level vision tasks, such as single-image super-resolution (SISR), similar principles apply. Networks composed of square kernels treat all spatial locations equivalently, potentially diluting salient edge or texture signals. Interleaving asymmetric convolutions within each block amplifies the contributions of local salient features along horizontal and vertical axes with minimal parameter overhead (Tian et al., 2021).

2. Mathematical Description and Block Construction

The ACB comprises three parallel convolutional branches:

• A square $d \times d$ convolution and batch normalization (BN) branch,
• A horizontal $1 \times d$ convolution and BN branch,
• A vertical $d \times 1$ convolution and BN branch.

Given input $M \in \mathbb{R}^{U \times V \times C}$, the three parallel filters $F \in \mathbb{R}^{d \times d \times C}$, $F_h \in \mathbb{R}^{1 \times d \times C}$, and $F_v \in \mathbb{R}^{d \times 1 \times C}$ compute outputs $O$, $\hat{O}_h$, and $\hat{O}_v$ respectively. The final pre-activation output is

$$O_{\text{ACB}} = O + \hat{O}_h + \hat{O}_v = \sum_k M_{:,:,k} * F_{:,:,k} + \sum_k M_{:,:,k} * F_{h,:,:k} + \sum_k M_{:,:,k} * F_{v,:,:,k}$$

After training, the linearity of convolution enables fusion: $1$) BN parameters are merged into their respective kernels; $2$) the three padded kernels are summed to form a single $d \times d$ filter and bias term, restoring the original model’s inference pattern without extra computational cost (Ding et al., 2019).

In super-resolution contexts, the Asymmetric Block (AB) generalizes the ACB design by stacking multiple layers (e.g., 17 layers, each with $3 \times 1$, $3 \times 3$, and $1 \times 3$ convolutions in parallel, followed by ReLU), as detailed in (Tian et al., 2021).

3. Training and Inference Pipeline

The standard procedure is as follows:

• Training: Replace every $d \times d$ convolution + BN with an ACB (or an AB in the SISR context) consisting of three parallel convolution+BN branches. The optimizer, loss function, and training schedules are retained from the baseline architecture. Training incurs approximately a $1.5\times$ increase in floating-point operations per layer, but requires no extra or tuned hyperparameters.
• Inference: After training, each ACB is collapsed into a single $d \times d$ convolution by fusing BNs and summing kernels. The final deployed model is structurally and computationally identical to the baseline but benefits from the enhanced kernel skeleton (Ding et al., 2019, Tian et al., 2021).

4. Empirical Results and Functional Benefits

Image Classification

Quantitative improvements using ACBs have been reported as follows (Ding et al., 2019):

• CIFAR-10/100: Across a range of architectures, ACB integration yields accuracy gains: Cifar-Quick (+1.11%, +1.08%), VGG-16 (+0.35%, +0.64%), ResNet-56 (+0.78%, +0.46%), WRN-16-8 (+0.59%, +0.79%), DenseNet-40 (+0.55%, +0.27%).
• ImageNet: Gains are observed for AlexNet, ResNet-18, and DenseNet-121, with top-1 improvements between +0.67% and +1.52%.

Robustness and Ablation

• Models equipped with only horizontal or vertical asymmetric branches demonstrate improved robustness to rotations and flips (full ACBs yield relative boosts of up to 2% on corrupted sets).
• Pruning experiments reveal that skeleton (central row/column) weights are significantly more critical than corner weights. After ACB-based training and fusion, this disparity widens, further evidencing ACB’s effect in reinforcing central kernel structure.

Single-Image Super-Resolution (SISR)

In SISR, the Asymmetric Block allows enhanced edge and texture modeling, outperforming prior architectures in both quantitative metrics (PSNR, SSIM) and perceptual measures (FSIM):

• On Urban100 ($\times 2$), ACNet achieves PSNR 31.79 dB (top ranking).
• In blind SISR and combined denoising scenarios, ACNet outperforms DnCNN and LESRCNN by 0.1–0.3 dB (Tian et al., 2021).
• Parameter and latency efficiency: ACNet achieves these results with ~1.28 M parameters, 8.1 GFLOPs, and 0.0195 s runtime on 512x512 input.

5. Implementation Strategies and Variations

For standard CNNs (stride-1, $d \times d$ kernels), integration is as simple as substituting each convolution+BN with an ACB. For large kernels (e.g., $5 \times 5$, $7 \times 7$), analogous $1 \times d$ and $d \times 1$ branches can be included, zero-padded as needed. The approach does not impact global architecture, skip connections, or concatenation mechanisms. No special initialization or hyperparameter tuning is required. Early layers with strongly non-square receptive fields show marginal gains with ACB integration.

In the SISR pipeline, the Asymmetric Block (AB) is chained with a Memory Enhancement Block (MEB)—which aggregates and up-samples feature channels using residual learning and sub-pixel convolution—and a High-Frequency Feature Enhancement Block (HFFEB) for feature fusion and image reconstruction (Tian et al., 2021).

6. Limitations and Potential Extensions

The factorization of 2D convolutions into 1D elements (i.e., assuming a dominant rank-1 structure) may diminish cross-directional expressivity, although use of the square kernel branch compensates for this. In the context of perceptual tasks such as SISR, pure MSE loss training can yield over-smoothed outputs compared to adversarial or perceptual-loss-driven networks.

Potential avenues for development include integrating channel-wise attention or dynamic routing mechanisms to adaptively balance the asymmetric branches, utilizing non-integer or deformable kernel sizes, and employing richer loss formulations for improved perceptual fidelity (Tian et al., 2021). The ACB framework is not limited to vision; its design may be extensible to other data modalities where axis-aligned spatial features are prominent.

7. Summary and Practical Recommendations

Asymmetric Convolution Blocks offer an effective methodology to fortify the skeleton of square convolutional kernels, boosting accuracy and robustness without incurring inference-time cost or architectural complexity. Their architecture-neutrality allows straightforward adoption within existing models. Empirical results validate the value of this approach across classification and super-resolution pipelines, with consistent boosts in both performance metrics and robustness. Researchers seeking to enhance mature CNN architectures—particularly those relying on $d \times d$ convolutions—can leverage ACBs as drop-in replacements for conventional convolutions, training as usual and fusing at inference with no parameter or FLOP increase (Ding et al., 2019, Tian et al., 2021).