Partial Asymmetry in Conv Architectures

• Partially asymmetric convolutional architecture is a family of neural designs that selectively augments symmetric operators with asymmetry to better capture anisotropic data properties.
• It employs techniques such as parallel kernel branches, axis-aware pooling, and decoupled kernel dimensions to reinforce directional features while preserving full square-kernel expressivity.
• These architectures have shown improvements in tasks ranging from image classification and super-resolution to hyperspectral analysis and time-series forecasting by optimizing operational efficiency and representational capacity.

Searching arXiv for the supplied papers to ground the article in current arXiv records. Partially asymmetric convolutional architecture denotes a class of neural architectures in which asymmetry is introduced selectively rather than uniformly. In the recent arXiv literature, the term has been used for several related but non-identical design strategies: augmenting square convolutions with horizontal and vertical one-dimensional branches, imposing different operations on different axes of anisotropic data, decoupling vertical and horizontal kernel lengths on constructed two-dimensional representations, unbalancing the two pointwise convolutions of an inverted residual bottleneck, and distributing convolutional and transformer blocks asymmetrically across depth (Ding et al., 2019, Tian et al., 2021, Zhan et al., 2023, He, 2023, Yang et al., 2021, Kag et al., 2024). Taken together, these works suggest a common principle: preserve symmetric operators where their inductive bias or hardware behavior is useful, and inject asymmetry only where the data geometry, feature hierarchy, or compute budget makes equal treatment suboptimal.

1. Conceptual scope and recurring forms

The expression does not denote a single canonical block. Rather, it names a family of controlled anisotropies. In some papers, the asymmetry is local to a convolutional kernel; in others, it is expressed through pooling, channel allocation, or macro-stage composition. The shared property is that the architecture remains only partly asymmetric: a symmetric backbone or operator is retained, while asymmetry is introduced in selected branches, dimensions, or stages.

Locus of asymmetry	Mechanism	Representative papers
Kernel augmentation	Parallel $d\times d$, $1\times d$, and $d\times 1$ branches	(Ding et al., 2019, Tian et al., 2021)
Axis-aware processing	Spectral-only or spatial-only pooling before 3D convolution	(Zhan et al., 2023)
Decoupled separable kernels	$V\times 1$ and $1\times H$ with $V\neq H$	(He, 2023, Li, 28 Jul 2025)
Bottleneck imbalance	First PW weakened, second PW strengthened via feature reuse	(Yang et al., 2021)
Hybrid macro-allocation	More convolution early, more attention late	(Kag et al., 2024)

A plausible implication is that “partial” asymmetry is best understood as a design regime rather than a single operator family. The surveyed works all reject full symmetry, but they do so at different levels of abstraction.

2. Kernel-level partial asymmetry in convolutional CNNs

At the kernel level, the clearest formulation appears in “ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks” (Ding et al., 2019). There, a standard square convolution is replaced during training by an Asymmetric Convolution Block comprising three parallel branches: one $d\times d$, one $1\times d$, and one $d\times 1$, each with its own BatchNorm, followed by element-wise summation. The central claim is not low-rank approximation, but “strengthening the kernel skeletons”: the middle row and middle column of a square kernel are treated as structurally important and are explicitly reinforced by the asymmetric branches. After training, the three branches are fused exactly into a single $d\times d$ convolution, so the deployed network has the same topology and inference cost as the baseline. On CIFAR-10, CIFAR-100, and ImageNet, this construction improved a variety of baselines, including ResNet-56, WRN-16-8, AlexNet, ResNet-18, and DenseNet-121; for example, ResNet-18 on ImageNet improved from 70.36/89.61 top-1/top-5 to 71.14/89.96, and pruning analyses showed that skeleton weights are substantially more critical than corner weights (Ding et al., 2019).

The super-resolution ACNet of “Asymmetric CNN for image super-resolution” (Tian et al., 2021) applies the same basic intuition to low-level vision, but does so inside a larger 23-layer architecture. It is explicitly described as a partially asymmetric convolutional architecture composed of a 17-layer Asymmetric Block, a 1-layer Memory Enhancement Block, and a 5-layer High-Frequency Feature Enhancement Block. Each AB layer uses three parallel convolutions,

$1\times d$0

followed by ReLU, so the design retains a full $1\times d$1 kernel while adding directional branches rather than factorizing the square kernel away. MEB aggregates all pre-ReLU AB outputs as

$1\times d$2

then upsamples via sub-pixel convolution, and HFFEB fuses deep HR features with shallow HR features to suppress excessive feature enhancement. Empirically, the asymmetric AB outperformed square alternatives while reducing complexity: on B100 and Urban100 at $1\times d$3, AB reached 31.07/0.9176 and 31.82/0.8951, compared with AEB at 30.98/0.9155 and 31.78/0.8945; AB used 1.025M parameters and 4.67G FLOPs, versus 1.813M and 7.90G for AEB. Full ACNet further improved to 31.79/0.9245 on B100 and 32.06/0.8978 on Urban100, and the paper reports effectiveness for SISR, blind SISR, and blind SISR with noise (Tian et al., 2021).

These two ACNet lines are closely related but not identical. In both, asymmetry augments rather than replaces square kernels. This suggests that one major interpretation of partial asymmetry is “directional reinforcement without surrendering full-rank square-kernel expressivity.”

3. Axis-aware asymmetry in anisotropic multidimensional data

In hyperspectral image classification, partial asymmetry is expressed less through literal non-square kernels than through different treatment of spectral and spatial axes. “A2S-NAS: Asymmetric Spectral-Spatial Neural Architecture Search For Hyperspectral Image Classification” (Zhan et al., 2023) starts from the premise that spectral and spatial dimensions are physically anisotropic: spectral bands are densely sampled and highly correlated, whereas spatial structure is coarser and semantically different. The proposed A2SConv block therefore combines an outer search space over axis-selective pooling with an inner search space over 3D convolutions of varying receptive field. The outer options are no pooling, spectral pooling with stride $1\times d$4, and spatial pooling with stride $1\times d$5; the inner options use kernel size $1\times d$6 and dilation $1\times d$7.

The searchable block is parameterized in DARTS style as

$1\times d$8

with softmax-normalized architecture parameters. The resulting three-stage architecture contains six A2SConv blocks, each with independent architecture parameters, so asymmetry is both dimension-wise and layer-wise. Because spectral-only and spatial-only pooling are searched per layer, the architecture remains only partly asymmetric: the 3D convolution kernels themselves are isotropic in form, but their effective behavior becomes anisotropic through the axis-selective preprocessing.

The empirical results support the importance of this axis-aware design. On Indian Pines, A2S-NAS achieved OA 97.17%, AA 97.05%, and $1\times d$9 96.73, exceeding 3DCNN, SSRN, 3DAutoCNN, 3D-ANAS, and SSTN. On Houston University, it reached OA 92.20%, AA 93.30%, and $d\times 1$0 91.56. The searched architectures also displayed a high prevalence of asymmetric pooling: 50% of layers on Indian Pines and 83.3% on Houston University contained asymmetric pooling operations (Zhan et al., 2023). A plausible implication is that, for anisotropic data, the decisive asymmetry may lie in axis-specific resolution control rather than in kernel shape alone.

4. Semi-asymmetric and partially asymmetric convolutions for time series

Time-series forecasting papers extend the concept further by first transforming a one-dimensional sequence into a two-dimensional representation whose axes have different semantics. In “Time Series Forecasting via Semi-Asymmetric Convolutional Architecture with Global Atrous Sliding Window” (He, 2023), the series is first differenced, segmented by sliding windows, mapped into a fuzzy-universe-based global interval representation, aligned by padding and cropping, and processed by a Central-Bidirectional Atrous Algorithm before entering a Semi-Asymmetric Convolutional Network. The semi-asymmetric layer is

$d\times 1$1

where $d\times 1$2 and $d\times 1$3. The paper states explicitly that when $d\times 1$4, the construction degenerates into regular asymmetric convolution; the intended regime is therefore $d\times 1$5, with different extents along the local-window axis and the global-interval axis. Extensive experiments over 27 univariate and 16 multivariate datasets showed state-of-the-art performance on most datasets, and the Nemenyi/Friedman analyses assigned the best average rank to the method (He, 2023).

“Adaptive Fuzzy Time Series Forecasting via Partially Asymmetric Convolution and Sub-Sliding Window Fusion” (Li, 28 Jul 2025) generalizes this logic and explicitly uses the phrase “partially asymmetric convolutional architecture.” Here the sequence is again differenced and fuzzified, but the representation is further processed by a bilateral Atrous algorithm that preserves the central reconstructed element while applying dilated convolution to left and right fuzzy contexts. The main branch of the CNN head is

$d\times 1$6

with $d\times 1$7, $d\times 1$8, and multiple $d\times 1$9 pairs in parallel. A residual-like branch

$V\times 1$0

is fused by

$V\times 1$1

The stated rationale is that decoupled vertical and horizontal kernel lengths allow the model to construct sub-windows of different lengths inside the original sliding window, thereby capturing short-term and longer intra-window dependencies at multiple granularities. On 43 datasets, including M1–M4, Tourism, Traffic, Electricity, and Bitcoin, the method is reported to achieve state-of-the-art MAE/RMSE on most datasets, although the authors note that it is not the best performer on some datasets with very long-range, quasi-periodic dynamics such as Sunspot, US Births, and Solar Weekly (Li, 28 Jul 2025).

Across these time-series papers, partial asymmetry is neither merely spatial nor merely computational. It is tied to the semantics of a constructed 2D tensor: one axis encodes local temporal position or sub-window span, and the other encodes global fuzzy location or tendency. This suggests that partial asymmetry can be induced by representation design as much as by the convolution operator itself.

5. Capacity asymmetry and asymmetric macro-allocation

A different interpretation appears in mobile CNN design. “AsymmNet: Towards ultralight convolution neural networks using asymmetrical bottlenecks” (Yang et al., 2021) does not alter the spatial shape of the kernel. Instead, it makes the inverted residual bottleneck asymmetric in channel allocation. In a standard MobileNetV2-style bottleneck, the first and second pointwise convolutions both operate at width $V\times 1$2. AsymmNet weakens the first PW to $V\times 1$3, concatenates reused input features, and lets DW and the second PW operate at $V\times 1$4. The block is written as

$V\times 1$5

and the MAC ratio is

$V\times 1$6

The intended effect is to migrate computation from the first PW, associated mainly with capacity and information flow, to the second PW, associated mainly with expressiveness after spatial mixing. Empirically, some asymmetry was beneficial but too much was not: on ImageNet, $V\times 1$7 outperformed $V\times 1$8 and $V\times 1$9 for both Large and Small variants. In the ultralight regime, gains were notable; for example, at 0.35-Small, MobileNetV3 Small achieved 49.8% top-1 while AsymmNet Small reached 55.0% (Yang et al., 2021).

In “AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation” (Kag et al., 2024), the locus of asymmetry moves again, from channels within a bottleneck to the stage-wise allocation of convolutional and transformer blocks. The architecture is explicitly asymmetric because it contains more convolutional blocks at higher resolutions and more transformer blocks at lower resolutions. The canonical tiny configuration is written as CC–CCCT–CCTT–CTTT across stages S1–S4: S1 is conv-only, S2 is 75% C / 25% T, S3 is 50% / 50%, and S4 is 25% / 75%. The empirical comparison against symmetric alternatives is direct. Configuration C1, the asymmetric CC–CCCT–CCTT–CTTT layout, achieved 83.4% top-1 with 55M parameters and A100 throughput 4295 img/s at batch size 64, whereas the symmetric C10 configuration CC–CCTT–CCTT–CCTT achieved 82.8% with 72M parameters and 3820 img/s. The same asymmetric principle transferred beyond classification: AsCAN-T, AsCAN-B, and AsCAN-L improved the accuracy–throughput trade-off on ImageNet and ADE20K, and the asymmetric UNet variants delivered strong class-conditional and text-to-image generation results, including FID 2.41 at $1\times H$0 for class-conditional ImageNet generation and strong GenEval performance for large-scale T2I (Kag et al., 2024).

These cases broaden the term considerably. Partial asymmetry can describe not only anisotropic receptive fields, but also non-uniform allocation of representational budget across sublayers or across depth. A plausible implication is that the common denominator is controlled imbalance in response to unequal functional roles.

6. Empirical regularities, misconceptions, and limitations

A recurring misconception is that partially asymmetric convolution necessarily means factorizing a square kernel into two one-dimensional kernels. The surveyed literature does not support that equivalence. The recognition ACNet and the super-resolution ACNet both retain a full square branch and add $1\times H$1 and $1\times H$2 branches in parallel, explicitly to avoid the information loss associated with pure factorization and rank-1 restrictions (Ding et al., 2019, Tian et al., 2021). In these cases, asymmetry is additive rather than substitutive.

A second misconception is that asymmetry must pervade the whole network. In fact, the defining feature of these architectures is selectivity. A2S-NAS keeps standard 3D convolutions but injects asymmetry through per-layer spectral or spatial pooling (Zhan et al., 2023). The time-series models apply asymmetric kernels only after a highly structured fuzzy and atrous preprocessing pipeline (He, 2023, Li, 28 Jul 2025). AsymmNet retains the standard bottleneck template but redistributes capacity across its two pointwise convolutions (Yang et al., 2021). AsCAN retains a standard four-stage hierarchy but varies the C/T ratio across stages rather than within every block uniformly (Kag et al., 2024). This suggests that “partial” is not incidental wording; it is the primary design choice.

A third misconception is that asymmetry is always introduced for parameter reduction. The evidence is more mixed. In the SR ACNet, AB reduced parameters and FLOPs relative to AEB while improving PSNR/SSIM (Tian et al., 2021). In the recognition ACNet, training-time cost increases because every square layer becomes a three-branch ACB, but post-training fusion eliminates inference overhead (Ding et al., 2019). In AsymmNet, total MACs are kept approximately constant while compute is reallocated to the second PW (Yang et al., 2021). In AsCAN, the goal is not raw FLOP minimization alone, but better real latency through hardware-friendly block placement and avoidance of expensive high-resolution attention (Kag et al., 2024). The role of asymmetry is therefore task- and architecture-dependent.

The limitations reported by the papers are similarly heterogeneous. The SR ACNet is tuned for super-resolution and evaluated only on SR and SR-with-noise, with fixed directional kernels $1\times H$3 and $1\times H$4 and no explicit analysis of diagonal orientations (Tian et al., 2021). A2S-NAS is validated on hyperspectral classification benchmarks and derives its motivation from spectral–spatial anisotropy rather than generic vision tasks (Zhan et al., 2023). The 2025 fuzzy forecasting model can lag on very long-range, quasi-periodic series and is sensitive to window size, growth rate, kernel-set choice, and fuzzy partition granularity (Li, 28 Jul 2025). AsymmNet’s gains diminish or invert at larger scales, suggesting that its benefits are strongest in the ultralight regime (Yang et al., 2021). AsCAN still uses vanilla quadratic attention and identifies efficient kernel support and operator substitutions as future work (Kag et al., 2024).

Taken together, these results indicate that partially asymmetric convolutional architecture is best treated as a general architectural principle for encoding anisotropy, unequal functional roles, or staged computation. Its concrete realization depends on what is being made asymmetric: kernel support, axis resolution, separable extent, channel flow, or block distribution across depth.