Compressed Convolution Networks (CoCN)

• CoCNs are convolutional neural networks engineered for efficient computation by compressing architecture and weights using methods like bottlenecks, pruning, and quantization.
• They achieve dramatic reductions in model size and computational load—up to 261× in some cases—while maintaining near-parity in accuracy.
• CoCNs extend to both Euclidean and non-Euclidean domains, employing techniques such as wavelet transforms and permutation calibration to optimize dense image and graph tasks.

A Compressed Convolution Network (CoCN) refers to any convolutional (or graph convolutional) neural network whose architecture or parameters are specifically engineered for reduced memory footprint, storage, and computational requirements, while maintaining high predictive accuracy. The term encompasses a broad family of design patterns, including architectural compression (e.g., bottleneck and “divide-and-conquer” modules), weight-level compression (basis decomposition, quantization, and pruning), and compressed-domain operations for both Euclidean and non-Euclidean (graph) data. Implementations vary from plug-in modules in classic CNNs to fully end-to-end learned hierarchical architectures for graphs and images.

1. Architectural Compression Principles and Early Models

Early CoCN approaches exploit architectural bottlenecks, residual connections, and module re-use to minimize parameter count without sacrificing accuracy. Residual-Squeeze-CNDS (ResSquCNDS) (Qassim et al., 2017) is a canonical early example. Built atop an 8-layer Residual-CNDS backbone, it replaces five standard $3 \times 3$ convolutions with SqueezeNet "Fire" modules—whose "squeeze" layers (1×1 convs) funnel spatial channels before "expand" ($1 \times 1$ and $3 \times 3$) layers re-project them, followed by three interleaved residual (shortcut) connections and a single deeply supervised branch. The main design heuristics, originating from SqueezeNet, are: maximize the use of $1 \times 1$ convolutions, minimize $3 \times 3$ channels by squeezing, and delay downsampling to late stages. Empirically, ResSquCNDS realizes an 87.6% model-size reduction (from $\approx$14GB to $\approx$1.73GB) and 13.3% training speedup on MIT Places365, retaining over 99% of the original Top-1 accuracy (51.32% vs. 51.98%) (Qassim et al., 2017).

Module-based architectures such as CompConv (Zhang et al., 2021) extend this pattern. CompConv networks recursively decompose output channels, replacing a full $k \times k$ convolution with a tree of small convs and identity (channel-copy) paths. At each level, some outputs are directly borrowed from the inputs, others are learned by smaller convs, culminating in channel shuffling for information mixing. For example, CompConv-128 (base channel 128, depth 3) reduces VGG-16 on CIFAR-10 from 15M to 3.3M params (79% reduction) with under 0.3% top-1 drop, and a corresponding drop in FLOPs (Zhang et al., 2021).

2. Parameter Space Compression: Basis, Binary, and Quantized Representations

Weight-level compression targets the convex hull of expressible filters with rank- or basis-constrained parameterizations, binarization, and quantization.

BasisConv (Tayyab et al., 2019) replaces each $K \times K \times C_{in} \to C_{out}$ conv with a two-layer structure: (a) a fixed basis conv layer (filters $F \in \mathbb{R}^{(K^2C_{in}) \times Q}$, either from SVD of trained weights or random orthogonal for train-from-scratch), followed by (b) a learnable $1 \times 1$0 conv with weights $1 \times 1$1. The basis is fixed, only the combiner is learnable. This yields up to 18× parameter and 5× FLOP reduction in benchmarks such as VGG/ResNet/DenseNet on CIFAR-100, typically with <3% accuracy loss for basis truncation fractions $1 \times 1$2.

Low-dimensional binary stacking (Lan et al., 2020) achieves even more aggressive compression by approximating every high-dimensional convolutional filter by stacking $1 \times 1$3 contiguous blocks from a shared, small dictionary of binary filters $1 \times 1$4, each selected and scaled for the target filter. The selection and scaling are learned by proxies and straight-through estimation. The split–transform–merge procedure decomposes all input channels, applies all possible binary convolutions once, then re-assembles using layer-specific indices and scales. This architecture attains 58–261× compression over full-precision models on VGG/ResNet variants, substantially outperforming classic binary networks (XNOR, BWN, etc.) in compression-per-accuracy (Lan et al., 2020).

Weight pruning and quantization (Marinò et al., 2021) (often combined) further reduce model storage. Magnitude-based pruning zeros out small weights; quantization (uniform, clustering, entropy-constrained) replaces real weights with $1 \times 1$5-level codebook references. For moderate quantization levels ($1 \times 1$6–128), convolutional layers permit only mild pruning (10–30%), while FC layers safely support extreme sparsity ($1 \times 1$7) and quantization (up to $1 \times 1$8 SHAM compressed). Realized models show 20× end-to-end size reduction with negligible accuracy impact (Marinò et al., 2021).

3. Compressed-Domain Operations and Feature Map Compression

Compressed convolution not only refers to parameters but can be extended to convolutional feature maps, particularly in the context of dense image prediction.

Wavelet Compressed Convolution (WCC) (Finder et al., 2022) applies Haar-Wavelet transforms to reduce the feature map spatial footprint before 1×1 (point-wise) convolution. Each feature map is multi-level decomposed; only the $1 \times 1$9-fraction (shrinkage rate) largest coefficients across all bands and channels are retained (joint shrinkage, $3 \times 3$0 or less). The 1×1 convolution is then performed directly in the compressed domain, with inverse HWT reconstructing the spatial map post-conv. This technique, implemented as a drop-in replacement for standard 1×1 convolution, reduces both multiplication operations ("BOPs") and memory traffic by factor $3 \times 3$1, with substantially lower induced MSE compared to standard quantization. For dense prediction tasks (segmentation, depth estimation, super-resolution), WCC + 8-bit quantization at $3 \times 3$2 maintains mIoU or PSNR within 1–2% of full-precision baselines even as BOPs are reduced by 20–50× (Finder et al., 2022).

4. CoCN in Non-Euclidean Domains: Graph Convolution

Generalizing compressed convolution to graphs, "Scalable Graph Compressed Convolutions" (Sun et al., 2024) introduces a differentiable permutation layer to "calibrate" graph node/adjacency orderings, allowing Euclidean-style convolution to operate on arbitrarily structured graphs. The permutation is a learned doubly-stochastic approximation to the permutation matrix, aligning local node neighborhoods into contiguous blocks. CoCN then applies "diagonal" compressed convolution: sliding $3 \times 3$3 kernels along the main diagonal of the permuted adjacency matrix and over sequences of permuted node features, with shared parameters $3 \times 3$4 and $3 \times 3$5. Hierarchical architectures are supported by stacking compressed-conv, pooling, and upsampling layers; residual and inception-style modules further enhance multiscale receptive field aggregation. Sparse and segment CoCN variants allow scalability to large graphs.

Experimentally, CoCN architectures exceed classical message-passing GNNs and recent hierarchical models on node and graph classification, graph isomorphism benchmarks, and link prediction, demonstrating higher expressive power (universal local aggregator via Euclidean convolution) and efficacy for both homophilic and heterophilic graphs (Sun et al., 2024).

5. Implementation and Training Methodologies

Implementation of CoCNs varies with the compression technique:

• Module-based compression: Direct replacement of vanilla conv layers (ResSquCNDS Fire, CompConv, etc.) with compressed modules; parameter initialization and training schemes adhere to standard SGD/Adam pipelines (Qassim et al., 2017, Zhang et al., 2021).
• Basis or binary decompositions: Fixed basis layers are constructed (from SVD or random orthonormal), followed by either fine-tuning only the combiner weights ($3 \times 3$6 conv in BasisConv) or synchronized proxy/STE updates (binary stacking) for both the dictionary and selector parameters (Tayyab et al., 2019, Lan et al., 2020).
• Pruning/quantization: Compression is performed post-training, optionally with retraining/fine-tuning. Storage utilizes lossless coding—HAM/SHAM (bitstream + index) or index-mapping—inference decodes or looks up weights at computation time (Marinò et al., 2021).
• Wavelet compressed domain: WCC layers require forward/inverse HWT implementations, compressed-domain 1×1 GEMM with reconstructed features, and optional quantization-aware training (Finder et al., 2022).
• Graph CoCN: Permutation calibration modules, compressed-conv/inception/residual building blocks, and hierarchical stacking are implemented as differentiable, end-to-end-trainable modules within standard GNN frameworks (Sun et al., 2024).

Data augmentation, batch normalization, and progressive learning-rate schedules remain standard, but certain variants (e.g., BasisConv, WCC) benefit from staged freezing/thawing and careful control of retained basis rank or coefficient shrinkage (Tayyab et al., 2019, Finder et al., 2022).

6. Empirical Trade-offs, Limitations, and Comparative Performance

Empirical results across architectures and datasets demonstrate that CoCNs typically achieve 5–300× reduction in model size or computation, often with less than 1–3% drop in task accuracy:

Model/Task	Method	Compression	Acc. Drop	Key Details
ResSquCNDS, Places	Fire+Residual	8.1×	<1%	87.6% smaller, 13% faster (Qassim et al., 2017)
VGG-16, CIFAR-10	CompConv-128	4.5×	0.3%	3.3M vs 15M params (Zhang et al., 2021)
AlexNet, CIFAR-100	BasisConv, t=0.85	13.4×	1.4%	2.8× FLOPs, only 1.4% Top-1 drop (Tayyab et al., 2019)
VGG/ResNet, CIFAR	Binary stacking	58–261×	<2%	Outperforms BWN/XNOR (Lan et al., 2020)
DeepLabV3+, Citys.	WCC, γ=0.25	54× BOPs	8% mIoU	Wavelet wins over quantization (Finder et al., 2022)
TUDatasets, Graphs	CoCN (vanilla)	−	+2–4%	Outperforms ChebNet, GCN, DiffPool (Sun et al., 2024)

Strengths of CoCNs generally include dramatic parameter/compute reduction at modest accuracy cost, hardware efficiency (bit operations, memory traffic), and applicability to architectures where post-training quantization or pruning would yield severe accuracy deterioration.

Limitations and considerations include:

• Severe compression (>90%) of convolutional layers can induce non-trivial accuracy loss if not combined with compensatory mechanisms (e.g., residual, deep supervision, fine-tuning) (Marinò et al., 2021).
• Some techniques (e.g., wavelet shrinkage, binary stacking) may complicate accelerator deployment or inference-time latency if not hardware aligned (Finder et al., 2022, Lan et al., 2020).
• Highly pruned or quantized FC layers may require increased decoding overhead, mitigated by parallelization (Marinò et al., 2021).
• Graph calibration for non-Euclidean CoCNs introduces $3 \times 3$7 or $3 \times 3$8 costs, but sparse/segment variants control this growth (Sun et al., 2024).

7. Extensions and Future Directions

• Multi-domain expansion: CoCNs now encompass not only classic image and video models but also large-scale graph learning, where permutation-calibrated compressed convolution provides new expressive power beyond 1-WL GNNs (Sun et al., 2024).
• Hybrid methods: Orthogonal compression strategies may be layered: module compression (e.g., CompConv) can be combined with quantization, pruning, or wavelet-domain operations for compound compression (Zhang et al., 2021, Finder et al., 2022).
• Task-specific regimes: For classification, aggressive quantization may be acceptable, but for dense prediction, compressed-domain convolutions (e.g., WCC) are preferred due to superior MSE/metric scaling (Finder et al., 2022).
• Hardware and deployment: Further work is needed on hardware-aligned coding schemes, fast decode paths for source-coded weights, and quantization for non-uniformly distributed feature maps (Marinò et al., 2021, Finder et al., 2022).
• Theoretical analyses: Universal approximation and expressiveness in graph compression, parameter–accuracy trade-off boundaries for learned bases and self-similar modules, and robust calibration for irregular data remain areas of active research (Sun et al., 2024, Lan et al., 2020).

In summary, Compressed Convolution Networks comprise a rigorously substantiated framework for decomposing, representing, and operating on convolutional weights, activations, and graph features in an optimally compressed form, blending architectural design, parameter encoding, and latent-space operations across Euclidean and non-Euclidean domains.