Filter-in-Filter Pruning for CNN Efficiency

• The paper introduces filter in filter pruning by decomposing convolution filters into 1x1 stripes and using learned multiplicative gates to remove redundant sub-components.
• The method balances fine-grained sparsity and hardware compatibility, achieving up to 92.7% parameter reduction on VGG16 and minimal accuracy loss on modern architectures.
• Structured stripe-wise pruning improves CNN efficiency, maintaining dense computation paths and enabling practical acceleration on general-purpose hardware.

Pruning Filter in Filter

The pruning of "filter in filter" refers to structured sparsification strategies that operate at a granularity finer than whole convolution filters but coarser than arbitrary unstructured weight pruning. In contemporary convolutional neural networks (CNNs), this methodology is exemplified by techniques that decompose each spatial filter into collections of smaller entities—such as stripes or sub-filters—then selectively remove these sub-components to achieve improved efficiency while maintaining computational structure for implementation on general-purpose hardware.

1. Conceptual Distinction and Motivation

Conventional filter pruning eliminates entire output channels (filters), enabling direct reductions in both computational and storage costs, and producing models with maintained structural regularity. Weight pruning, in contrast, sets individual weights to zero, maximizing parameter reduction but yielding highly unstructured sparsity that confounds efficient execution on standard hardware platforms. The "filter in filter" paradigm, as developed in "Pruning Filter in Filter" (Meng et al., 2020), seeks to reconcile these approaches by enabling finer-grained, yet still hardware-friendly, pruning.

A convolutional filter $F \in \mathbb{R}^{C \times K \times K}$, with $C$ input channels and a $K \times K$ spatial kernel, is amenable to decomposition into $K^2$ stripes, where each stripe is a $1 \times 1$ filter along the channel axis. "Stripe-wise pruning" (SWP) thus prunes at the level of these stripes, increasing granularity by a factor of $K^2$ relative to full-filter pruning, and providing a more precise mechanism for shaping the effective receptive field and sensitivity of the convolution operator (Meng et al., 2020).

2. Mathematical Framework and Algorithmic Implementation

In SWP, each filter is decomposed as

$$F_n = [s_{n,i,j}]_{i,j=1}^{K}, \quad s_{n,i,j} \in \mathbb{R}^C,$$

for $n = 1, ..., N$ (number of output filters/channels). To learn and govern the significance of each stripe, a learnable "filter skeleton" $I^l \in \mathbb{R}^{N \times K \times K}$ is introduced. The modified convolutional operation at layer $l$ becomes

$$X^{l+1}_{n,h,w} = \sum_{c=1}^C \sum_{i,j=1}^K [I^l_{n,i,j} \cdot W^l_{n,c,i,j}] \cdot X^l_{c,h+i-\frac{K+1}{2},w+j-\frac{K+1}{2}},$$

where $I^l_{n,i,j}$ is a learned multiplicative gate for stripe $(n,i,j)$. The training objective includes a sparsity-promoting term:

$$\mathcal{L}(W, I) = \sum_{(x, y)} \mathrm{loss}(f(x; W \odot I), y) + \alpha \sum_{l=1}^L \sum_{n,i,j} |I^l_{n,i,j}|,$$

with elementwise multiplication denoted $\odot$ and $\alpha > 0$ the regularization coefficient.

During training, gradient-based optimization updates both $W$ and $I$, with a typical thresholding step that freezes or zeros stripes where $I^l_{n,i,j} < \delta$ after training converges. The final model thus has a subset of stripes per filter removed, reducing both parameter count and computational demand. The preserved structured sparsity at the stripe level ensures compatibility with matrix multiplication or im2col implementations, incurring negligible indexing overhead (Meng et al., 2020).

3. Relationship to Conventional Pruning and Structural Implications

Compared with filter-wise and weight-wise sparsification, "filter in filter" methods occupy an intermediate position in the sparsity hierarchy. They substantially increase the flexibility and pruning ratio relative to standard filter pruning by allowing non-axis-aligned, nonrectangular support for convolution kernels, which in turn enables the preservation of critical signal trajectories within filters while excising unnecessary responses.

By maintaining stripe-level (i.e., $1 \times 1 \times C$) patterning, SWP preserves the memory and compute regularity that is required for practical acceleration on CPUs and GPUs, avoiding the inefficiency of arbitrary sparsity. The learned filter skeletons ($I^l$) reflect the empirical importances of subregions within each filter, thus allowing for the emergence of effective shapes or receptive field patterns customized to the dataset and task (Meng et al., 2020).

4. Experimental Evidence and Empirical Performance

Extensive evaluations on CIFAR-10 and ImageNet demonstrate that SWP attains state-of-the-art or superior pruning ratios with minimal loss of accuracy. For instance, on VGG16 and ResNet56 backbones on CIFAR-10, pruning using SWP achieved up to 92.7% parameter reduction (VGG16) and 77.7% (ResNet56), with corresponding FLOPs reductions of 71.2% and 75.6%. The change in accuracy was within $-0.4\%$ (VGG16) and even a $+0.12\%$ gain (ResNet56). On ImageNet with ResNet-18, SWP reduced FLOPs by 50.5–54.6% with less than $0.2\%$ change in top-1/top-5 accuracy. Ablations on hyperparameters $\alpha$ and $\delta$ confirmed that optimal performance balances aggressive stripe elimination with conservative preservation of critical kernel regions.

Importantly, these compression ratios and empirical losses exceed those of structured filter pruning methods such as L1-norm, ThiNet, GAL, and HRank, while retaining dense execution paths required for deployment (Meng et al., 2020).

5. Hardware Efficiency and Structured Sparsity

A principal advantage of the "filter in filter" strategy is its preservation of structured sparsity. Each pruned stripe corresponds to an entire $1 \times 1$ convolution across all input channels, maintaining alignment with the memory access patterns exploited by tensor core or SIMD matrix-multiply implementations. The mask overhead scales as $O(N K^2)$ per layer and is negligible compared to the base parameter storage. Because the surviving stripes can be arranged into contiguous segments and processed as dense submatrices, the effective speedup approaches theoretical parameter and FLOPs reduction, without requiring specialized hardware for sparse tensor arithmetic (Meng et al., 2020).

6. Limitations and Prospects for Further Development

While SWP increases granularity by a factor of $K^2$, it does not approach the arbitrary flexibility of weight pruning. Further, the technique as presented works directly for $K \times K$ standard convolutions but requires adaptation for grouped or depthwise convolutions, or generalization to architectures such as transformers. Open questions include the design of adaptive per-layer or per-filter regularization schedules ($\alpha$, $\delta$), automatic hybridization with macro-level pruning, and synergy with quantization or architectural search techniques. Extensions to grouped/depthwise filters and integration with dynamic inference policies represent promising avenues for further enhancing pruning efficiency (Meng et al., 2020).

---

In summary, "pruning filter in filter" via methods such as SWP constitutes a central advance in CNN compression, enabling high sparsity at the sub-filter level while fully preserving hardware compatibility and minimizing accuracy degradation. It achieves pruning ratios that challenge those of unstructured weight pruning, but with none of the deployment overhead, making it a practical solution for deep network optimization in performance-critical environments (Meng et al., 2020).