Kernel-Group Pruning in Convolutional Networks

• Kernel-group pruning is a structured approach that removes groups of convolution kernel weights to balance granularity and implementation efficiency.
• It leverages diverse grouping strategies—from spatial position slicing to block pruning and clustering—to optimize network computation and reduce overhead.
• Empirical results demonstrate that kernel-group pruning can markedly reduce FLOPs and accelerate inference with minimal accuracy loss.

Kernel-group pruning denotes a family of structured pruning methods for convolutional networks in which the pruned unit is a group aligned with convolutional structure rather than an arbitrary scalar coefficient. In the literature, these groups range from kernel-position × input-channel coefficient vectors shared across all output filters, to groups of complete kernels sharing an input channel, to contiguous blocks of kernels, and to clusters of similar kernels used to drive filter selection. The common objective is to retain finer granularity than whole-filter pruning while avoiding the implementation pathologies of unstructured sparsity, especially when practical acceleration or dense deployable subnetworks are required (Lebedev et al., 2015, Lin et al., 2021, Park et al., 2024).

1. Definitions and scope

The literature does not use a single uniform definition of kernel-group pruning. Some papers prune groups of coefficients inside the convolution tensor; some prune groups of complete $k\times k$ kernels; some use kernel groups only as an intermediate representation and still output a channel-pruned dense model. The essential commonality is that the pruning decision is made over a structured kernel-level object rather than over isolated weights.

Representative form	Group definition	Resulting structure
"Group-wise brain damage" (Lebedev et al., 2015)	$\Gamma_{ijs}=K(i,j,s,:)$	Smaller dense GEMM
"$1\times N$" pruning (Lin et al., 2021)	$N$ consecutive kernels with the same input channel index	Block-sparse kernel matrix
Feature-wise hierarchical grouping (Mitsuno et al., 2020)	$g_j^l=W^l_{:,j,:,:}$	Filter/channel-pruned dense CNN
REPrune (Park et al., 2024)	Clusters in $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$	Filter subset with maximum cluster coverage

This range is important because the phrase “kernel-group pruning” can otherwise be conflated with filter pruning, channel pruning, or unstructured kernel sparsity. It is not synonymous with whole-filter removal: in the canonical early formulation, the group is all coefficients at one kernel location $(i,j)$ and one input channel $s$, across all output channels, which is more fine-grained than filter pruning but more structured than scalar pruning (Lebedev et al., 2015). Conversely, some later methods are best understood as channel pruning informed by kernel grouping rather than direct kernel-group deletion (Park et al., 2024).

2. Lowering-aligned kernel-group pruning

The clearest early formulation is "Fast ConvNets Using Group-wise Brain Damage" (Lebedev et al., 2015). There, a convolution kernel tensor

$K \in \mathbb{R}^{d \times d \times S \times T}$

is reshaped into a filter matrix

$F \in \mathbb{R}^{T \times d^2 S},$

while the input is lowered into a patch matrix

$\Gamma_{ijs}=K(i,j,s,:)$0

so that convolution becomes

$\Gamma_{ijs}=K(i,j,s,:)$1

The decisive grouping rule is

$\Gamma_{ijs}=K(i,j,s,:)$2

namely all weights obtained by fixing one spatial position $\Gamma_{ijs}=K(i,j,s,:)$3 and one input channel $\Gamma_{ijs}=K(i,j,s,:)$4, and taking all output channels. In the lowered view, one such group is exactly one column of $\Gamma_{ijs}=K(i,j,s,:)$5 and one row of $\Gamma_{ijs}=K(i,j,s,:)$6. If $\Gamma_{ijs}=K(i,j,s,:)$7 is pruned to zero, the corresponding spatial sample is removed from both lowered matrices, so the computation becomes a smaller dense multiplication rather than a sparse one.

The retained offsets are encoded as channel-specific patterns

$\Gamma_{ijs}=K(i,j,s,:)$8

yielding reduced matrix sizes

$\Gamma_{ijs}=K(i,j,s,:)$9

With density

$1\times N$0

the paper states a theoretical speedup of approximately $1\times N$1 (Lebedev et al., 2015). This is the central computational argument of lowering-aligned kernel-group pruning: practical speedup follows because the backend still executes dense linear algebra, but on reduced inner dimensions.

The regularizer is a standard group lasso / $1\times N$2-norm:

$1\times N$3

Its gradient shrinks all coefficients in a group together, rather than independently. The paper studies two optimization procedures for pretrained networks. The first is group-wise sparsification followed by hard pruning and long fine-tuning. The second, and more successful, is gradual group-wise sparsification with truncated penalty

$1\times N$4

together with a validation-controlled update of $1\times N$5, greedy removal when $1\times N$6, and monotonic growth of sparsity (Lebedev et al., 2015).

This formulation already contains several themes that remained central in later work. First, the group is chosen to coincide with a removable GEMM inner dimension, not merely with a semantically convenient tensor slice. Second, the method is more fine-grained than channel or filter pruning, but more hardware-friendly than scalar sparsity. Third, the learned pattern can alter receptive-field geometry: in AlexNet conv2, surviving supports tend to shrink toward the center and often become roughly circular rather than rectangular (Lebedev et al., 2015).

Empirically, the AlexNet/ImageNet results established the practical viability of the approach. For conv2 and conv3 together, gradual sparsification reached density $1\times N$7, speedup $1\times N$8, and accuracy drop $1\times N$9. For all five convolutional layers, gradual sparsification achieved density $N$0, speedup $N$1, and accuracy drop $N$2. The authors also report that measured CPU forward time in modified Caffe is close to linear in density, which is unusually strong correspondence between theoretical and realized speedup for a pruning method (Lebedev et al., 2015).

3. Alternative kernel-group constructions

Later work broadened the design space of what counts as a kernel group. "Structured Pruning of Deep Convolutional Neural Networks" (Anwar et al., 2015) distinguishes channel-wise sparsity, kernel-wise sparsity, and intra-kernel strided sparsity. Its kernel-wise unit is one full $N$3 kernel

$N$4

that is, one connection between input channel $N$5 and output channel $N$6. Although the paper does not define a standalone “kernel group” primitive, channel pruning removes rows or columns of the source-destination kernel matrix jointly, and the intra-kernel stride/offset constraint can enforce shared sparsity structure across multiple kernels associated with one source feature map. In that sense, it treats groups of kernels either as channel-connected sets or as sets sharing a common intra-kernel support.

"1xN Pattern for Pruning Convolutional Neural Networks" (Lin et al., 2021) gives an explicit kernel-group definition. It rewrites the weight tensor as a matrix of kernels

$N$7

and defines a block

$N$8

namely $N$9 consecutive output kernels with the same input channel index. This is neither scalar pruning nor whole-filter pruning; it is block pruning in the matrix of kernels. The paper explicitly treats $g_j^l=W^l_{:,j,:,:}$0 blocks as the basic pruning granularity and shows that the resulting sparse computation can be stored in BSR and executed by block-wise vectorized kernels on CPUs (Lin et al., 2021).

"Filter Pruning using Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks" (Mitsuno et al., 2020) defines a feature-wise kernel group

$g_j^l=W^l_{:,j,:,:}$1

the set of all kernels in layer $g_j^l=W^l_{:,j,:,:}$2 connected to one input channel $g_j^l=W^l_{:,j,:,:}$3. This is a group of complete kernels spanning all output channels. The regularizer acts on these groups hierarchically, with individual kernels $g_j^l=W^l_{:,j,:,:}$4 as lower-level subgroups. When such a group becomes near-zero, the corresponding filter/output channel in layer $g_j^l=W^l_{:,j,:,:}$5 can be removed together with dependent kernels in layer $g_j^l=W^l_{:,j,:,:}$6, so direct sparsity is imposed on kernel groups but the deployed model is channel-pruned (Mitsuno et al., 2020).

REPrune (Park et al., 2024) uses yet another grouping rule. For a fixed input channel $g_j^l=W^l_{:,j,:,:}$7, it collects

$g_j^l=W^l_{:,j,:,:}$8

that is, kernels occupying the same input-channel position across different filters. These kernels are clustered by agglomerative clustering with Ward linkage. The groups are therefore similarity-based kernel clusters rather than tensor-axis slices. Filters are then retained according to maximum cluster coverage, so the method is channel pruning driven by kernel representative selection rather than direct kernel-group removal (Park et al., 2024).

The boundary of the topic is also clarified by "Kernel Cluster Pruning for Dense Labeling Neural Networks" (Yu et al., 2021). KCP prunes individual kernels $g_j^l=W^l_{:,j,:,:}$9, but its “cluster” is the entire set of kernels in a layer summarized by one centroid $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$0. Kernels closest to that centroid are pruned. This is kernel-level structured pruning, but not a multi-group kernel-pruning framework in the usual sense (Yu et al., 2021).

4. Optimization and selection strategies

Kernel-group pruning has been optimized through regularization, combinatorial search, clustering, and second-order approximations. The most classical route is structured sparsity regularization. In the lowering-aligned formulation, the $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$1 penalty over $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$2 produces coordinated shrinkage of whole groups; in the hierarchical feature-wise formulation, the structured objective

$\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$3

is combined with a hierarchical group penalty

$\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$4

so that sparsity emerges both at the level of individual kernels and at the level of channel-connected kernel groups (Lebedev et al., 2015, Mitsuno et al., 2020).

A second route is explicit search over connectivity masks. In (Anwar et al., 2015), a particle filtering framework treats each particle as a pruning mask and assigns importance $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$5 from misclassification rate on an evaluation set. Sequential importance resampling is combined with evolutionary particle filtering to search over channel-wise, kernel-wise, and intra-kernel structured masks. This is not a group-lasso method; it is a search-heavy strategy over structured connectivity patterns (Anwar et al., 2015).

A third route is kernel clustering and representative selection. KCP computes a single centroid

$\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$6

then prunes kernels with smallest distance to that centroid, iteratively increasing the pruning portion $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$7 and using soft fine-tuning so removed kernels can recover during pruning (Yu et al., 2021). REPrune instead performs bottom-up agglomerative clustering within each $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$8, derives a layer-specific cutoff $\mathcal{K}_j^l=\{\kappa_{i,j}^l\}$9, and then solves a maximum coverage problem over filters: the retained filter set $(i,j)$0 is chosen to maximize coverage of kernel clusters while respecting the target number of surviving channels (Park et al., 2024).

A fourth route is group-wise saliency derived from second-order criteria. "Group Fisher Pruning for Practical Network Compression" (Liu et al., 2021) introduces shared masks over coupled channels and derives a group importance

$(i,j)$1

where $(i,j)$2 is the set of coupled channels sharing one mask. This is a joint score: gradients are summed over the entire structural group before squaring. The paper then normalizes importance by memory reduction, arguing that practical GPU speedup correlates more strongly with memory reduction than with FLOPs reduction (Liu et al., 2021).

Across these strategies, a recurrent distinction emerges between direct kernel-group pruning and proxy methods. Direct methods delete the grouped kernels themselves, as in $(i,j)$3 or $(i,j)$4 blocks. Proxy methods use kernel groups only to decide which dense channel/filter structure should remain, as in hierarchical feature-wise pruning and REPrune (Mitsuno et al., 2020, Park et al., 2024).

5. Systems implications and hardware mapping

A defining feature of kernel-group pruning is that its grouping rule is often chosen to match an execution substrate. In (Lebedev et al., 2015), the group is selected precisely because it corresponds to one removable inner dimension of an im2col/GEMM computation. The result is a dense-but-thinner matrix product, and the reported CPU forward time scales almost linearly with density. The paper also states a clear boundary condition: the speedup argument depends on convolution being implemented by lowering plus dense GEMM, and does not map as directly to direct kernels, Winograd, FFT, or highly specialized hardware (Lebedev et al., 2015).

The systems orientation is also explicit in (Anwar et al., 2015). With shared intra-kernel stride/offset patterns, the lowering step can omit pruned entries from both feature and kernel matrices. The paper gives a concrete example in which the feature matrix size is reduced from $(i,j)$5 to $(i,j)$6, and the kernel matrix from $(i,j)$7 to $(i,j)$8. The point is not merely parameter sparsity, but reduced matrix dimensions under regular structured support (Anwar et al., 2015).

"Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning" (Yang et al., 2018) addresses a different hardware problem: how to retain fine-grained kernel sparsity while keeping enough regularity for an accelerator. Kernels are partitioned into sets $(i,j)$9, each set has a limited pattern library $s$0, and every mask in a layer preserves the same number of nonzeros $s$1. The resulting compressed sparse pattern format stores only nonzero values plus a pattern index, and the Sparse Processing Engine uses a pattern selector and only $s$2 multipliers. This is not classical kernel-group pruning of removable kernel sets, but it shows how kernel-structured sparsity can be turned into deterministic hardware datapaths (Yang et al., 2018).

"Efficient Structured Pruning and Architecture Searching for Group Convolution" (Zhao et al., 2018) makes the implementation target even more explicit by pruning a dense layer into a group-convolution-compatible block-diagonal structure after channel permutation. Important kernels are retained inside diagonal blocks, and all off-block kernels are removed. Here the “group” is the block structure imposed by group convolution, and pruning is combined with channel permutation so that the structural constraint becomes less destructive (Zhao et al., 2018).

The CPU-oriented $s$3 method in (Lin et al., 2021) is arguably the most explicit kernel-group-to-kernel mapping. Because a nonzero block consists of $s$4 consecutive kernels with the same input channel, one scalar input activation can be broadcast and multiplied against an $s$5-vector of weights. The paper stores the result in BSR and implements output computation by parallelized block-wise vectorized operations. Its main systems claim is that this intermediate granularity preserves much more vectorizability than unstructured pruning while retaining substantially finer control than filter pruning (Lin et al., 2021).

This suggests a general systems principle: kernel-group pruning is most effective when the group definition coincides with a removable or vectorizable unit in the target backend. When the grouping rule is only semantically meaningful but not execution-aligned, the resulting sparsity may remain difficult to exploit.

6. Empirical record, misconceptions, and limitations

Representative results show that kernel-group pruning can occupy a useful middle ground between unstructured pruning and coarse filter/channel pruning.

Paper	Setting	Reported result
(Lebedev et al., 2015)	AlexNet conv2+conv3	density $s$6, speedup $s$7, accuracy drop $s$8
(Mitsuno et al., 2020)	ResNet20 on CIFAR-10	$s$9 fewer parameters, $K \in \mathbb{R}^{d \times d \times S \times T}$0 drop
(Yu et al., 2021)	Dense labeling networks	more than $K \in \mathbb{R}^{d \times d \times S \times T}$1 FLOPs with less than $K \in \mathbb{R}^{d \times d \times S \times T}$2 accuracy drop
(Lin et al., 2021)	MobileNet-V2, $K \in \mathbb{R}^{d \times d \times S \times T}$3, $K \in \mathbb{R}^{d \times d \times S \times T}$4	about $K \in \mathbb{R}^{d \times d \times S \times T}$5 Top-1 improvement over filter pruning
(Park et al., 2024)	ResNet-50 on ImageNet	$K \in \mathbb{R}^{d \times d \times S \times T}$6 FLOPs reduction, $K \in \mathbb{R}^{d \times d \times S \times T}$7 Top-1

Several misconceptions recur. First, kernel-group pruning is not identical to filter pruning. The grouped unit may be one kernel-location vector across all output filters, one contiguous block of kernels, or one set of kernels connected to an input channel; these are structurally different objects (Lebedev et al., 2015, Lin et al., 2021, Mitsuno et al., 2020). Second, it is not identical to arbitrary kernel sparsity. Some methods prune complete kernels but preserve all channels; others reason over kernels but still remove only full channels/filters in the final model (Anwar et al., 2015, Park et al., 2024). Third, not every method with “kernel” in its title is a kernel-group method in the strict sense; KCP, for example, is kernel-level pruning organized around a single centroid per layer rather than a multi-group formulation (Yu et al., 2021).

The limitations are equally consistent. Deployment benefits are backend-dependent. Group-wise brain damage is strongest under lowering+GEMM and CPU forward propagation in Caffe, while its extension to efficient GPU or backpropagation support is acknowledged as nontrivial (Lebedev et al., 2015). KCP reports FLOPs reduction but not latency measurements, so its practical speedup on commodity dense libraries is less directly established (Yu et al., 2021). REPrune remains hardware-friendly precisely because it restricts itself to channel pruning in the final structure; it does not realize arbitrary kernel-level sparsity (Park et al., 2024). The particle-filter approach of (Anwar et al., 2015) is search-heavy, and its scalability to large modern CNNs is left unclear.

A broader inference, clearly suggested by this literature, is that “kernel-group pruning” is best treated as a family resemblance term rather than a single canonical operator. What unifies the family is not one tensor formula, but a design principle: choose a pruning unit at kernel granularity that is structured enough to preserve implementation regularity or architectural validity, yet fine enough to avoid the bluntness of whole-filter removal. The exact form of that unit—$K \in \mathbb{R}^{d \times d \times S \times T}$8, $K \in \mathbb{R}^{d \times d \times S \times T}$9, $F \in \mathbb{R}^{T \times d^2 S},$0, $F \in \mathbb{R}^{T \times d^2 S},$1, or a cluster in $F \in \mathbb{R}^{T \times d^2 S},$2—is what differentiates the major branches of the field (Lebedev et al., 2015, Lin et al., 2021, Mitsuno et al., 2020, Park et al., 2024).