Multi-Kernel Depth-Wise Convolution Block

• MKDC is a design pattern that replaces single-kernel depthwise convolutions with multiple kernels to capture multi-scale spatial features efficiently.
• It has been empirically validated across image classification, object detection, and medical segmentation tasks while balancing accuracy and computational cost.
• MKDC implementations vary from channel partitioning to full parallel branches with different fusion techniques, offering flexible integration into modern CNN architectures.

Searching arXiv for the cited MKDC-related papers to ground the article in the source literature. Multi-Kernel Depth-Wise Convolution Block (MKDC) denotes a family of depthwise-convolution operators that replace a single spatial kernel with multiple kernel sizes so that a layer can encode multi-scale spatial structure within an efficient separable-convolution regime. In the literature represented by MixConv, Depthwise Multiception Convolution, and MK-UNet, the common principle is to apply depthwise convolutions at several receptive fields and then fuse the resulting feature maps through concatenation or summation, sometimes followed by pointwise mixing or channel shuffle (Tan et al., 2019). The term does not refer to a single canonical micro-architecture; rather, it names a design pattern whose concrete realization varies across papers, particularly in how channels are assigned to kernels, how branch outputs are merged, and whether the block is embedded inside an inverted residual or U-shaped segmentation backbone (Bao et al., 2020).

1. Origins and conceptual scope

The modern formulation of the idea appears in "MixConv: Mixed Depthwise Convolutional Kernels" (Tan et al., 2019), which systematically studies kernel size in depthwise convolution and reports that combining the benefits of multiple kernel sizes can lead to better accuracy and efficiency. In that work, mixed depthwise convolution is introduced as a simple drop-in replacement of vanilla depthwise convolution, and the authors further integrate it into an AutoML search space to build MixNets. The same general principle is later instantiated in different forms: Depthwise Multiception Convolution uses parallel depthwise branches with kernels $\{3,5,7\}$ followed by concatenation and a $1\times1$ convolution (Bao et al., 2020), whereas MK-UNet defines MKDC as parallel depth-wise convolutions with $K=\{1,3,5\}$, branch-wise BN and ReLU6, element-wise summation, and channel-shuffle (Rahman et al., 23 Sep 2025).

These formulations share a common objective: to improve the representation of multi-resolution spatial structure while retaining the low parameter count associated with depthwise separable operators. A plausible implication is that MKDC is best understood as a reusable operator class rather than a single block definition. This interpretation is directly supported by the supplied technical summary, which abstracts and generalizes the MixConv idea into a reusable MKDC block with explicit mathematical formulation, implementation details, design choices, empirical results, and an integration recipe.

2. Formal definitions and principal variants

A generalized split-channel formulation begins with an input feature map

$X \in \mathbb{R}^{H\times W\times C},$

whose channels are partitioned into $G$ groups with counts $c_1,\dots,c_G$ satisfying

$\sum_{g=1}^G c_g = C.$

If $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$ denotes the $g$-th slice and each group is assigned a depthwise kernel $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$, then the per-group output is

$1\times1$0

and the final output is

$1\times1$1

In compact form,

$1\times1$2

This is the generalized MKDC description derived from MixConv-style channel partitioning (Tan et al., 2019).

Depthwise Multiception adopts a different but closely related formulation. Let $1\times1$3 and let $1\times1$4 denote depthwise kernels for $1\times1$5. Each branch computes

$1\times1$6

the outputs are concatenated,

$1\times1$7

then normalized and mixed by a pointwise convolution to produce $1\times1$8 (Bao et al., 2020).

MK-UNet defines the per-branch depth-wise convolution block as

$1\times1$9

and the MKDC operator as

$K=\{1,3,5\}$0

where $K=\{1,3,5\}$1 is channel shuffle and $K=\{1,3,5\}$2 in the reported experiments. Inside the multi-kernel inverted residual,

$K=\{1,3,5\}$3

This makes explicit that MKDC can function either as a stand-alone mixed depthwise operator or as the central spatial transform within a broader residual bottleneck (Rahman et al., 23 Sep 2025).

Variant	Kernel handling	Fusion mechanism
MixConv / generalized MKDC	Split channels across groups with different $K=\{1,3,5\}$4	Concatenation along channel axis
Depthwise Multiception	Parallel branches for $K=\{1,3,5\}$5 on all channels	Concatenation, BN, then $K=\{1,3,5\}$6 convolution
MK-UNet MKDC	Parallel branches for $K=\{1,3,5\}$7 on all channels	Element-wise sum, then channel-shuffle

The main technical distinction is therefore not whether multiple kernels are used, but whether multi-scale processing is realized by channel partitioning or by full-channel parallel branches.

3. Computational characteristics

For the generalized split-channel MKDC, the parameter count is

$K=\{1,3,5\}$8

and the FLOPs, assuming output spatial size $K=\{1,3,5\}$9, are

$X \in \mathbb{R}^{H\times W\times C},$0

If a multiply-add is counted as two FLOPs, the technical summary specifies multiplying $X \in \mathbb{R}^{H\times W\times C},$1 by $X \in \mathbb{R}^{H\times W\times C},$2 (Tan et al., 2019).

For Depthwise Multiception, with $X \in \mathbb{R}^{H\times W\times C},$3 kernel sizes $X \in \mathbb{R}^{H\times W\times C},$4, the total parameter count is stated as

$X \in \mathbb{R}^{H\times W\times C},$5

and the total MACs as

$X \in \mathbb{R}^{H\times W\times C},$6

In the default setting $X \in \mathbb{R}^{H\times W\times C},$7 with $X \in \mathbb{R}^{H\times W\times C},$8, $X \in \mathbb{R}^{H\times W\times C},$9, $G$0, the paper gives $G$1 (Bao et al., 2020).

MK-UNet reports the complexity of its specific instantiation inside MKIR. With expansion ratio $G$2, $G$3 upsamples $G$4, MKDC operates on $G$5 channels, and the point-wise convolutions $G$6 and $G$7 each cost $G$8 parameters total. The depth-wise parameters in MKDC are

$G$9

so the total MKDC parameters per stage, excluding BN’s learned $c_1,\dots,c_G$0, are $c_1,\dots,c_G$1, and the total MKIR parameters per stage are approximately

$c_1,\dots,c_G$2

For the example $c_1,\dots,c_G$3, the reported counts are $c_1,\dots,c_G$4 parameters for $c_1,\dots,c_G$5, $c_1,\dots,c_G$6 for the depth-wise multi-kernel stage, and $c_1,\dots,c_G$7 for $c_1,\dots,c_G$8, for a total of approximately $c_1,\dots,c_G$9 k parameters plus a handful for BN. For one $\sum_{g=1}^G c_g = C.$0 feature-map, the paper specifies depth-wise FLOPs as

$\sum_{g=1}^G c_g = C.$1

and point-wise FLOPs as

$\sum_{g=1}^G c_g = C.$2

for both $\sum_{g=1}^G c_g = C.$3 and $\sum_{g=1}^G c_g = C.$4 (Rahman et al., 23 Sep 2025).

Across these variants, the recurring computational property is that MKDC increases spatial-kernel diversity without introducing a full dense $\sum_{g=1}^G c_g = C.$5 convolution. This suggests that the efficiency gains arise from preserving the channel-wise factorization of depthwise convolution while broadening the set of receptive fields available to each layer.

4. Design parameters and implementation patterns

The generalized MKDC design guide identifies three channel splitting strategies. The equal split sets $\sum_{g=1}^G c_g = C.$6 or $\sum_{g=1}^G c_g = C.$7. The exponential, or inverse-$\sum_{g=1}^G c_g = C.$8, split assigns more channels to small kernels using $\sum_{g=1}^G c_g = C.$9, then normalizes so that $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$0. A search-based strategy treats $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$1 as hyperparameters in an AutoML search (Tan et al., 2019).

The same guide states that each group can use its own $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$2, although stride and dilation are typically shared across groups. Each $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$3 is padded so that spatial output sizes match, and outputs are concatenated along the channel axis. In the design choices, small sets such as $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$4 with $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$5 trade off minimal overhead for slightly larger context, whereas larger sets such as $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$6 or $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$7 capture multi-scale patterns more richly but increase parameter/FLOP diversity and kernel overhead. The guide states that, empirically, $X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$8–$X^{(g)} \in \mathbb{R}^{H\times W\times c_g}$9 groups strike a good balance (Tan et al., 2019).

Depthwise Multiception gives an explicit implementation pattern for the full-channel parallel-branch variant. For each kernel in $g$0, a depthwise convolution is applied with groups equal to the number of input channels and padding $g$1 to preserve spatial size; the branch outputs are concatenated, followed by BatchNorm2d, a $g$2 convolution, and another BatchNorm2d. The paper also states a configurable layer-based kernel scheduling policy for large networks: the first third of convolution layers use all three kernels $g$3, middle layers may reduce to two kernel sizes, and the final third may use only $g$4, degenerating back to standard DSConv (Bao et al., 2020).

MK-UNet fixes $g$5, uses branch-wise BN and ReLU6, sums branch outputs element-wise, and then applies channel-shuffle. The block is placed inside MKIR, where $g$6 expands channels by factor $g$7 and $g$8 projects back to the original channel count. Within the U-shaped encoder, each downsampling stage is a MaxPool followed by one MKIR, while the decoder mirrors this with upsampling, skip-gate, and MKIRA, an attention-enhanced version of MKIR (Rahman et al., 23 Sep 2025).

A common misconception is that MKDC necessarily means concatenation of multi-kernel branches. The cited literature does not support that restriction. Concatenation is central in MixConv-style channel partitioning and in Depthwise Multiception, whereas MK-UNet uses element-wise summation followed by channel-shuffle. Another common misconception is that kernel multiplicity alone defines the block. The literature instead treats fusion, channel allocation, and surrounding pointwise transforms as equally consequential design choices.

5. Empirical behavior across tasks

On ImageNet classification with a MobileNetV2 backbone, the MixConv-based summary reports the following comparisons. The baseline $g$9 depthwise configuration has $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$0 M FLOPs, $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$1 M parameters, and $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$2 top-1. A $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$3-only variant has approximately $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$4 M FLOPs, $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$5 M parameters, and $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$6. MixConv $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$7 with equal split has approximately $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$8 M FLOPs, $$K^{(g)} \in \mathbb{R}^{k_g\times k_g\times c_g\times m}$$9 M parameters, and $1\times1$00. MixConv $1\times1$01 has approximately $1\times1$02 M FLOPs, $1\times1$03 M parameters, and $1\times1$04 (Tan et al., 2019).

For COCO detection with SSD plus MobileNet, the same source reports that MobileNetV1-based SSD with $1\times1$05 depthwise convolution yields $1\times1$06 B FLOPs, $1\times1$07 M parameters, and $1\times1$08. A $1\times1$09 configuration gives $1\times1$10 B FLOPs, $1\times1$11 M parameters, and $1\times1$12. MixConv $1\times1$13 gives $1\times1$14 B FLOPs, $1\times1$15 M parameters, and $1\times1$16. A $1\times1$17 configuration gives $1\times1$18 B FLOPs, $1\times1$19 M parameters, and $1\times1$20. MixConv $1\times1$21 gives $1\times1$22 B FLOPs, $1\times1$23 M parameters, and $1\times1$24 (Tan et al., 2019).

The same paper also reports AutoML-found MixNets: MixNet-S at $1\times1$25 M FLOPs achieves $1\times1$26 top-1 versus ProxylessNAS $1\times1$27; MixNet-M at $1\times1$28 M FLOPs achieves $1\times1$29 versus FBNet-C $1\times1$30; and MixNet-L at $1\times1$31 M FLOPs achieves $1\times1$32 versus MobileNetV2 $1\times1$33 at $1\times1$34 (Tan et al., 2019).

Depthwise Multiception reports improvements across Cifar-10, Cifar-100, STL-10, and ImageNet32x32. On Cifar-10, the average Top-1 error reduction is approximately $1\times1$35 versus DSConv and approximately $1\times1$36 versus MixConv. On Cifar-100, the corresponding reductions are approximately $1\times1$37 and approximately $1\times1$38. On STL-10, ResNet-50 records $1\times1$39 error versus $1\times1$40 for DSConv and $1\times1$41 for MixConv, and MobileNet-V2 records $1\times1$42 versus $1\times1$43 for DSConv. On ImageNet32$1\times1$4432, ResNet-50 records $1\times1$45 error versus $1\times1$46 for DSConv and $1\times1$47 for MixConv (Bao et al., 2020).

In medical image segmentation, MK-UNet attributes its performance to MKDC together with attention mechanisms and the broader MKIR/MKIRA design. The network is reported to use only $1\times1$48 M parameters and $1\times1$49 G FLOPs, and to provide higher accuracy over state-of-the-art methods across six binary medical imaging benchmarks. Relative comparisons given in the abstract include nearly $1\times1$50 and $1\times1$51 fewer parameters and FLOPs than TransUNet, respectively, and up to $1\times1$52 higher DICE than UNeXt with $1\times1$53 fewer parameters. The paper further states that the full model achieves average DICE $1\times1$54 over six datasets (Rahman et al., 23 Sep 2025).

Taken together, these results indicate that MKDC has been empirically validated in image classification, object detection, and medical image segmentation. A plausible implication is that the block’s utility is not tied to a single task domain, provided that the surrounding architecture can exploit multi-scale local context.

6. Ablations, interpretation, and limitations

The MixConv design guide reports several ablations. Single-layer replacement shows that large kernels improve some layers, such as stride-2 layers, but hurt others; mixed kernels stabilize layer-wise performance. Exponential and equal partition achieve similar accuracy, while the exponential split saves a few parameters. Dilated $1\times1$55 with large dilation underperforms true large kernels because it skips local context (Tan et al., 2019).

Depthwise Multiception reports that the full $1\times1$56 kernel set gives the best overall accuracy across four datasets when compared with two-kernel variants $1\times1$57, $1\times1$58, and $1\times1$59. It also reports an average parameter saving of $1\times1$60 versus standard convolution with an average accuracy difference of approximately $1\times1$61, described as no loss, and individual models with $1\times1$62–$1\times1$63 fewer parameters and same or slightly better accuracy on large datasets (Bao et al., 2020).

MK-UNet provides direct ablations for the impact of its MKDC-centered inverted residual. On BUSI, IRB gives $1\times1$64 and MKIR with MKDC gives $1\times1$65, a gain of $1\times1$66. For kernel-set ablation, a single $1\times1$67 branch yields $1\times1$68 with $1\times1$69 M parameters and $1\times1$70 G FLOPs, whereas $1\times1$71 yields $1\times1$72 with $1\times1$73 M parameters and $1\times1$74 G FLOPs. The paper also states that adding MKDC in MKIR plus gated-attention gate and MKIRA yields the best BUSI $1\times1$75 versus $1\times1$76 for the original UNeXt baseline, with $1\times1$77 M versus $1\times1$78 M parameters (Rahman et al., 23 Sep 2025).

These ablations clarify several points that are sometimes obscured in informal discussion. First, MKDC is not simply “a larger kernel at lower cost”; the evidence emphasizes multi-kernel composition rather than monotonic kernel enlargement. Second, the best kernel set is architecture- and task-dependent: MixConv examples highlight $1\times1$79 and $1\times1$80, Depthwise Multiception uses $1\times1$81, and MK-UNet uses $1\times1$82. Third, fusion choice matters. This suggests that the gains arise from the interaction between receptive-field diversity and the specific mixing mechanism rather than from kernel multiplicity in isolation.

7. Integration into neural architectures

The generalized integration recipe for MKDC is explicit. The first step is to identify vanilla depthwise convolution layers in a network, for example in inverted bottlenecks. The second step is to choose a kernel set $1\times1$83; the summary gives $1\times1$84 with $1\times1$85 as a common choice. The third step is to choose the channel split $1\times1$86, with equal split as the simplest option and $1\times1$87 suggested for more efficiency. The fourth step is to replace each depthwise layer by MKDC while preserving stride and dilation. The fifth step is to re-tune batch-normalization and activation hyperparameters if needed; training from scratch with standard schedules generally converges in the same number of epochs. The sixth step is optional inclusion of both $1\times1$88 and the $1\times1$89 ratios in an AutoML search space. The final step is validation using top-1 accuracy or mAP, FLOPs, and end-to-end latency (Tan et al., 2019).

The same source provides a budget-driven heuristic. Given target FLOPs $1\times1$90, one solves for $1\times1$91 subject to

$1\times1$92

For simplicity, letting

$1\times1$93

the assignment can be made as

$1\times1$94

where $1\times1$95 (Tan et al., 2019).

In MK-UNet, integration is more structurally constrained: MKDC sits inside MKIR, itself positioned after MaxPool in the encoder, and in an attention-enhanced form, MKIRA, in the decoder. In this setting, the function of MKDC is not only to replace a depthwise operator but to define the central multi-resolution transform within the U-shaped backbone (Rahman et al., 23 Sep 2025).

Latency considerations are also reported. The MixConv summary states that on TPU/GPU, MixConv shows approximately $1\times1$96–$1\times1$97 speed-up compared to same-FLOPs larger-kernel depthwise convolution, and that grouping overhead is minimal when implemented as fused grouped convolutions (Tan et al., 2019). This suggests that practical deployment depends not only on FLOP counts but also on whether the implementation maps efficiently to the target hardware kernel library.

MKDC therefore occupies a specific position within efficient CNN design: it preserves the parameter-light character of depthwise processing while extending receptive-field diversity through multi-kernel composition. The literature shows that this principle can be realized by split-channel mixed kernels, by full-channel parallel depthwise branches plus pointwise fusion, or by branch summation plus channel-shuffle. What remains constant across these variants is the attempt to improve accuracy per computational budget by embedding multiple spatial scales directly inside the depthwise stage (Tan et al., 2019).