Grouped Multi-scale Deformable Convolution

• Grouped multi-scale deformable convolution is an adaptive operator design that combines parallel, scale-specific deformable sampling paths with attention-based fusion.
• It distinguishes between canonical grouped convolution and multi-branch deformable modules by separating channel grouping from spatially adaptive branch aggregation.
• The architecture, exemplified by ASPDC, enhances tasks such as deblurring through efficient feature integration and specialized processing across varying receptive fields.

Searching arXiv for the specified papers and closely related deformable-convolution work to ground the article. Grouped multi-scale deformable convolution denotes a family of adaptive feature operators that combine deformable sampling with parallel scale-specialized processing paths. In the strict operator-design sense, the defining intuition is that multiple deformable pathways operate at different effective receptive fields and are then combined, often to address spatially variant structure, motion, or geometry. The literature summarized here shows that this label is frequently used too loosely: some methods are genuinely multi-branch and multi-scale but not grouped in the channel-partition sense, while others are deformable and adaptive but neither grouped nor explicitly multi-scale. The clearest example is Atrous Spatial Pyramid Deformable Convolution (ASPDC), introduced for blind non-uniform motion deblurring, which is best characterized as a multi-branch atrous deformable pyramid with spatial attention fusion rather than grouped deformable convolution (Huo et al., 2021).

1. Terminological scope and conceptual boundaries

The central ambiguity surrounding grouped multi-scale deformable convolution is that three distinct ideas are often conflated: grouping, multi-scale processing, and deformable sampling. In the normal CNN sense, grouping refers to channel partitioning into disjoint groups processed by separate kernel banks. Multi-scale processing refers to parallel or hierarchical operators with different receptive fields, such as distinct dilation rates or pyramid levels. Deformable sampling refers to learned displacements of sampling locations, typically parameterized by offsets and sometimes modulations.

ASPDC illustrates why these distinctions matter. It resembles grouped multi-scale deformable convolution only at a high conceptual level because several parallel branches process the same input at different effective receptive fields and then combine the results, but it does not describe any explicit channel grouping, kernel grouping, grouped offsets, or groupwise partition of feature channels (Huo et al., 2021). Instead, one input feature map is fed to four parallel branches, each branch computes a full output feature map, and the resulting full feature maps are fused spatially by attention. This suggests that the most precise description is not grouped convolution, but a parallel expert architecture over scale.

A similar terminological issue appears in Deformable PV-RCNN. That paper introduces a learned deformation mechanism inspired by deformable convolution, but it is not a true grouped multi-scale deformable convolution layer in the usual sense. The deformable operation is a learned relocation of sparse 3D keypoints before proposal refinement, rather than a convolution kernel over a regular grid, and there is no explicit grouping mechanism of the kind used in grouped deformable convolution (Bhattacharyya et al., 2020).

The same pattern recurs in multidirectional snake convolution (MDSConv) for coronary artery segmentation. MDSConv is a four-branch module consisting of one standard 3D convolution branch and three axis-specific snake convolution branches aligned with the $x$-, $y$-, and $z$-axes. It is multi-branch and multi-view, but the paper does not state that channels are split into groups, nor does it define group-specific offset fields (Yuan et al., 23 Mar 2026). Accordingly, the broader literature around the phrase grouped multi-scale deformable convolution is best read as a spectrum of related but non-equivalent designs.

2. ASPDC as a canonical multi-branch multi-scale deformable block

Within the surveyed material, ASPDC is the most explicit instance of a multi-scale deformable convolution block. It sits in the middle of a single-stage deblurring network whose full arrangement begins with two residual blocks and strided convolutions for feature extraction, contains six ASPDC modules stacked sequentially in the middle, and ends with two deconvolutions plus a final RGB projection layer (Huo et al., 2021). The output feature maps of the six ASPDC modules are further concatenated to stabilize training, so the architecture contains both intra-module and inter-module aggregation.

Each ASPDC module contains four parallel branches, called deformable modules 1–4, built with different dilation rates to generate four offset maps $\Delta p$, modulations $\Delta m$, and four deformable convolution outputs. The branch configuration is asymmetric. Module 1 uses dilation rate $1$ but ignores offsets by setting $\Delta p$ as zero; it is intended to recover static regions. Modules 2, 3, and 4 use dilation rates $1$, $2$, and $4$, respectively, and learn $y$0 and $y$1 (Huo et al., 2021). In effect, the module contains three deformable branches with different effective receptive fields plus one static branch.

No parameter sharing across branches is stated. The wording indicates four offset maps, four modulations, and four deformable convolution outputs, which strongly implies separate branch-specific predictors and branch-specific feature transforms. The ablation caption further notes that “$y$2” in Versions $y$3 represents three duplicated modules with the same dilation rate but no parameter-sharing, which is consistent with the broader design philosophy of independent branch specialization (Huo et al., 2021).

ASPDC therefore provides a useful reference point for what grouped multi-scale deformable convolution is often trying to achieve in practice: multiple deformable processing paths operating in parallel at different receptive-field scales. However, because all branches receive the same incoming feature tensor and fusion is not done by channel-group concatenation, it remains outside the usual grouped-convolution definition.

3. Fusion mechanisms: attention-weighted branch integration rather than grouping

A decisive feature of ASPDC is that the branch outputs are not merely concatenated or summed uniformly. They are fused by an Attention Feature Integration Module (AFIM), with explicit branch aggregation given by

$y$4

where $y$5 is the output of the $y$6 branch, $y$7 is a single-channel attention map for the $y$8 branch, and $y$9 is the ASPDC output (Huo et al., 2021). The attention maps are normalized by softmax along the channel so that branch weights sum to $z$0 at each spatial location. The result is a spatially varying weighted mixture of branch outputs.

This is structurally different from grouped deformable convolution. In grouped convolution, output channels are typically partitioned and concatenated by design. In ASPDC, branch selection is spatially adaptive, and different image regions can prefer different branches. The visualized attention maps show branch specialization: attention maps of small receptive fields, $z$1 and $z$2, focus more on static objects or objects with small movements, while attention maps of large receptive fields, especially $z$3, pay more attention on objects with large movements (Huo et al., 2021). A plausible implication is that ASPDC functions more like a spatially adaptive mixture-of-experts than a grouped convolution layer.

MDSConv adopts a different fusion strategy. Its four outputs—one from standard 3D convolution and three from directional snake branches—are concatenated channel-wise, normalized with Group Normalization, activated with ReLU, and fused through another convolution (Yuan et al., 23 Mar 2026). Here, too, specialization is branchwise, not groupwise. The only explicit “group” notion in the module is Group Normalization, which is not grouped convolution.

Deformable PV-RCNN provides yet another form of adaptive weighting through context gating: $z$4 This is a per-keypoint, per-channel multiplicative gate that modulates refinement features, but it is not grouped aggregation, grouped offsets, or scale-wise routing (Bhattacharyya et al., 2020). Across these cases, the recurring pattern is adaptive fusion or gating after deformable specialization, rather than channel grouping as the primary compositional principle.

4. Mathematical formulations and the role of offsets

The surveyed papers expose different levels of mathematical specificity. ASPDC introduces the offset map $z$5 and modulation $z$6, and states that different branches generate these using different dilation rates, but it does not reproduce the canonical deformable convolution formula often associated with DCN/DCNv2 (Huo et al., 2021). The only explicit branch-wise aggregation equation in that paper is the AFIM fusion rule. Thus, any branch-wise DCN equation for ASPDC would be an interpretation based on DCNv2 rather than a quoted equation from the paper.

The deblurring paper instead formalizes the underlying inverse problem and the training losses. Blur formation is modeled temporally as

$z$7

and spatially as

$z$8

The deblurring, reblurring, and consistency objectives are

$z$9

$\Delta p$0

$\Delta p$1

with $\Delta p$2 in the reported experiments (Huo et al., 2021).

Deformable PV-RCNN is more explicit about deformation itself. For each sampled keypoint $\Delta p$3, with coordinate $\Delta p$4, feature $\Delta p$5, and neighborhood $\Delta p$6, the deformation feature and aligned coordinate are

$\Delta p$7

$\Delta p$8

The offset is therefore a bounded 3D displacement, predicted per keypoint from local geometric-feature interactions (Bhattacharyya et al., 2020). This is deformable sampling, but not convolution over a regular kernel support.

MDSConv formalizes a third variant of deformation. Starting from a standard $\Delta p$9 kernel support

$\Delta m$0

it introduces directional snake branches in which deformation is recursively accumulated from the center (Yuan et al., 23 Mar 2026). For the $\Delta m$1-axis branch, for example,

$\Delta m$2

with analogous equations for the $\Delta m$3- and $\Delta m$4-axis branches. Unlike standard deformable convolution, where each kernel point usually gets an independently learned offset, MDSConv uses directional and recursively accumulated offsets to preserve continuity along tubular structures (Yuan et al., 23 Mar 2026).

5. Multi-scale, multi-view, and directional specialization

The multi-scale property of ASPDC comes primarily from the atrous spatial pyramid design inside each module. Multi-scale behavior is achieved by parallel branches with dilation rates $\Delta m$5, branch-specific offset and modulation prediction, attention-based fusion so different spatial regions can prefer different scales, and stacking six ASPDC modules sequentially (Huo et al., 2021). Within one module the scales are processed in parallel; across the network the ASPDC modules are stacked sequentially. This differs from coarse-to-fine multi-scale deblurring methods that process explicit image pyramids across stages, because the architecture is explicitly single-stage (Huo et al., 2021).

The ablation study substantiates the importance of heterogeneous receptive fields. Version 1, which contains only the no-offset static branch, gives $\Delta m$6. Adding a single additional deformable branch improves performance: Version 2 yields $\Delta m$7, Version 3 yields $\Delta m$8, and Version 4 yields $\Delta m$9. Combining multiple different branches helps further: Version 5 gives $1$0, Version 6 gives $1$1, Version 7 gives $1$2, and Version 12, which includes modules 1+2+3+4 with AFIM, achieves $1$3, the best result in the table (Huo et al., 2021). Versions 8–10 replace mixed dilation rates with duplicated same-rate modules and perform worse than Version 12, supporting the paper’s conclusion that simply duplicating modules cannot get results as good as combining different modules (Huo et al., 2021).

By contrast, MDSConv is not explicitly multi-scale at the operator level. The paper does not claim multiple kernel sizes, multiple dilation rates, or scale-specific grouped branches inside the module (Yuan et al., 23 Mar 2026). Its diversity is directional and anatomical rather than scale-wise. The four-branch structure comprises one isotropic local baseline branch and three directional deformable branches associated with sagittal, coronal, and axial views. Scale enters only at the broader architectural level through the hierarchical encoder-decoder, dense skip pathways generating multi-scale intermediate representations, and a two-stage coarse-to-fine strategy with whole-image segmentation on downsampled volume $1$4 followed by segmentation on $1$5 blocks at original resolution (Yuan et al., 23 Mar 2026).

Deformable PV-RCNN sits somewhere in between. The method is motivated by differing object scales and varying point-cloud density, and it inherits PV-RCNN’s multi-scale keypoint features corresponding to either of Conv3 or Conv4 layers, but it does not define a new explicit multi-scale deformable aggregation formula (Bhattacharyya et al., 2020). The deformable mechanism itself is single-offset-per-keypoint rather than scale-specific multi-branch deformable sampling.

6. Architectural roles and empirical behavior across domains

The practical significance of these operators emerges most clearly from their task-specific integration. In blind motion deblurring, ASPDC appears in the middle of the deblurring network, after feature extraction and before upsampling. The network learns a residual correction rather than directly predicting the sharp image, which the paper states stabilizes and accelerates training (Huo et al., 2021). The auxiliary reblurring network enforces deblurring-reblurring consistency during fine-tuning. It has two encoder-decoder branches with shared weights; the upper branch takes the concatenation of blurred and sharp images, the lower branch takes duplicated sharp images, and a convolution reduces channels to $1$6 to form a dynamic local filter for each pixel, with $1$7 for lower computational cost (Huo et al., 2021). Notably, the reported objective uses MSE losses only; there are no adversarial, perceptual, or feature losses in the reported training objective.

The paper frames ASPDC as a more efficient alternative to multi-stage methods. On $1$8 images using a single RTX 2080 Ti, the reported runtime and memory are: Ours $1$9 sec and $\Delta p$0 GB; Zhang et al. 2019 $\Delta p$1 sec and $\Delta p$2 GB; Nah et al. 2017 $\Delta p$3 sec and $\Delta p$4 GB; Tao et al. 2018 $\Delta p$5 sec and $\Delta p$6 GB; Park et al. 2019 $\Delta p$7 sec and $\Delta p$8 GB; DeblurGAN-v2 $\Delta p$9 sec and $1$0 GB (Huo et al., 2021). Within the logic of grouped multi-scale deformable designs, this suggests that parallel adaptive branches can offer a strong performance-efficiency trade-off without explicit multi-stage pyramids.

In Deformable PV-RCNN, deformability is inserted into the proposal refinement stage of a two-stage detector. Sampled keypoints are shifted to more informative locations, features are recomputed with PointNet++ similar to the PV-RCNN pipeline, and context gating modulates the refinement features (Bhattacharyya et al., 2020). The ablation on KITTI val moderate using 40 recall positions reports: baseline Car $1$1, Cyclist $1$2, Pedestrian $1$3; with deformations only, Car $1$4, Cyclist $1$5, Pedestrian $1$6; with deformations plus context fusion, Car $1$7, Cyclist $1$8, Pedestrian $1$9 (Bhattacharyya et al., 2020). The largest deformation-only gain is for pedestrians, while context fusion especially helps cyclists. Distance-based evidence at $2$0–$2$1 m shows Car $2$2 vs $2$3 and Cyclist $2$4 vs $2$5, indicating gains under sparsity (Bhattacharyya et al., 2020).

In MDSVM-UNet, MDSConv occupies the encoder, which comprises four MDSConv blocks and one bottleneck RVM block. The network channel schedule is $2$6 (Yuan et al., 23 Mar 2026). The whole model parameter count is $2$7M, compared with ImageCAS baseline $2$8M, DSU-Net $2$9M, and SwinUnet $4$0M (Yuan et al., 23 Mar 2026). The ablation table reports: baseline DSC $4$1, HD $4$2, AHD $4$3; $4$4MDSConv DSC $4$5, HD $4$6, AHD $4$7; $4$8RVM DSC $4$9, HD $y$00, AHD $y$01; $y$02MDSConv $y$03 RVM DSC $y$04, HD $y$05, AHD $y$06 (Yuan et al., 23 Mar 2026). The paper explicitly states that adding MDSConv alone improves over baseline by DSC $y$07, HD $y$08 lower, and AHD $y$09 lower.

7. Relationship to grouped multi-scale deformable convolution proper

Taken together, these papers establish a useful taxonomy. ASPDC is similar to grouped multi-scale deformable convolution in that it uses multiple deformable processing paths operating at different scales, each path can be viewed as a scale-specific deformable operator, the branches run in parallel on the same input features, and the outputs are then combined (Huo et al., 2021). However, it differs in several decisive ways: there is no channel partitioning into groups, no grouped offsets in the standard sense, the module is architecturally a parallel branch pyramid rather than a single grouped convolution operator, fusion is by spatial attention rather than group concatenation, and one branch is explicitly non-offset with $y$10 (Huo et al., 2021). The most faithful characterization is therefore a multi-branch atrous deformable pyramid with spatial attention fusion.

Deformable PV-RCNN is even further from grouped multi-scale deformable convolution. It supports the general principle that learned spatial deformation helps gather informative content and can improve refinement under sparsity, clutter, and part deformation, but it provides neither grouped channel partitioning nor explicit multi-scale deformable branches (Bhattacharyya et al., 2020). It is best understood as point-based deformable aggregation for proposal refinement.

MDSConv offers a different neighboring design pattern. It has several specialized sub-operators running in parallel, each capturing a different geometric aspect, and outputs are fused downstream. This resembles grouped or multi-head specialization only loosely. Yet the paper does not partition channels into groups, does not define offsets per group, and does not present the operator as grouped convolution in the usual implementation sense (Yuan et al., 23 Mar 2026). Nor is it multi-scale at the operator level; its diversity is view-directional, not scale-wise.

A common misconception is therefore to treat any parallel deformable operator as grouped multi-scale deformable convolution. The surveyed evidence does not support that equivalence. Grouped multi-scale deformable convolution, in the strict sense, would require explicit channel-group decomposition or analogous grouped parameterization together with scale-differentiated deformable sampling. Among the works considered here, ASPDC comes closest in functional spirit but not in literal grouped design (Huo et al., 2021). MDSConv is best read as multi-branch, multi-view, directional deformable convolution (Yuan et al., 23 Mar 2026), and Deformable PV-RCNN as deformable keypoint refinement (Bhattacharyya et al., 2020).

The resulting picture is that grouped multi-scale deformable convolution is less a single settled operator than a design space. One axis concerns whether specialization is by scale, direction, or instance geometry; another concerns whether branches are true groups of channels or full parallel experts; a third concerns whether fusion is fixed, gated, or attention-weighted. The strongest evidence in the present corpus supports multi-branch specialization with adaptive fusion, but not the claim that grouping in the canonical convolutional sense is necessary for effective deformable multi-scale behavior.