---
title: Grouped Multi-scale Deformable Convolution
url: https://www.emergentmind.com/topics/grouped-multi-scale-deformable-convolution
type: topic
---

# Grouped Multi-scale Deformable Convolution

Searching arXiv for the specified papers and closely related deformable-convolution work to ground the article.
Grouped multi-scale deformable convolution denotes a family of adaptive feature operators that combine deformable sampling with parallel scale-specialized processing paths. In the strict operator-design sense, the defining intuition is that multiple deformable pathways operate at different effective receptive fields and are then combined, often to address spatially variant structure, motion, or geometry. The literature summarized here shows that this label is frequently used too loosely: some methods are genuinely multi-branch and multi-scale but not grouped in the channel-partition sense, while others are deformable and adaptive but neither grouped nor explicitly multi-scale. The clearest example is Atrous Spatial Pyramid Deformable Convolution (ASPDC), introduced for blind non-uniform motion deblurring, which is best characterized as a multi-branch atrous deformable pyramid with spatial attention fusion rather than grouped deformable convolution [2106.14336].

## 1. Terminological scope and conceptual boundaries

The central ambiguity surrounding grouped multi-scale deformable convolution is that three distinct ideas are often conflated: grouping, multi-scale processing, and deformable sampling. In the normal CNN sense, grouping refers to channel partitioning into disjoint groups processed by separate kernel banks. Multi-scale processing refers to parallel or hierarchical operators with different receptive fields, such as distinct dilation rates or pyramid levels. Deformable sampling refers to learned displacements of sampling locations, typically parameterized by offsets and sometimes modulations.

ASPDC illustrates why these distinctions matter. It resembles grouped multi-scale deformable convolution only at a high conceptual level because several parallel branches process the same input at different effective receptive fields and then combine the results, but it does **not** describe any explicit channel grouping, kernel grouping, grouped offsets, or groupwise partition of feature channels [2106.14336]. Instead, one input feature map is fed to four parallel branches, each branch computes a full output feature map, and the resulting full feature maps are fused spatially by attention. This suggests that the most precise description is not grouped convolution, but a parallel expert architecture over scale.

A similar terminological issue appears in Deformable PV-RCNN. That paper introduces a learned deformation mechanism inspired by deformable convolution, but it is not a true grouped multi-scale deformable convolution layer in the usual sense. The deformable operation is a learned relocation of sparse 3D keypoints before proposal refinement, rather than a convolution kernel over a regular grid, and there is no explicit grouping mechanism of the kind used in grouped deformable convolution [2008.08766].

The same pattern recurs in multidirectional snake convolution (MDSConv) for coronary artery segmentation. MDSConv is a four-branch module consisting of one standard 3D convolution branch and three axis-specific snake convolution branches aligned with the \(x\)-, \(y\)-, and \(z\)-axes. It is multi-branch and multi-view, but the paper does not state that channels are split into groups, nor does it define group-specific offset fields [2603.21829]. Accordingly, the broader literature around the phrase grouped multi-scale deformable convolution is best read as a spectrum of related but non-equivalent designs.

## 2. ASPDC as a canonical multi-branch multi-scale deformable block

Within the surveyed material, ASPDC is the most explicit instance of a multi-scale deformable convolution block. It sits in the middle of a single-stage deblurring network whose full arrangement begins with two residual blocks and strided convolutions for feature extraction, contains six ASPDC modules stacked sequentially in the middle, and ends with two deconvolutions plus a final RGB projection layer [2106.14336]. The output feature maps of the six ASPDC modules are further concatenated to stabilize training, so the architecture contains both intra-module and inter-module aggregation.

Each ASPDC module contains four parallel branches, called deformable modules 1–4, built with different dilation rates to generate four offset maps \(\Delta p\), modulations \(\Delta m\), and four deformable convolution outputs. The branch configuration is asymmetric. Module 1 uses dilation rate \(1\) but ignores offsets by setting \(\Delta p\) as zero; it is intended to recover static regions. Modules 2, 3, and 4 use dilation rates \(1\), \(2\), and \(4\), respectively, and learn \(\Delta p\) and \(\Delta m\) [2106.14336]. In effect, the module contains three deformable branches with different effective receptive fields plus one static branch.

No parameter sharing across branches is stated. The wording indicates four offset maps, four modulations, and four deformable convolution outputs, which strongly implies separate branch-specific predictors and branch-specific feature transforms. The ablation caption further notes that “\(\times 3\)” in Versions \(8\sim10\) represents three duplicated modules with the same dilation rate but no parameter-sharing, which is consistent with the broader design philosophy of independent branch specialization [2106.14336].

ASPDC therefore provides a useful reference point for what grouped multi-scale deformable convolution is often trying to achieve in practice: multiple deformable processing paths operating in parallel at different receptive-field scales. However, because all branches receive the same incoming feature tensor and fusion is not done by channel-group concatenation, it remains outside the usual grouped-convolution definition.

## 3. Fusion mechanisms: attention-weighted branch integration rather than grouping

A decisive feature of ASPDC is that the branch outputs are not merely concatenated or summed uniformly. They are fused by an Attention Feature Integration Module (AFIM), with explicit branch aggregation given by
\[
f_o= \sum_{i = 1}^{4} a_i * f_i,\ \sum_{i = 1}^{4} a_{ij} = 1, 1 \leq j \leq h \times w,
\]
where \(f_i\) is the output of the \(i^{th}\) branch, \(a_i\) is a single-channel attention map for the \(i^{th}\) branch, and \(f_o\) is the ASPDC output [2106.14336]. The attention maps are normalized by softmax along the channel so that branch weights sum to \(1\) at each spatial location. The result is a spatially varying weighted mixture of branch outputs.

This is structurally different from grouped deformable convolution. In grouped convolution, output channels are typically partitioned and concatenated by design. In ASPDC, branch selection is spatially adaptive, and different image regions can prefer different branches. The visualized attention maps show branch specialization: attention maps of small receptive fields, \(a_1\) and \(a_2\), focus more on static objects or objects with small movements, while attention maps of large receptive fields, especially \(a_4\), pay more attention on objects with large movements [2106.14336]. A plausible implication is that ASPDC functions more like a spatially adaptive mixture-of-experts than a grouped convolution layer.

MDSConv adopts a different fusion strategy. Its four outputs—one from standard 3D convolution and three from directional snake branches—are concatenated channel-wise, normalized with Group Normalization, activated with ReLU, and fused through another convolution [2603.21829]. Here, too, specialization is branchwise, not groupwise. The only explicit “group” notion in the module is Group Normalization, which is not grouped convolution.

Deformable PV-RCNN provides yet another form of adaptive weighting through context gating:
\[
g = \sigma(W_{\text{gate} f_i + b_{\text{gate}), \qquad f_i^g = g \odot W_{\text{fc}f_i.
\]
This is a per-keypoint, per-channel multiplicative gate that modulates refinement features, but it is not grouped aggregation, grouped offsets, or scale-wise routing [2008.08766]. Across these cases, the recurring pattern is adaptive fusion or gating after deformable specialization, rather than channel grouping as the primary compositional principle.

## 4. Mathematical formulations and the role of offsets

The surveyed papers expose different levels of mathematical specificity. ASPDC introduces the offset map \(\Delta p\) and modulation \(\Delta m\), and states that different branches generate these using different dilation rates, but it does **not** reproduce the canonical deformable convolution formula often associated with DCN/DCNv2 [2106.14336]. The only explicit branch-wise aggregation equation in that paper is the AFIM fusion rule. Thus, any branch-wise DCN equation for ASPDC would be an interpretation based on DCNv2 rather than a quoted equation from the paper.

The deblurring paper instead formalizes the underlying inverse problem and the training losses. Blur formation is modeled temporally as
\[
I_b= g\left(\frac{1}{T}\int_{t=0}^{T}I_{S(t)}dt\right),
\]
and spatially as
\[
I_b= I_s \circledast k + n.
\]
The deblurring, reblurring, and consistency objectives are
\[
L_{deblurring} = || I_s - I_d ||^2_F,
\]
\[
L_{reblurring} = || I_r - I_b ||^2_F,
\]
\[
L_{consistency} =L_{deblurring} + \lambda L_{reblurring},
\]
with \(\lambda = 0.1\) in the reported experiments [2106.14336].

Deformable PV-RCNN is more explicit about deformation itself. For each sampled keypoint \(i\), with coordinate \(v_i \in \mathbb{R}^3\), feature \(f_i\), and neighborhood \(\mathcal{N}(i)\), the deformation feature and aligned coordinate are
\[
f'_i = \frac{1}{n}\operatorname{ReLU}\left(\sum_{j \in \mathcal{N}(i)} W_{\text{offset}(f_i-f_j)\cdot(v_i-v_j)\right),
\]
\[
v'_i = v_i + \tanh\!\left(W_{\text{align}[f'_i]\right).
\]
The offset is therefore a bounded 3D displacement, predicted per keypoint from local geometric-feature interactions [2008.08766]. This is deformable sampling, but not convolution over a regular kernel support.

MDSConv formalizes a third variant of deformation. Starting from a standard \(3 \times 3 \times 3\) kernel support
\[
\mathcal{K} = \{(x-1, y-1, z-1), \ldots, (x+1, y+1, z+1)\},
\]
it introduces directional snake branches in which deformation is recursively accumulated from the center [2603.21829]. For the \(x\)-axis branch, for example,
\[
\begin{cases}
(x_{i+c}, y_{i+c}, z_{i+c}) = (x_{i+c}, y_i + \sum_{i}^{i+c} \Delta y, z_i + \sum_{i}^{i+c} \Delta z) \\
(x_{i-c}, y_{i-c}, z_{i-c}) = (x_{i-c}, y_i + \sum_{i}^{i-c} \Delta y, z_i + \sum_{i}^{i-c} \Delta z)
\end{cases}
\]
with analogous equations for the \(y\)- and \(z\)-axis branches. Unlike standard deformable convolution, where each kernel point usually gets an independently learned offset, MDSConv uses directional and recursively accumulated offsets to preserve continuity along tubular structures [2603.21829].

## 5. Multi-scale, multi-view, and directional specialization

The multi-scale property of ASPDC comes primarily from the atrous spatial pyramid design inside each module. Multi-scale behavior is achieved by parallel branches with dilation rates \((1,1,2,4)\), branch-specific offset and modulation prediction, attention-based fusion so different spatial regions can prefer different scales, and stacking six ASPDC modules sequentially [2106.14336]. Within one module the scales are processed in parallel; across the network the ASPDC modules are stacked sequentially. This differs from coarse-to-fine multi-scale deblurring methods that process explicit image pyramids across stages, because the architecture is explicitly single-stage [2106.14336].

The ablation study substantiates the importance of heterogeneous receptive fields. Version 1, which contains only the no-offset static branch, gives \(30.24 / 0.942\). Adding a single additional deformable branch improves performance: Version 2 yields \(30.65 / 0.944\), Version 3 yields \(30.94 / 0.948\), and Version 4 yields \(30.85 / 0.947\). Combining multiple different branches helps further: Version 5 gives \(31.82 / 0.957\), Version 6 gives \(31.73 / 0.956\), Version 7 gives \(31.53 / 0.954\), and Version 12, which includes modules 1+2+3+4 with AFIM, achieves \(32.12 / 0.959\), the best result in the table [2106.14336]. Versions 8–10 replace mixed dilation rates with duplicated same-rate modules and perform worse than Version 12, supporting the paper’s conclusion that simply duplicating modules cannot get results as good as combining different modules [2106.14336].

By contrast, MDSConv is not explicitly multi-scale at the operator level. The paper does **not** claim multiple kernel sizes, multiple dilation rates, or scale-specific grouped branches inside the module [2603.21829]. Its diversity is directional and anatomical rather than scale-wise. The four-branch structure comprises one isotropic local baseline branch and three directional deformable branches associated with sagittal, coronal, and axial views. Scale enters only at the broader architectural level through the hierarchical encoder-decoder, dense skip pathways generating multi-scale intermediate representations, and a two-stage coarse-to-fine strategy with whole-image segmentation on downsampled volume \(128 \times 128 \times 64\) followed by segmentation on \(64\times64\times64\) blocks at original resolution [2603.21829].

Deformable PV-RCNN sits somewhere in between. The method is motivated by differing object scales and varying point-cloud density, and it inherits PV-RCNN’s multi-scale keypoint features corresponding to either of Conv3 or Conv4 layers, but it does not define a new explicit multi-scale deformable aggregation formula [2008.08766]. The deformable mechanism itself is single-offset-per-keypoint rather than scale-specific multi-branch deformable sampling.

## 6. Architectural roles and empirical behavior across domains

The practical significance of these operators emerges most clearly from their task-specific integration. In blind motion deblurring, ASPDC appears in the middle of the deblurring network, after feature extraction and before upsampling. The network learns a residual correction rather than directly predicting the sharp image, which the paper states stabilizes and accelerates training [2106.14336]. The auxiliary reblurring network enforces deblurring-reblurring consistency during fine-tuning. It has two encoder-decoder branches with shared weights; the upper branch takes the concatenation of blurred and sharp images, the lower branch takes duplicated sharp images, and a convolution reduces channels to \(K\times K\) to form a dynamic local filter for each pixel, with \(K=3\) for lower computational cost [2106.14336]. Notably, the reported objective uses MSE losses only; there are no adversarial, perceptual, or feature losses in the reported training objective.

The paper frames ASPDC as a more efficient alternative to multi-stage methods. On \(1280\times720\) images using a single RTX 2080 Ti, the reported runtime and memory are: Ours \(0.28\) sec and \(2.25\) GB; Zhang et al. 2019 \(0.40\) sec and \(2.10\) GB; Nah et al. 2017 \(0.93\) sec and \(9.70\) GB; Tao et al. 2018 \(0.78\) sec and \(6.09\) GB; Park et al. 2019 \(0.05\) sec and \(8.49\) GB; DeblurGAN-v2 \(1.68\) sec and \(2.41\) GB [2106.14336]. Within the logic of grouped multi-scale deformable designs, this suggests that parallel adaptive branches can offer a strong performance-efficiency trade-off without explicit multi-stage pyramids.

In Deformable PV-RCNN, deformability is inserted into the proposal refinement stage of a two-stage detector. Sampled keypoints are shifted to more informative locations, features are recomputed with PointNet++ similar to the PV-RCNN pipeline, and context gating modulates the refinement features [2008.08766]. The ablation on KITTI val moderate using 40 recall positions reports: baseline Car \(84.20\), Cyclist \(69.65\), Pedestrian \(54.49\); with deformations only, Car \(84.24\), Cyclist \(70.21\), Pedestrian \(57.31\); with deformations plus context fusion, Car \(84.71\), Cyclist \(73.03\), Pedestrian \(57.65\) [2008.08766]. The largest deformation-only gain is for pedestrians, while context fusion especially helps cyclists. Distance-based evidence at \(30\)–\(50\) m shows Car \(52.02\) vs \(50.00\) and Cyclist \(47.00\) vs \(35.15\), indicating gains under sparsity [2008.08766].

In MDSVM-UNet, MDSConv occupies the encoder, which comprises four MDSConv blocks and one bottleneck RVM block. The network channel schedule is \([16, 32, 64, 128, 256]\) [2603.21829]. The whole model parameter count is \(26.7\)M, compared with ImageCAS baseline \(27.6\)M, DSU-Net \(29.0\)M, and SwinUnet \(256.3\)M [2603.21829]. The ablation table reports: baseline DSC \(0.7824\), HD \(36.3886\), AHD \(1.7116\); \(+\)MDSConv DSC \(0.8241\), HD \(27.5515\), AHD \(0.9046\); \(+\)RVM DSC \(0.8236\), HD \(27.6082\), AHD \(0.9192\); \(+\)MDSConv \(+\) RVM DSC \(0.8365\), HD \(27.8430\), AHD \(0.9023\) [2603.21829]. The paper explicitly states that adding MDSConv alone improves over baseline by DSC \(+4.17\%\), HD \(8.8371\) lower, and AHD \(0.8070\) lower.

## 7. Relationship to grouped multi-scale deformable convolution proper

Taken together, these papers establish a useful taxonomy. ASPDC is similar to grouped multi-scale deformable convolution in that it uses multiple deformable processing paths operating at different scales, each path can be viewed as a scale-specific deformable operator, the branches run in parallel on the same input features, and the outputs are then combined [2106.14336]. However, it differs in several decisive ways: there is no channel partitioning into groups, no grouped offsets in the standard sense, the module is architecturally a parallel branch pyramid rather than a single grouped convolution operator, fusion is by spatial attention rather than group concatenation, and one branch is explicitly non-offset with \(\Delta p=0\) [2106.14336]. The most faithful characterization is therefore a multi-branch atrous deformable pyramid with spatial attention fusion.

Deformable PV-RCNN is even further from grouped multi-scale deformable convolution. It supports the general principle that learned spatial deformation helps gather informative content and can improve refinement under sparsity, clutter, and part deformation, but it provides neither grouped channel partitioning nor explicit multi-scale deformable branches [2008.08766]. It is best understood as point-based deformable aggregation for proposal refinement.

MDSConv offers a different neighboring design pattern. It has several specialized sub-operators running in parallel, each capturing a different geometric aspect, and outputs are fused downstream. This resembles grouped or multi-head specialization only loosely. Yet the paper does not partition channels into groups, does not define offsets per group, and does not present the operator as grouped convolution in the usual implementation sense [2603.21829]. Nor is it multi-scale at the operator level; its diversity is view-directional, not scale-wise.

A common misconception is therefore to treat any parallel deformable operator as grouped multi-scale deformable convolution. The surveyed evidence does not support that equivalence. Grouped multi-scale deformable convolution, in the strict sense, would require explicit channel-group decomposition or analogous grouped parameterization together with scale-differentiated deformable sampling. Among the works considered here, ASPDC comes closest in functional spirit but not in literal grouped design [2106.14336]. MDSConv is best read as multi-branch, multi-view, directional deformable convolution [2603.21829], and Deformable PV-RCNN as deformable keypoint refinement [2008.08766].

The resulting picture is that grouped multi-scale deformable convolution is less a single settled operator than a design space. One axis concerns whether specialization is by scale, direction, or instance geometry; another concerns whether branches are true groups of channels or full parallel experts; a third concerns whether fusion is fixed, gated, or attention-weighted. The strongest evidence in the present corpus supports multi-branch specialization with adaptive fusion, but not the claim that grouping in the canonical convolutional sense is necessary for effective deformable multi-scale behavior.

Source: https://www.emergentmind.com/topics/grouped-multi-scale-deformable-convolution