Self-Adaptive Convolution Module

• Self-adaptive convolution module is a data-dependent operator that replaces static kernels with dynamic alternatives based on local signal content.
• Various mechanisms such as mask-gated, candidate-kernel assembly, and deformable sampling are used to adapt receptive fields to scene geometry and context.
• Empirical studies show that integrating these adaptive methods enhances performance in tasks like depth estimation, image restoration, and segmentation with modest cost increases.

Searching arXiv for the cited works to ground the article in the latest paper records. The term self-adaptive convolution module has been used for a family of operators that replace a fixed convolutional kernel, fixed sampling grid, or fixed receptive field with a data-dependent alternative. Across the literature, this dependence may be driven by local validity masks and depth discontinuities in RGB-D restoration, frame-wise attention weights in speech enhancement, cluster assignments in non-local pansharpening, deformable offsets in segmentation and deblurring, dataset fingerprints in medical segmentation, graph edge attributes in point-cloud analysis, or relative offsets that explicitly connect convolution and self-attention (Xian et al., 2020, Duan et al., 2024, Wang et al., 20 Feb 2025, Wang et al., 2019, Purohit et al., 2019, Deng et al., 1 Sep 2025, Fan et al., 11 Oct 2025). The common objective is to make the effective operator vary with signal content, scene geometry, or structural context, rather than applying a single static kernel uniformly.

1. Definition and taxonomy

In standard convolution, a learned kernel is shared across all inputs and spatial or temporal positions. Several papers in this area define self-adaptation precisely as a departure from that assumption. In speech enhancement, a static kernel $W$ is replaced by a small set of $M$ candidate kernels $\{W^1,\dots,W^M\}$, and at each frame $t$ a data-dependent kernel $K_t$ is assembled (Wang et al., 20 Feb 2025). In RGB-D completion and super-resolution, the learned kernel weights remain global, but each location is modulated by a run-time mask $m_{k,l}$, so the effective kernel becomes spatially varying (Xian et al., 2020). In deformable variants, adaptation is transferred from weights to sampling coordinates, so the receptive field shape changes with the input (Purohit et al., 2019, Wang et al., 2019).

A concise taxonomy of mechanisms used under this label is given below.

Mechanism	Representative rule	Representative papers
Mask-gated convolution	$$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$	(Xian et al., 2020)
Candidate-kernel assembly	$K_t=\sum_{m=1}^M a_t^m W^m$	(Wang et al., 20 Feb 2025)
Cluster-wise kernel generation	CANConv = SRP + PWAC	(Duan et al., 2024)
Branch-gated mixture	$Y(x)=\sum_{i=1}^3 \alpha_i F_i$	(Cui et al., 15 May 2025)
Deformable sampling	$Y(p_0)=\sum_n W_n\cdot X(p_0+p_n+\Delta p_n)+b$	(Purohit et al., 2019)
Scale-adaptive sampling grid	$M$0	(Gao et al., 9 Apr 2026)
Relative-offset adaptive encoding	$M$1 with $M$2	(Fan et al., 11 Oct 2025)

This diversity is important: self-adaptive convolution is not a single canonical layer. The term has covered dynamic kernels, dynamic supports, dynamic scales, dynamic offsets, and dynamic mixtures of experts.

2. Principal adaptation mechanisms

One major class keeps the underlying convolutional weights but gates their local support. Xian et al. introduce an adaptive convolution in which each spatial position has an extra “filter-mask” coefficient $M$3. Their region-adaptive mask marks valid versus invalid depth samples for hole filling, while the depth-adaptive mask suppresses neighbors whose completed depth differs by more than a small amount, thereby preserving sharp depth discontinuities (Xian et al., 2020). In HA-GCN, the same idea appears on graphs: a filter-generating network produces $M$4, and the base graph weight matrix is modulated as $M$5 before aggregation (Zhou et al., 2017).

A second class synthesizes kernels explicitly. In speech enhancement, adaptive convolution performs frame-wise causal dynamic convolution, generating time-varying kernels for each frame by assembling multiple parallel candidate kernels, with causal weights derived from current and historical information (Wang et al., 20 Feb 2025). In “Boosting Medical Image Segmentation Performance with Adaptive Convolution Layer,” the per-pixel kernel is a linear combination of pre-defined multi-scale Fourier–Bessel basis filters, and the coefficient vector is produced by a small Coefficient Generator Network (Modaresi et al., 2024). In CANConv, the kernel is not predicted for each pixel independently; instead, pixels are partitioned by Similarity Relationship Partition, and a single kernel $M$6 and bias $M$7 are generated per cluster and applied to every pixel in that cluster (Duan et al., 2024).

A third class adapts geometry rather than kernel coefficients. ACE uses three cascaded modulated-deformable convolution blocks so that the effective sampling grid morphs to align with object boundaries and automatically adjusts its spatial extent (Wang et al., 2019). The deformable residual module in motion deblurring predicts offsets $M$8 for each regular kernel point, allowing the network to steer sampling along local motion-blur direction (Purohit et al., 2019). SACNet’s Adaptive Receptive Field Module uses a DCNv3 core with grouped offsets and modulation weights, while SAFDConvolution learns a full-image, per-pixel, two-dimensional displacement field $M$9 via multi-head attention plus a feed-forward network, warps the feature map, and then applies a standard convolution on the warped map (Zhang et al., 2024, Zhu et al., 24 Jul 2025).

A fourth class adapts scale or expert selection. DcSConv converts a depth estimate into a kernel scale through the prior relation $\{W^1,\dots,W^M\}$0, explicitly tying receptive-field size to scene depth in monocular depth estimation (Gao et al., 9 Apr 2026). MRFFIConv uses three parallel experts—MSDC, DCN, and MDDC—and fuses them with Softmax-normalized gating weights $\{W^1,\dots,W^M\}$1, so the operator emphasizes the branch mixture best suited to the current input (Cui et al., 15 May 2025). MSA$\{W^1,\dots,W^M\}$2-Net goes further toward dataset-level adaptation: it computes a dataset-dependent quartile shift vector, forms a candidate-kernel matrix $\{W^1,\dots,W^M\}$3, and uses a learnable selection-probability matrix $\{W^1,\dots,W^M\}$4 to choose kernel sizes automatically (Deng et al., 1 Sep 2025).

3. Representative mathematical formulations

Several formulations recur across this literature.

For mask-gated adaptive convolution, the operator can be written as

$\{W^1,\dots,W^M\}$5

with $\{W^1,\dots,W^M\}$6. Here the kernel weights are fixed after training, but the mask is computed at run time from validity or depth-difference rules (Xian et al., 2020).

For dynamic-kernel assembly, the canonical rule is

$\{W^1,\dots,W^M\}$7

where $\{W^1,\dots,W^M\}$8 is produced by a lightweight attention mechanism and normalized by softmax. In the speech setting, the pooled descriptor is

$\{W^1,\dots,W^M\}$9

and the adaptive kernel at frame $t$0 depends only on $t$1, so causality is preserved (Wang et al., 20 Feb 2025).

For cluster-wise adaptive convolution, CANConv first computes local descriptors

$t$2

runs K-means to obtain an index map $t$3, computes cluster centroids $t$4, and then generates one kernel per cluster. The output is

$t$5

To avoid an MLP with $t$6 parameters, the kernel generator attends to a single global parameter tensor $t$7 through three gating vectors $t$8, $t$9, and $K_t$0 (Duan et al., 2024).

For deformable sampling, the archetypal equation is

$K_t$1

with bilinear interpolation at fractional locations. This equation appears directly in motion deblurring and underlies later receptive-field adaptation modules built on DCNv2 or DCNv3 (Purohit et al., 2019, Wang et al., 2019, Zhang et al., 2024).

For explicit scale adaptation, DcSConv begins from the pinhole relation $K_t$2 and converts depth to kernel size via

$K_t$3

The convolution then samples over a fractional grid $K_t$4, with bilinear interpolation used when $K_t$5 is non-integer (Gao et al., 9 Apr 2026).

Graph and point-cloud variants show that self-adaptive convolution is not restricted to regular image lattices. MG-SAGC defines edge-dependent filters by Chebyshev expansions in distance $K_t$6 and angle $K_t$7,

$K_t$8

so the weighting of each neighbor adapts to local geometry (Wu et al., 2020).

4. Architectural integration patterns

A common integration strategy is wholesale replacement of standard convolutions inside an existing backbone. CANNet is U-Net style, and all standard convolutions in the backbone are replaced by CANConv grouped into CAN-ResBlocks (Duan et al., 2024). In CNN-based speech enhancement networks such as DPCRN, DCCRN, GTCRN, and LiSenNet, vanilla depthwise or pointwise convolutions in the encoder and decoder are replaced by adaptive convolution layers, while AdaptCRN uses a repeated sequence of LayerNorm, adaptive depthwise convolution, batch normalization, PReLU, and adaptive pointwise convolutions with joint attention (Wang et al., 20 Feb 2025). ARFC-WAHNet replaces every standard $K_t$9 convolution in the encoder-decoder with MRFFIConv (Cui et al., 15 May 2025).

Another pattern is plug-and-play augmentation. The Adaptive Convolution Layer in AdaptUCTransNet sits immediately in front of the backbone segmentation network and leaves all other parts of UCTransNet unchanged (Modaresi et al., 2024). DcSConv is described as a plug-and-play module that can be applied on top of existing CNN based methods to enhance the conventional convolution block (Gao et al., 9 Apr 2026). SAFDConvolution is presented as having an interface similar to conventional convolution and is inserted into encoder and decoder stages of GDCUnet in place of plain convolutions (Zhu et al., 24 Jul 2025).

Hybridization with attention is a third pattern. ConvAttn is introduced specifically to replace most self-attention layers in a super-resolution transformer with a light convolution-based block that still captures long-range spatial context via a shared large kernel and per-instance weighting via a small dynamic kernel (Lee et al., 9 Mar 2025). Translution makes that relation explicit by defining relative query, key, and value encodings $m_{k,l}$0, $m_{k,l}$1, and $m_{k,l}$2, thereby unifying the adaptive identification capability of self-attention and the relative encoding advantage of convolution (Fan et al., 11 Oct 2025).

Graph architectures use analogous integration principles. HA-GCN concatenates high-order adaptive graph convolutions across orders $m_{k,l}$3 (Zhou et al., 2017). MG-SAGC constructs multiscale graphs, applies the same SAGC operator at each scale, and fuses the resulting feature maps by element-wise max-pooling across scale (Wu et al., 2020).

5. Empirical behavior and ablation evidence

Ablation studies repeatedly show that the form of adaptation matters at least as much as the presence of adaptation. In CANConv, turning off SRP by setting $m_{k,l}$4 causes performance to drop significantly, while $m_{k,l}$5, corresponding to pixel-wise dynamic filters, is also worse; replacing the lightweight attention kernel generator by a fully connected MLP of the same input/output size increases parameters by $m_{k,l}$6 and the network fails to converge; and removing the small-cluster global centroid trick degrades full-res HQNR from $m_{k,l}$7 to $m_{k,l}$8 (Duan et al., 2024). These results separate useful non-local partitioning from both global static convolution and excessively fine-grained per-pixel kernel prediction.

In frame-wise speech enhancement, GRU-based temporal attention outperforms multi-frame Conv1D and single-frame squeeze-and-excitation, global utterance-level dynamic convolution underperforms frame-wise dynamic convolution, and increasing the number of candidate kernels from $m_{k,l}$9 steadily improves PESQ before returns diminish beyond $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$0 (Wang et al., 20 Feb 2025). The same study reports that parameter increases in adaptive layers are typically $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$1–$$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$2 per convolution layer, but MAC increases remain $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$3 overall; for DPCRN-light, convolutional parameters rise from $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$4 to $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$5 while MACs grow from $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$6 to $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$7 (Wang et al., 20 Feb 2025).

In super-resolution, ConvAttn ablations show that “Only Self-attn” and “Only ConvAttn” each reduce PSNR by $$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$8–$$x'_{i,j}= b + \frac{1}{M_{i,j}}\sum (m_{k,l}w_{k,l})x_{k,l}$$9 dB relative to the hybrid ESC design, $K_t=\sum_{m=1}^M a_t^m W^m$0 is superior to $K_t=\sum_{m=1}^M a_t^m W^m$1 and $K_t=\sum_{m=1}^M a_t^m W^m$2 for the shared large kernel, removing the dynamic kernel $K_t=\sum_{m=1}^M a_t^m W^m$3 loses about $K_t=\sum_{m=1}^M a_t^m W^m$4 dB, and removing sharing of the large kernel doubles parameters without improving PSNR (Lee et al., 9 Mar 2025). In monocular depth estimation, deformable convolution, which only learns local shape offsets but not scale, yields no improvement or slight degradation, while explicit depth-to-scale conversion reduces SqRel from $K_t=\sum_{m=1}^M a_t^m W^m$5 to $K_t=\sum_{m=1}^M a_t^m W^m$6, and the full DcS-F fusion yields a further reduction to $K_t=\sum_{m=1}^M a_t^m W^m$7, reported as $K_t=\sum_{m=1}^M a_t^m W^m$8 (Gao et al., 9 Apr 2026).

Task-specific evaluations show similar patterns. In MSA$K_t=\sum_{m=1}^M a_t^m W^m$9-Net on Synapse, the full model with self-adaptive convolution in both MSConvBridge and MSADecoder achieves Dice $Y(x)=\sum_{i=1}^3 \alpha_i F_i$0 and HD95 $Y(x)=\sum_{i=1}^3 \alpha_i F_i$1, while removing both gives Dice $Y(x)=\sum_{i=1}^3 \alpha_i F_i$2 and HD95 $Y(x)=\sum_{i=1}^3 \alpha_i F_i$3; however, on Kvasir-SEG the self-adaptive result $Y(x)=\sum_{i=1}^3 \alpha_i F_i$4 is competitive but not strictly best, and a static Q3 setting reaches $Y(x)=\sum_{i=1}^3 \alpha_i F_i$5 (Deng et al., 1 Sep 2025). In fundus vessel segmentation, SAFDConvolution with a $Y(x)=\sum_{i=1}^3 \alpha_i F_i$6 kernel yields IoU $Y(x)=\sum_{i=1}^3 \alpha_i F_i$7 and Dice $Y(x)=\sum_{i=1}^3 \alpha_i F_i$8, exceeding Deformable Conv V3 at IoU $Y(x)=\sum_{i=1}^3 \alpha_i F_i$9 and Dice $Y(p_0)=\sum_n W_n\cdot X(p_0+p_n+\Delta p_n)+b$0 under the same configuration (Zhu et al., 24 Jul 2025).

6. Conceptual boundaries, misconceptions, and directions

A recurrent misconception is that self-adaptive convolution is synonymous with deformable convolution. The literature does not support that equivalence. Some modules adapt by masking a fixed kernel (Xian et al., 2020), some by mixing candidate kernels causally over time (Wang et al., 20 Feb 2025), some by generating one kernel per similarity cluster (Duan et al., 2024), some by selecting among expert branches with Softmax gates (Cui et al., 15 May 2025), some by converting depth into receptive-field scale (Gao et al., 9 Apr 2026), and some by using relative-offset-specific projections that recover either convolution or self-attention as limiting cases (Fan et al., 11 Oct 2025). Deformable sampling is one important branch, but not the only one.

A second misconception is that adaptation is always per-pixel. The granularity of adaptation varies widely: per-pixel in RGB-D restoration and adaptive basis synthesis (Xian et al., 2020, Modaresi et al., 2024), per-frame in streaming speech enhancement (Wang et al., 20 Feb 2025), per-cluster in CANConv (Duan et al., 2024), per-dataset in MSA$Y(p_0)=\sum_n W_n\cdot X(p_0+p_n+\Delta p_n)+b$1-Net (Deng et al., 1 Sep 2025), per-edge in graph and point-cloud convolution (Zhou et al., 2017, Wu et al., 2020), and per-instance in ConvAttn through a global pooled descriptor (Lee et al., 9 Mar 2025). This suggests that “self-adaptive” refers more to input-conditioned operator selection than to any fixed spatial granularity.

The trajectory of recent work also indicates several open directions already stated in the papers themselves. MSA$Y(p_0)=\sum_n W_n\cdot X(p_0+p_n+\Delta p_n)+b$2-Net points to more fine-grained, possibly region-aware kernel adaptation and the incorporation of boundary or shape priors (Deng et al., 1 Sep 2025). The adaptive medical segmentation layer based on Fourier–Bessel bases proposes learnable base filters and extension beyond the very first layer (Modaresi et al., 2024). ACE explicitly notes possible extensions to object detection, instance segmentation, and video segmentation, including conditioning offsets on temporal features (Wang et al., 2019). SAFDConvolution is suggested for more machine vision tasks with complex global self-similar features (Zhu et al., 24 Jul 2025). Taken together, these directions indicate that the field is moving from fixed receptive-field design toward increasingly structured, task-specific, and often hybrid adaptive operators rather than toward a single universal module.