Global Depthwise Convolution (GDConv)

• Global Depthwise Convolution (GDConv) is defined as a depthwise operator concept that ideally uses full spatial support across the feature map.
• It contrasts global kernels with localized depthwise filters that approximate self-attention in Vision Transformers and standard convolutions in CNNs.
• Empirical studies highlight trade-offs in latency and accuracy when replacing attention heads with static, local depthwise kernels in pretrained models.

Global Depthwise Convolution (GDConv) commonly denotes a depthwise operator with global spatial support, such as an $m \times m$ depthwise kernel spanning the full feature map. In the two arXiv works most directly relevant here, however, that operator is distinguished rather than introduced. "Accelerating Vision Foundation Models with Drop-in Depthwise Convolution" studies a drop-in depthwise convolution replacement for selected self-attention heads in pretrained Vision Transformers, but explicitly uses a finite-kernel, local, static, content-independent depthwise convolution rather than a true global one (Scribano et al., 21 May 2026). "Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks" introduces Generalized Depthwise-Separable (GDWS) convolution, where "G" stands for generalized, not global; the method remains a factorization of standard local $K \times K$ convolution rather than a full-spatial depthwise operator (Dbouk et al., 2021). The resulting literature position is terminologically delicate: GDConv, in the strict global-spatial sense, is not the object implemented in either paper, but both works clarify neighboring regimes in which depthwise operators approximate either self-attention or standard convolution.

1. Terminological scope and formal distinction

The central distinction is between global spatial support and local spatial support. A true GDConv would typically use a kernel spanning the full feature map, such as $m \times m$ depthwise kernels, so that spatial aggregation is globally supported but remains channelwise. By contrast, the Vision Transformer acceleration work replaces certain attention heads with a local operator defined on a neighborhood $\Delta_k \subset \Delta_m$, with experiments fixing $k=3$. The replacement is therefore neither global by kernel size nor global in the sense of a fully connected token mixer (Scribano et al., 21 May 2026).

A second distinction concerns the meaning of the initialism itself. In the CNN approximation work, the term is GDWS, not GDConv, and its expansion is Generalized Depthwise-Separable. The operator allows each input channel to have multiple depthwise filters before a pointwise mixing stage, but the spatial kernels remain ordinary local $K \times K$ kernels. There is no notion of full-image support or global filtering in that formulation (Dbouk et al., 2021).

This terminological separation matters because superficially similar names can conceal materially different operators. In one case, depthwise convolution is used as a local surrogate for selected self-attention heads; in the other, it is used as a generalized factorization of pretrained convolutional weights. Neither mechanism, as defined, is equivalent to GDConv in the full-spatial sense.

2. Attention as a precursor to convolutional structure

The transformer-based analysis begins from standard self-attention on a ViT token matrix $X \in \mathbb{R}^{n \times d}$, with $n=m^2$ patch tokens arranged on an $m \times m$ grid. For head $h$,

$K \times K$0

The attention output is

$K \times K$1

with attention matrix

$K \times K$2

and full multi-head output

$K \times K$3

After reshaping tokens to spatial form, the head output at location $K \times K$4 is written as

$K \times K$5

This formulation exposes the bridge to convolution: attention is a weighted spatial aggregation over values. The crucial difference is that the weights depend on the input and on the query position. The proposed approximation replaces these input-dependent weights with a learned local kernel,

$K \times K$6

where $K \times K$7 and $K \times K$8. This establishes a convolutional relation, but only over a truncated local support and with static rather than input-generated coefficients (Scribano et al., 21 May 2026).

The paper formalizes when an attention head becomes convolution-like through three properties of the attention matrix $K \times K$9: locality, translation invariance, and input invariance. For a receptive field $m \times m$0,

$m \times m$1

$m \times m$2

$m \times m$3

Under these conditions, the head becomes equivalent to a fixed local operator on values. These conditions imply convolutional structure, but not global convolutional structure.

3. Local depthwise replacement inside pretrained Vision Transformers

The ViT acceleration method derives two convolutional realizations. A full convolution form folds the local kernel into the value projection: $m \times m$4 with

$m \times m$5

where $m \times m$6. This corresponds to a standard spatial convolution after merging value projection and local mixing.

The implemented replacement is the depthwise decomposition

$m \times m$7

with

$m \times m$8

and

$m \times m$9

Each output channel within head $\Delta_k \subset \Delta_m$0 therefore receives its own $\Delta_k \subset \Delta_m$1 spatial filter. In the reported experiments, $\Delta_k \subset \Delta_m$2, so the operational module is a $\Delta_k \subset \Delta_m$3 depthwise convolution on the value features of selected heads (Scribano et al., 21 May 2026).

This operator is static and content-independent: the kernel is a learned parameter rather than a function of the input. It is also explicitly local, since $\Delta_k \subset \Delta_m$4 is a subset of the full attention support $\Delta_k \subset \Delta_m$5. Any apparent relation to global mixing comes only from the fact that the module replaces a mechanism—self-attention—that originally had full token-to-token connectivity. The replacement itself remains a finite-kernel local DWConv.

Architecturally, the substitution occurs inside a standard MHSA block at the head level. The method does not remove the entire MHSA mechanism uniformly across the network. Instead, it replaces either selected heads across the model or all heads in selected blocks. The unensembled formulation preserves concatenation and output projection $\Delta_k \subset \Delta_m$6, while an ensembled variant merges multiple heads into one effective depthwise-convolution branch.

4. Selection criteria, head ensembling, and approximation regime

The viability of the replacement depends on identifying heads whose attention is already close to a static convolution. The selection criterion is based on the pointwise standard deviation of the attention matrix $\Delta_k \subset \Delta_m$7 across data. For each head $\Delta_k \subset \Delta_m$8, an online estimate of the standard deviation tensor $\Delta_k \subset \Delta_m$9 is computed across $k=3$0 samples using Welford’s algorithm, and summarized by

$k=3$1

For blockwise replacement,

$k=3$2

Heads or blocks with the smallest scores are selected. Small $k=3$3 indicates that the attention pattern changes little across inputs and is interpreted as evidence for approximate input invariance (Scribano et al., 21 May 2026).

This criterion does not test for global support. It tests whether a head is static across inputs. The conceptual target is therefore the opposite of a richly content-dependent global mixer: the replaceable heads are those that behave in a localized, nearly translation-like, and largely positional fashion. The paper explicitly motivates the approximation in terms of heads learning highly localized, static patterns.

The work also introduces a head-ensembling mechanism with learnable head weights $k=3$4,

$k=3$5

and defines

$k=3$6

The paper argues that SPViT corresponds to a special case under strong assumptions, but the resulting branch remains a local-kernel depthwise module rather than a GDConv. A stochastic-gating selection approach adapted from Differentiable Subset Pruning is also discussed, yet the simpler $k=3$7 criterion is reported to work better empirically.

5. Generalized depthwise-separable convolution and its non-global character

The second relevant operator is the $k=3$8 Generalized Depthwise-Separable (GDWS) convolution. For input $k=3$9, output channels $K \times K$0, and kernel size $K \times K$1, GDWS introduces a channel distribution vector

$K \times K$2

and consists of two stages: a generalized depthwise convolution, where input channel $K \times K$3 is convolved with $K \times K$4 depthwise filters, followed by a $K \times K$5 pointwise convolution mapping the resulting

$K \times K$6

intermediate channels to the final output (Dbouk et al., 2021).

The method is most naturally understood through its matrix factorization. A standard convolution with vectorized filters $K \times K$7 is written as

$K \times K$8

with MAC complexity

$K \times K$9

GDWS factorizes the weight matrix as

$X \in \mathbb{R}^{n \times d}$0

where $X \in \mathbb{R}^{n \times d}$1 and $X \in \mathbb{R}^{n \times d}$2. Channelwise partitioning yields blocks $X \in \mathbb{R}^{n \times d}$3, each satisfying

$X \in \mathbb{R}^{n \times d}$4

This means GDWS approximates a standard convolution by independently low-rank approximating each channel block.

Several special cases are explicit. Setting

$X \in \mathbb{R}^{n \times d}$5

reduces GDWS to standard MobileNet-style depthwise-separable convolution. Setting

$X \in \mathbb{R}^{n \times d}$6

permits exact representation of any standard convolution. The construction remains local throughout: the novelty lies in generalized factorization and per-channel-block rank allocation, not in any global receptive field.

Approximation is optimized by truncated SVD. For fixed $X \in \mathbb{R}^{n \times d}$7,

$X \in \mathbb{R}^{n \times d}$8

with error

$X \in \mathbb{R}^{n \times d}$9

The weighted objective is

$n=m^2$0

The algorithms MEGO and LEGO solve, respectively, minimum-error approximation under a complexity budget and minimum-complexity approximation under an error budget. These results make GDWS a rigorously characterized post-training approximation of standard 2D convolution, but not a GDConv in the global-spatial sense.

6. Efficiency regimes, empirical behavior, and implications for GDConv research

The local DWConv replacement in pretrained ViTs is motivated partly by complexity. The full-convolution form has complexity $n=m^2$1, whereas the depthwise decomposition reduces this to

$n=m^2$2

For a single MHSA block with $n=m^2$3 and $n=m^2$4 tokens, the reported benchmarks are: standard MHSA at $n=m^2$5 GFLOPs, $n=m^2$6 ms, and $n=m^2$7 MB; full convolution replacement at $n=m^2$8 GFLOPs, $n=m^2$9 ms, and $m \times m$0 MB; depthwise replacement at $m \times m$1 GFLOPs, $m \times m$2 ms, and $m \times m$3 MB; SPViT-style at $m \times m$4 GFLOPs and $m \times m$5 ms; and Ens+DW at $m \times m$6 GFLOPs and $m \times m$7 ms (Scribano et al., 21 May 2026).

At model scale, replacing 12 of 24 blocks in DINOv2-ViT-L reduces latency from $m \times m$8 ms to $m \times m$9 ms, a $h$0 speedup, with $h$1 mIoU on COCO segmentation; replacing 16 of 24 blocks gives $h$2 ms, a $h$3 speedup, with $h$4 mIoU. The reported method targets pretrained foundation models through a two-stage procedure: first fine-tune the original backbone on the target task with normal MHSA, then replace selected heads or blocks with convolutional layers, then fine-tune again for half the original number of epochs. The learned convolution kernels are introduced in the second phase, while $h$5 and $h$6 retain pretrained information.

The GDWS work presents a different efficiency profile. Its MAC count is

$h$7

compared with

$h$8

for standard convolution. A central claim is improved actual FPS on an NVIDIA Jetson Xavier NX. Reported examples include ResNet-18 on CIFAR-10 from 28 FPS to 104 FPS with GDWS ($h$9); VGG-16 on CIFAR-10 from 36 FPS to 129 FPS with GDWS ($K \times K$00); WRN-28-4 on CIFAR-10 from 17 FPS to 65 FPS; and ResNet-50 on ImageNet from 15 FPS to 19 FPS (Dbouk et al., 2021). The same paper emphasizes that GDWS is a post-training approximation that does not require any additional training for its main results, and that adversarial robustness is usually preserved within about $K \times K$01.

For GDConv research, the combined implication is narrow but useful. The transformer work provides evidence that many pretrained ViT heads can be replaced by static depthwise spatial filtering, yet the successful regime is explicitly local rather than global. The CNN work shows that depthwise factorization can be generalized and made provably optimal as a post-training approximation of local convolution, but again without any global-spatial operator. A plausible implication is that future work on strict GDConv would need to establish not only input invariance and translation-like behavior, but also sufficiently broad spatial support to justify full-map depthwise kernels. The evidence here addresses the easier and more hardware-friendly case: local depthwise operators that substitute for either selected attention heads or standard convolutions.