Dynamic Short Convolutions

• Dynamic short convolutions are input-conditioned operators that adapt local filter weights based on the current token, preserving locality with enhanced expressivity.
• They employ diverse strategies such as per-token kernel synthesis, kernel aggregation with attention, and head-wise low-rank factorization to balance accuracy and efficiency.
• Applications in sequence modeling, computer vision, and segmentation demonstrate improved performance metrics while reducing computational overhead.

Searching arXiv for recent and foundational papers on dynamic short convolutions and closely related dynamic convolution methods. arXiv search query: "dynamic short convolutions transformers (Sieberling et al., 2 Jun 2026)" Dynamic short convolutions are input-conditioned local convolutional operators in which a short receptive field is retained but the filter is no longer fixed across positions or inputs. In the sequence setting, a static short convolution of width $W$ applies a shared filter $w$ as $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$, whereas the dynamic variant replaces $w$ with a time-varying filter $w^{(t)}$ predicted from the current hidden state, $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$ (Sieberling et al., 2 Jun 2026). Closely related formulations appear in earlier LLMs, where kernels are predicted from the current token representation only (Wu et al., 2019), and in compact CNNs, where a layer aggregates multiple small kernels using input-dependent attention, $$\tilde W(\mathbf{x})=\sum_{k=1}^K \pi_k(\mathbf{x})\tilde W_k$$ (Chen et al., 2019). Across these settings, the unifying objective is to preserve the locality bias and low computational footprint of short convolutions while increasing expressivity through input dependence.

1. Formal definition and canonical formulations

The defining property of a dynamic short convolution is that the local mixing rule is conditioned on the input rather than fixed a priori. In the formulation introduced for LLMs, the filter generator is an affine transformation of the current hidden state, so each token position emits its own local convolution kernel. The operation is local, causal in the autoregressive setting, and structurally distinct from attention because it does not compute content-based weights by query-key similarity; instead, it directly predicts local mixing weights from the current position (Sieberling et al., 2 Jun 2026).

An earlier sequence-model formulation expresses the same principle through dynamic lightweight convolutions. There, the output at position $i$ and channel $c$ is

$$\mathrm{DynamicConv}(X,i,c)=\mathrm{LightConv}(X,f(X_i)_{h,:},i,c),$$

where $w$0 is implemented as a linear projection from the current token representation $w$1. The kernel therefore depends on the current time step only, not on pairwise interactions with the full sequence (Wu et al., 2019).

In CNNs, dynamic short convolution is frequently instantiated as kernel aggregation rather than direct per-position kernel synthesis. A layer learns $w$2 parallel kernels $w$3 of identical shape and computes an input-specific convex combination using attention weights $w$4, subject to $w$5 and $w$6. The dynamic layer output is then

$w$7

with both kernel and bias conditioned on the input (Chen et al., 2019). This formulation makes the convolution nonlinear not only through the feature map but through the operator itself.

These formulations share a common abstraction: a short local operator plus learned routing over a small family of local transformations, whether that routing is implemented by kernel prediction, mixture over kernels, or a controller that instantiates a small per-example mask head. This suggests that “dynamic short convolution” is best understood as a design pattern rather than a single architecture.

2. Parameterization strategies

Several parameterization schemes have been proposed to make dynamic short convolutions practical.

In compact CNNs, the attention-over-kernels approach uses a squeeze-and-excitation style branch that globally average-pools the input feature map, applies two fully connected layers with a ReLU in between, and then applies a softmax over $w$8 kernels. The first FC layer uses a reduction ratio of 4. If the input has shape $w$9, the attention overhead is approximately

$y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$0

while the convolution cost is

$y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$1

Kernel aggregation adds

$y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$2

Multi-Adds, which is negligible when $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$3 (Chen et al., 2019).

In LLMs, dynamic lightweight convolutions reduce parameter count by sharing kernels across channel groups. LightConv uses $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$4 kernels shared across $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$5 channels, with softmax normalization over the kernel dimension. DynamicConv preserves this grouped structure but replaces the fixed kernel with a kernel predicted from the current token representation. The paper gives a concrete example with $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$6 and $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$7: regular convolution uses 7,340,032 weights, depthwise separable convolution uses 7,168 weights, and shared-kernel LightConv with $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$8 uses 112 weights (Wu et al., 2019).

For Transformer augmentation at scale, two parameterizations are emphasized: a low-rank factorization of the filter generator, and a head-wise parameterization in which channels are grouped and the same kernel is broadcast across a head of size $y_t := \sum_{k=0}^{W-1} w_k \odot x_{t-k}$9. The low-rank form is usually better in quality, while the head-wise form is especially useful for efficient GPU kernels. A bias term is also included, so the dynamic convolution is effectively affine in the input (Sieberling et al., 2 Jun 2026).

A more radical parameterization appears in instance segmentation. CondInst uses a controller branch that outputs a parameter vector $w$0 for each instance location, and mask prediction is written as

$w$1

where $w$2 encodes relative coordinates. The dynamically generated mask head is deliberately tiny: three $w$3 convolution layers, each with 8 channels, followed by a final 1-channel output layer, for a total of 169 parameters (Tian et al., 2021). In this sense, the head is a dynamic short FCN instantiated per instance.

3. Relation to self-attention and relative position mechanisms

A central theoretical development is the reinterpretation of relative position mechanisms in Transformers as convolutional operators. In pre-trained LLMs, fixed relative position terms correspond to lightweight convolutions, and Shaw-style relative position embeddings are essentially identical to dynamic lightweight convolutions. Under this view, the relative embedding $w$4 is a learned convolution kernel over relative positions, and self-attention itself can be viewed as a dynamic convolution over token neighborhoods, where the dynamic kernel is determined by the query (Chang et al., 2021).

This equivalence motivates composite attention, which combines standard self-attention, dynamic lightweight convolution, and fixed lightweight convolution in a single attention score. The reported interpretation is that dynamic convolution provides token-specific relative-position flexibility, fixed convolution provides a stable relative bias, and together they form a unified framework for relative position handling (Chang et al., 2021).

The relationship to earlier dynamic-convolution LLMs is direct but not identical. The 2019 sequence-model formulation predicts kernels from the current token only and thereby reduces complexity from quadratic in sequence length to linear for fixed kernel size $w$5. Self-attention is described there as computing weights by comparing the current time step to all context elements, whereas DynamicConv computes one local kernel per position and applies it over a fixed window (Wu et al., 2019). The later Transformer work sharpens the distinction: dynamic short convolutions are analogous in spirit to attention, but they preserve a stronger relative-position locality bias and do not require query-key similarity (Sieberling et al., 2 Jun 2026).

A common misconception is that dynamic short convolutions and attention are competing paradigms. The literature instead presents multiple hybridizations: convolutions can replace absolute position embeddings, serve as relative-position mechanisms inside attention, or augment query, key, and value projections without displacing attention entirely (Chang et al., 2021).

4. Computer vision: compact CNNs, segmentation, and sparse execution

In lightweight CNNs, dynamic convolution was introduced to increase model complexity without increasing network depth or width. Replacing static convolutions with dynamic convolutions in MobileNetV2 improves top-1 accuracy by 3.2 to 5.6 points depending on width multiplier, with only a small increase in Multi-Adds; MobileNetV2 $w$6 rises from 65.4% to 69.9% top-1 while MAdds increase from 97.0M to 101.4M. On MobileNetV3-Small, top-1 accuracy rises from 67.4% to 70.3%, a gain of 2.9 points, with only about 4% extra FLOPs, and on COCO keypoint detection the same model gains 2.9 AP in the abstracted headline result (Chen et al., 2019).

Ablation results in the same work indicate that increasing the number of kernels helps up to around $w$7, that placing dynamic convolution in the last $w$8 projection layer of MobileNetV2 bottlenecks gives the largest single-layer improvement, and that later stages benefit more because attention becomes more sparse in deeper layers. Dynamic convolution is also reported to be compatible with $w$9, $w^{(t)}$0, depthwise, and group convolutions, as well as with SE and h-swish (Chen et al., 2019).

For instance and panoptic segmentation, CondInst replaces ROI-based mask heads with instance-aware conditional convolutions. The framework remains fully convolutional, eliminates ROI cropping and alignment, and predicts masks at high spatial resolution directly from shared feature maps. On COCO test-dev, CondInst with ResNet-50-FPN reaches 35.3 AP, compared with 34.6 AP for the original Mask R-CNN, while running at 49 ms versus 65 ms per image; with ResNet-101-FPN it reaches 39.1 AP, and for panoptic segmentation it achieves 46.1 PQ on COCO test-dev (Tian et al., 2021). The method’s dynamic head is extremely compact, which is why the work explicitly describes it as a short convolutional head.

A distinct but related line of work uses dynamic convolutions to exploit spatial sparsity rather than to alter filter coefficients. DynConv inserts a gating branch into each residual block, predicts a hard execution mask with the Gumbel-Softmax trick, and computes the residual function only at active spatial positions. On human pose estimation, the paper reports a 60% speedup with no loss in accuracy, and a 125% speedup with only 0.6% accuracy loss (Verelst et al., 2019). This is conceptually adjacent to dynamic short convolutions but mechanistically different: the operator is dynamic because execution is spatially conditional, not because the short filter itself is reparameterized.

5. Sequence modeling and large-scale Transformer augmentation

Dynamic short convolutions first established strong empirical performance in sequence transduction as an alternative to self-attention. In machine translation, DynamicConv achieves 29.7 BLEU on WMT’14 English-German, compared with 29.3 for Transformer Big; on IWSLT’14 German-English it improves a self-attention baseline from 34.4 BLEU to 35.2 BLEU; on WMT Zh-En it improves 23.8 to 24.4 BLEU; on Billion Word it yields 26.60 valid and 26.67 test perplexity, slightly better than the self-attention baseline; and on CNN/DailyMail it improves ROUGE-1/2/L from 39.26/15.98/36.35 to 39.84/16.25/36.73. The paper also reports about 20% faster inference than a highly optimized self-attention baseline (Wu et al., 2019).

The 2026 Transformer study extends the idea from replacing attention modules to augmenting standard Llama-style blocks. Dynamic short convolutions are applied to queries, keys, and values before RoPE in residual form, with width $w^{(t)}$1, and the projection that generates the dynamic convolution weights is taken from the post-attention-norm activations. A broader variant inserts dynamic convolutions after every linear layer, not only on QKV (Sieberling et al., 2 Jun 2026).

The motivating synthetic benchmark is modified multi-query associative recall, where keys can have variable lengths $w^{(t)}$2 and the model must retrieve the matching value when the pair reappears. With kernel width $w^{(t)}$3, dynamic convolutions outperform both the plain Transformer and a Transformer with static convolutions, with especially large gains on the harder fuzzy recall setting. The same pattern is reported on MAD, particularly on tasks requiring variable-length local aggregation (Sieberling et al., 2 Jun 2026).

At language-model scale, dense Transformer models from 150M to 2B parameters trained on Nemotron-CC consistently outperform both plain Transformers and Transformers augmented with static short convolutions. Fitting scaling laws yields an approximate $w^{(t)}$4 compute advantage for the QKV-only dynamic-convolution variant and about $w^{(t)}$5 when dynamic convolutions are inserted after all linear layers. The same study reports improvements on held-out Nemotron-CC, Wikitext-103, LAMBADA, zero-shot reasoning benchmarks from lm-eval-harness, RULER, a 7B-parameter MoE model with 1B active parameters, and linear-RNN-style architectures including Mamba-2 and Gated DeltaNet (Sieberling et al., 2 Jun 2026).

These results position dynamic short convolutions not merely as substitutes for attention, but as local composition modules that can improve Transformers, MoE models, and linear RNNs under a common architectural principle.

6. Optimization, efficiency, and implementation

Training stability has been a recurring issue. In kernel-mixture CNNs, two design choices are identified as especially important: enforcing $w^{(t)}$6 by softmax so that the aggregated kernel remains inside the convex hull of the learned kernels, and using a large softmax temperature early in training to keep attentions near-uniform. The attention is computed as

$w^{(t)}$7

with $w^{(t)}$8 reported to work well, and annealing $w^{(t)}$9 from 30 to 1 over the first 10 epochs improving results further (Chen et al., 2019).

In sequence models, softmax normalization is likewise empirically important. LightConv normalizes kernels with softmax over the kernel dimension, and the ablation study reports that removing softmax normalization from DynamicConv causes training to diverge. DropConnect on the normalized weights yields a small additional improvement (Wu et al., 2019).

The efficiency story depends strongly on implementation. In MobileNetV2 with $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$0, dynamic convolution adds about 4% MAdds, while CPU runtime rises by about 10% in a PyTorch implementation, partly because pooling and small FC operations are relatively efficient on CPU (Chen et al., 2019). In LLMs, dynamic convolutions are described as memory-bound, and naive PyTorch implementations are slow because they repeatedly materialize intermediate tensors in HBM. To address this, custom Triton kernels perform the convolution on-chip and write back only the final output. On an H100 SXM5 80GB GPU with $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$1, $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$2, $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$3, $y_t := \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$4, and BF16, the custom kernels are reported to be 1.8–3.9× faster than the best torch.compile baseline on combined forward and backward pass, with head-wise kernels sustaining roughly 2.6–3.0 TB/s of HBM bandwidth and the QKV dynamic-convolution model incurring only about an 8% end-to-end throughput slowdown at 2B scale; the all-linear variant incurs roughly a 22–25% throughput reduction (Sieberling et al., 2 Jun 2026).

Sparse-execution methods arrive at efficiency through a different route. DynConv uses a gather-scatter CUDA strategy that gathers active positions into a compact dense tensor, applies pointwise operations efficiently there, and then scatters results back. The paper argues explicitly that FLOPs alone are not a good predictor of runtime and targets wall-clock speed instead (Verelst et al., 2019).

7. Scope, distinctions, and open directions

The literature uses closely related terminology for several distinct mechanisms. One family predicts a position-specific short kernel from the current hidden state (Wu et al., 2019, Sieberling et al., 2 Jun 2026). Another forms an input-dependent convex combination of a small bank of kernels (Chen et al., 2019). Another generates a tiny per-instance FCN for segmentation (Tian et al., 2021). A neighboring but different family uses spatial gating to skip computation rather than to synthesize filters (Verelst et al., 2019). Treating these as identical obscures meaningful architectural differences.

Another recurrent distinction concerns locality versus long-range modeling. Dynamic short convolutions strengthen local composition, but they do not by themselves imply full-sequence memory. This is evident in fMRI-S4, where a 1D convolutional front end captures short-range temporal dependencies and denoises local structure, while S4 blocks model long-range temporal evolution; removing either the S4 layers or the convolutional front end reduces accuracy by about 2–4% (El-Gazzar et al., 2022). A plausible implication is that dynamic short convolutions are especially effective when paired with a complementary long-range mechanism rather than asked to subsume it.

For Transformer research, the main open question stated explicitly is scaling: the 2021 reinterpretation of relative position embeddings notes that convolutional positional information is especially helpful in smaller or more data-limited models and argues that future work should study how to make these gains scale more robustly to larger models and broader pre-training settings (Chang et al., 2021). The 2026 results partially answer that question by showing consistent gains from 150M to 2B parameters, as well as benefits in MoE and linear-RNN settings, while also noting that more aggressive fusion, especially into matmul epilogues, could further reduce overhead (Sieberling et al., 2 Jun 2026).

Taken together, the field presents dynamic short convolutions as a locality-preserving, input-adaptive primitive whose exact realization varies by domain. In compact CNNs they increase representational capacity without increasing depth or width; in sequence models they offer a linear-complexity alternative or complement to self-attention; in segmentation they instantiate instance-aware short heads; and in modern Transformers they improve local, input-conditioned composition at scale.