Dynamic Short Convolutions Improve Transformers

Abstract: Transformers have become the dominant architecture for LLMs, largely due to the scalability and flexibility of attention, feed-forward layers, residual connections, and normalization. This paper introduces dynamic short convolutions as an additional neural network primitive for improving Transformers. Unlike static short convolutions, dynamic convolutions use input-dependent filters, which preserves the locality bias of convolution while increasing expressivity. Motivating experiments show that applying dynamic short convolutions to key, query, and value representations improves performance on challenging associative recall tasks compared with static convolutional variants. Across language-modeling experiments ranging from 150M to 2B parameters, dynamic convolutions consistently outperform standard Transformers and Transformers augmented with static short convolutions. Fitting scaling laws indicates a 1.33$\times$ compute advantage over compute-matched Transformers when dynamic convolutions are applied to the key, query, and value vectors, and a 1.60$\times$ advantage when adding dynamic convolutions after every linear layer. Dynamic convolutions also offer improvements on linear RNNs (Mamba-2/Gated DeltaNet) and mixture-of-experts architectures. We make these gains practical with custom Triton kernels that enable efficient training with a manageable end-to-end slowdown. These results suggest that dynamic short convolutions are a scalable, hardware-efficient, and expressive primitive for advancing Transformer-based LLMs.

• The paper introduces dynamic short convolutions as an input-conditioned primitive that enhances local computation and expressivity in Transformer models.
• It details custom Triton GPU kernels and low-rank parameterizations that yield up to 1.60× compute advantages over traditional static convolutions.
• Empirical evaluations show superior accuracy on synthetic tasks and language modeling benchmarks, demonstrating significant improvements across varied model scales.

Dynamic Short Convolutions as a Transformer Primitive

Introduction

The study proposes dynamic short convolutions as a new architectural primitive for Transformer-based LLMs, aiming to enhance locality bias and expressivity. Unlike static short convolutions, dynamic variants generate input-dependent filters at each position, bridging expressivity typical of attention with the efficiency and inductive bias toward local computation provided by convolutions. The research presents both theoretical motivation and empirical evidence across synthetic and real-world language-modelling benchmarks, coupled with high-throughput custom kernel implementations for scalable training.

Parameterization and Implementation

Dynamic short convolution layers are formalized by generalizing static (depthwise separable) convolutions to be input-conditional. For input sequence $x_t \in \mathbb{R}^D$ and kernel width $W$, the filter $w^{(t)}_k$ at position $t$ is dynamically generated via a learned transformation, typically as a low-rank factorized projection or a head-wise projection with parameter sharing across contiguous channel groups. The result is:

$y_t = \sum_{k=0}^{W-1} w^{(t)}_k \odot x_{t-k}$

where the dynamic weights $w^{(t)}$ can be implemented efficiently through either low-rank projections or head-wise tying.

To overcome bandwidth bottlenecks due to the large size of dynamic kernel tensors in naïve implementations, the authors developed custom Triton GPU kernels. These kernels perform on-chip computation of convolutions and dynamically materialized weights, minimizing HBM traffic. Kernel benchmarks demonstrate that these implementations achieve comparable or better latency than CUDA-optimized static convolution baselines for sufficiently large head sizes.

Figure 1: Latency scaling of dynamic short-convolution kernels and baselines on an H100 GPU, illustrating competitive throughput and favorable scaling with larger head sizes.

Empirical Evaluation: Synthetic and Language Modeling Tasks

Synthetic Benchmarks

Dynamic short convolutions are validated on synthetic tasks requiring variable-length local context composition (e.g., variable-key multi-query associative recall). These tasks are constructed such that static convolutions (or fixed local weighting) cannot solve them optimally. The dynamic variant achieves substantially higher accuracy compared to both vanilla Transformers and static convolution-augmented Transformers. Additional experiments on the MAD diagnostic benchmark confirm these gains, especially in tasks requiring context-sensitive recall mechanisms.

Figure 2: Left: Dynamic short convolutions outperform static convolutions and vanilla Transformers in variable-key MQAR. Right: Significant gains on the MAD benchmark for dynamic short convolutions.

Language Modeling Experiments

A series of dense and Mixture-of-Experts Transformer models, ranging from 150M to 2B parameters (plus a 7B param MoE), were trained on the Nemotron-CC corpus using standard optimization settings. Dynamic convolutions were applied to Q, K, V projections (and in a variant, to all linear layers). Consistently, dynamic convolution-augmented models outperform both vanilla and static-convolution baselines on perplexity across held-out datasets (Nemotron-CC, WikiText-103, LAMBADA), and also on zero-shot reasoning tasks. The differences are pronounced even under compute/parameter-matched comparisons.

Scaling law fits indicate a 1.33× compute advantage for QKV-only placement and a 1.60× advantage for all-linear-layer placement, relative to compute-matched Transformer baselines. Training throughput with custom kernels introduces a manageable 8–22% slowdown versus vanilla models, but the scaling law improvements more than offset this overhead in practical runtime.

Impact on RNN/Linear Attention Architectures

Contemporary linear RNNs (e.g., Mamba-2, Gated DeltaNet) already utilize static short convolutions in their mixers. Replacing these with dynamic short convolutions provides further perplexity reductions and improved downstream performance, evidencing that input-conditioned local mixing enhances expressivity and sequence modeling capacity beyond what static local mixing affords.

Ablations and Architectural Study

A thorough series of ablations were performed:

• Kernel width: Optimal performance occurs for $W \in \{3, 4\}$, echoing prior results for static convolutions.
• Parameter sharing and expressivity: Low-rank parameterization delivers the best compute–performance tradeoff relative to head-wise or high-expressivity variants.
• Layer placement: Placing dynamic convolutions on Q, K, and V yields the largest empirical gain.
• QK-norm compatibility: Dynamic convolutions remain additive even when QK-norm is used, unlike static convolutions whose benefit is largely neutralized under normalization.

Practical and Theoretical Implications

The findings establish dynamic short convolutions as a scalable, hardware-efficient primitive that integrates the advantages of both local bias and position-dependent expressivity. The practical Triton kernels enable their deployment in high-throughput training. Theoretically, dynamic depthwise convolution provides a natural interpolation between static locality and content-based attention, and future work might extend its scope to hybridizing with other state-mixing mechanisms, joint kernel-parameter learning, or more expressive dynamic-filter parameterizations.

Implications for AI research involve reconsidering the range of primitives in LLM design: while attention mechanisms dominate, augmenting or mixing with dynamic short convolutions can accelerate scaling law progress, enhance context modeling, and potentially improve wall-clock and energy efficiency for LLM training. Future avenues include applying dynamic convolutions at larger model/data scales, integrating with long-context and retrieval-augmented architectures, and optimizing inference kernels for deployment.

Conclusion

Dynamic short convolutions introduce an input-conditioned, locality-biased layer that consistently improves performance over both vanilla and static-convolution-augmented Transformers. Empirically, these layers provide measurable compute advantages in scaling law fits, realize hardware efficiency via custom kernels, yield gains over modern linear RNNs, and remain robust under normalization and ablation regimes. The evidence positions dynamic short convolutions as a potent, practical addition to the modern LLM architect’s toolkit.