---
title: Large Kernel Attention (LKA)
url: https://www.emergentmind.com/topics/large-kernel-attention-lka
type: topic
---

# Large Kernel Attention (LKA)

Searching arXiv for core and recent LKA papers to ground the article with citations.
arXiv search query: Large Kernel Attention Visual Attention Network LSKA deformable LKA LKCA large kernel convolution attention MetaSSC PuYun LKA-ReID LKASeg
Large Kernel Attention (LKA) is a convolutional attention mechanism that builds an attention map from large-receptive-field convolutional operations and applies it multiplicatively to the input feature map. In its canonical form, introduced in the Visual Attention Network (VAN), LKA replaces quadratic-cost self-attention with a sequence of depthwise local convolution, depthwise dilated convolution, and pointwise channel mixing, thereby preserving 2D structure, maintaining linear complexity in spatial size for fixed kernels, and introducing channel adaptability through a $1\times1$ projection [2202.09741]. Subsequent work generalized this idea into separable, deformable, three-dimensional, and task-specific variants for image classification, detection, segmentation, super-resolution, depth estimation, semantic scene completion, audio event detection, vehicle re-identification, and medium-range weather forecasting [2309.01439, 2309.00121, 2207.11225, 2411.03672, 2409.02123].

## 1. Conceptual basis and historical emergence

LKA emerged from a specific critique of applying vanilla self-attention to vision. VAN identifies three issues: treating images as 1D token sequences neglects their 2D structure, the quadratic complexity is too expensive for high-resolution images, and standard self-attention captures spatial adaptability but ignores channel adaptability [2202.09741]. LKA was proposed as a linear attention mechanism that preserves convolutional inductive biases while extending the receptive field sufficiently to model long-range correlations.

The same motivation recurs across later domains. Remote-sensing segmentation emphasizes that high-resolution aerial imagery requires global semantic capture without quadratic costs, while still preserving 2D topology and local detail [2410.10433]. In 3D medical segmentation, large kernels are motivated by extensive anatomical variation, low contrast, and the need for long-range volumetric context [2207.11225]. In autonomous-driving semantic scene completion, the argument is formulated in voxel space: self-attention scales as $O((HWD)^2)$ after flattening, whereas LKA preserves 3D topology and offers linear-time spatial modeling on sparse voxel grids [2411.03672].

A common misconception is that LKA denotes a single immutable block. The literature instead treats it as a design family. The canonical VAN operator omits sigmoid or softmax normalization in the attention path [2202.09741], whereas some later implementations normalize the attention map explicitly with sigmoid, including 3D medical segmentation, self-supervised monocular depth estimation, and sound event detection [2207.11225, 2409.17895, 2306.06461]. What remains stable across these variants is the reliance on large-kernel convolutional filtering to generate an input-dependent gating map.

## 2. Canonical operator, decomposition, and complexity

In VAN, LKA takes an input feature map $X \in \mathbb{R}^{C\times H\times W}$ and computes
$$
U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad
V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad
M(X) = \operatorname{Conv}_{1\times 1}(V),
$$
followed by elementwise gating
$$
Y = X \odot A(X), \qquad A(X)=M(X).
$$
The large kernel is not implemented as a monolithic dense $K\times K$ operator. Instead, VAN decomposes a $K\times K$ convolution into a $(2d-1)\times(2d-1)$ depthwise convolution, a $\lceil K/d\rceil\times\lceil K/d\rceil$ depthwise dilated convolution with dilation $d$, and a $1\times1$ pointwise convolution [2202.09741]. For the default choice $K=21$, the paper uses $d=3$, giving a $5\times5$ depthwise convolution and a $7\times7$ depthwise dilated convolution [2202.09741].

This decomposition is the main source of LKA’s efficiency. VAN gives the parameter and FLOP expressions
$$
P(K,d)=C\lceil K/d\rceil^2 + C(2d-1)^2 + C^2, \qquad
F(K,d)=P(K,d)\cdot H\cdot W.
$$
For fixed $K$ and $d$, the spatial complexity is linear in $H\cdot W$, unlike self-attention, whose spatial complexity is quadratic in the number of locations [2202.09741]. The same linear-versus-quadratic contrast is restated in later 2D, 3D, and voxel-grid settings [2410.10433, 2207.11225, 2411.03672].

Ablation studies in VAN isolate the contribution of each component. On ImageNet-1K with VAN-B0, removing the local depthwise convolution yields $74.9\%$ top-1, removing the dilated depthwise convolution yields $74.1\%$, removing the $1\times1$ convolution yields $74.6\%$, adding sigmoid yields $75.2\%$, and the full LKA reaches $75.4\%$ [2202.09741]. Kernel-size ablations further show diminishing returns beyond the standard setting: $K=21,d=3$ and $K=28,d=4$ both give $75.4\%$ top-1 in VAN-B0 [2202.09741].

## 3. Structural variants and reinterpretations

A major line of work revisits the internal factorization of the large kernel. Large Separable Kernel Attention (LSKA) replaces the 2D depthwise kernels in LKA with cascaded horizontal and vertical 1D depthwise kernels. Formally, the local and dilated stages become
$$
\bar Z^C = W^{C}_{(2d-1)\times1} * \big(W^{C}_{1\times(2d-1)} * F^C\big), \qquad
Z^C = W^{C}_{\lfloor k/d\rfloor\times1} * \big(W^{C}_{1\times\lfloor k/d\rfloor} * \bar Z^C\big),
$$
followed by a $1\times1$ attention projection and multiplicative gating [2309.01439]. This changes the depthwise spatial cost from quadratic in kernel size to linear in kernel size. The paper gives
$$
\text{Params}_{\mathrm{DW\text{-}2D}} = Ck^2,\qquad
\text{Params}_{\mathrm{DW\text{-}1D}} = 2Ck,
$$
with analogous FLOP reductions [2309.01439]. In VAN-Tiny at $k=53$, LKA-trivial has $8.15$M parameters, $2.61$ GFLOPs, and speed $146$ images/s, whereas LSKA-trivial has $4.16$M parameters, $0.89$ GFLOPs, and speed $599$ images/s, with top-1 accuracy $75.1$ versus $74.9$ [2309.01439].

A second reinterpretation is Large Kernel Convolutional Attention (LKCA), which treats attention as a single learned large convolution on the patch grid. Given $N=HW$ patch embeddings, LKCA learns a kernel of size $(2H-1)\times(2W-1)$ and applies it by convolution after a pointwise projection:
$$
\mathrm{LKCA}(x)=\mathrm{Conv2d}(\mathrm{MLP}(x),\text{Large Kernel}).
$$
Under translation invariance and parameter sharing, the resulting attention matrix is block Toeplitz and equivalent to convolution with a static relative-position kernel [2401.05738]. This formulation retains full-image receptive field on the patch grid while avoiding explicit $N\times N$ attention matrices [2401.05738].

A third variant introduces geometric adaptivity. Deformable Large Kernel Attention (D-LKA) augments the large-kernel path with deformable convolutions so that the sampling grid itself is learned. In 2D, the deformable convolution is written as
$$
y(p_0)=\sum_{p_n\in R} w(p_n)\,x(p_0+p_n+\Delta p_n)\,m(p_n),
$$
and the D-LKA attention block is summarized as
$$
F'=\mathrm{GELU}(\mathrm{Conv}(F)),\quad
\mathrm{Attention}=\mathrm{Conv}_{1\times1}\big(\mathrm{DDW\text{-}D\text{-}Conv}(\mathrm{DDW\text{-}Conv}(F'))\big),
$$
$$
\mathrm{Output}=\mathrm{Conv}_{1\times1}(\mathrm{Attention}\otimes F') + F.
$$
The 3D version inserts only one deformable convolution into the large-kernel stack because a fully deformable 3D offset head would require $3k^3$ channels and become prohibitively expensive [2309.00121].

## 4. Three-dimensional and sequence-oriented generalizations

LKA was extended early to volumetric medical segmentation. In the 3D medical formulation, the module applies Group Normalization and leaky ReLU, then a depthwise $3$D convolution, a depthwise dilated $3$D convolution, a pointwise $1\times1\times1$ convolution, sigmoid, and residual gating:
$$
A = \sigma_{\mathrm{sigmoid}}\!\big(\mathrm{Conv}_{1\times1\times1}(\mathrm{ConvDW}(\mathrm{ConvDWD}(\sigma_{\mathrm{lReLU}}(\mathrm{GN}(X)))))\big),
$$
$$
Y = A \odot \sigma_{\mathrm{lReLU}}(\mathrm{GN}(X)) + \sigma_{\mathrm{lReLU}}(\mathrm{GN}(X)).
$$
The best “Mid” configuration uses a $5\times5\times5$ depthwise convolution, a $7\times7\times7$ depthwise dilated convolution with $d=3$, and approximates an effective kernel close to $21\times21\times21$ [2207.11225]. The same paper provides the decomposed 3D parameter expression
$$
N_{\mathrm{PRM,D}} = C\big((2d-1)^3 + (K/d)^3 + C + 3\big),
$$
contrasted with a dense large-kernel 3D convolution [2207.11225].

Autonomous-driving semantic scene completion adopts a related 3D instantiation inside a D-LKA-M backbone. MetaSSC computes a 3D attention map
$$
A=\mathrm{Conv}_{1\times1\times1}\big(\mathrm{DW3DConv}(\mathrm{DW3DConv}_{\mathrm{dil}}(x_1))\big),\qquad
\tilde x_1 = A \odot x_1,
$$
then passes the result to a Mamba block for long-sequence modeling [2411.03672]. Here LKA is explicitly positioned as a structure-aware alternative to flattening voxel grids for self-attention, and as a cheaper substitute for deep stacks of large-kernel 3D convolutions [2411.03672].

Large-kernel ideas also migrated outside vision grids. Time-aware Large Kernel (TaLK) Convolutions define a per-time-step adaptive window $[a_i^l,a_i^r]$ and compute
$$
r_i=\sum_{j=a_i^l}^{a_i^r} x_j,
$$
using prefix sums so that each adaptive large-kernel summation is obtained in $O(1)$ after building the summed-area table [2002.03184]. This is not the canonical VAN-style multiplicative LKA block, but it represents a sequence-modeling extension of the same large-kernel principle: broad contextual aggregation with linear-time complexity rather than quadratic token-token interaction [2002.03184].

## 5. Integration into full architectures and empirical adoption

Architecturally, LKA is usually not a standalone operator but a block embedded inside larger backbones. VAN places it in blocks of the form BatchNorm $\rightarrow$ $1\times1$ Conv $\rightarrow$ GELU $\rightarrow$ LKA $\rightarrow$ FFN, with a LayerScale-style residual update [2202.09741]. Task-specific models often relocate LKA to the decoder or bottleneck, where long-range context and boundary recovery are most critical. LKASeg uses LKA-based decoder units together with Full-Scale Skip Connections [2410.10433]; the self-supervised monocular depth model inserts one LKA block per decoder stage on top of HRNet-18 features [2409.17895]; MetaSSC populates both encoder and decoder stages of a U-Net-like 3D hierarchy with D-LKA blocks followed by Mamba [2411.03672].

Empirically, LKA has been adopted across a wide range of domains with consistent gains in either accuracy, efficiency, or both.

| Domain | Representative model | Reported result |
|---|---|---|
| General vision | VAN-B6 | 87.8% ImageNet top-1; 58.2 PQ panoptic segmentation [2202.09741] |
| Remote sensing segmentation | LKASeg | mF1 90.33%; mIoU 82.77% on ISPRS Vaihingen [2410.10433] |
| 3D medical segmentation | Mid-type LK attention U-Net | mean Dice 92.15 on CT-ORG; 85.81 on BraTS 2020 [2207.11225] |
| Autonomous-driving SSC | MetaSSC | 1st in IoU for scene completion; 2nd in Precision and SSC mIoU [2411.03672] |
| Vehicle re-identification | LKA-ReID | mAP 86.65%; Rank-1 98.03% [2409.17908] |
| Medium-range weather forecasting | PuYun cascade | day-10 Z500 RMSE 638 $m^2/s^2$ versus 641 for FuXi [2409.02123] |

Ablation results indicate that these gains are usually attributable to the large-kernel attention path itself rather than only to surrounding architectural changes. In LKASeg, replacing a Transformer decoder with LKA improves mIoU from $81.98\%$ to $82.49\%$ and reduces parameters and FLOPs from $11.69$M to $11.55$M and from $5.88$G to $5.76$G [2410.10433]. In self-supervised monocular depth estimation, adding LKA reduces GFLOPs from $10.78$ to $10.16$ and improves AbsRel from $0.096$ to $0.095$ [2409.17895]. In MetaSSC, the D-LKA backbone raises IoU from $51.4$ to $58.8$ and mIoU from $16.8$ to $20.4$, while D-LKA plus Mamba reaches IoU $65.0$ and mIoU $21.3$ [2411.03672].

## 6. Limitations, trade-offs, and ongoing directions

Despite its efficiency relative to self-attention, LKA is not free from scaling issues. The standard depthwise $k\times k$ component still grows quadratically with kernel size, which is the explicit motivation for LSKA, LCKA, and MM-LKA-style decompositions in later work [2309.01439, 2405.09353, 2506.12475]. In lightweight super-resolution, Large Coordinate Kernel Attention (LCKA) argues that the crucial property of LKA is the “adjacent direct interaction of local information and long-distance dependencies,” but that a naïve 2D depthwise design scales as $O(k^2)$ and should be replaced by coordinated horizontal and vertical 1D branches with overall linear growth in $k$ [2405.09353]. SDAN’s Multi-shape Multi-scale LKA further argues that a single square-kernel path is limited for “multi-shape multi-scale information,” and mixes strip and square branches to broaden shape coverage under low computational cost [2506.12475].

Another limitation is that larger kernels do not monotonically improve results. VAN reports identical $75.4\%$ top-1 for $K=21$ and $K=28$ in VAN-B0 [2202.09741]. PuYun’s weather ablation improves steadily from $K=5$ to $K=9$ but shows only marginal change from $K=9$ to $K=11$, leading the authors to select $K=9$ as a practical accuracy-memory trade-off [2409.02123]. This suggests that receptive-field enlargement can saturate once it exceeds the task’s useful spatial support.

Several papers also record accuracy trade-offs that complicate a purely efficiency-centered narrative. In MetaSSC, introducing D-LKA alone increases IoU but drops recall from $79.3$ to $70.8$ before the addition of Mamba recovers it to $74.0$ [2411.03672]. In 3D medical segmentation, bladder Dice slightly decreases relative to the base model even though mean Dice improves overall [2207.11225]. LKA-ReID improves retrieval performance, but the paper explicitly notes that the four-branch design increases architectural complexity [2409.17908].

Open directions in the literature are correspondingly concrete rather than rhetorical. They include adaptive kernel sizes and learned dilation across blocks, multi-scale LKA, unified temporal-frequency dynamic attention for audio, additional regularization or class balancing for 3D scene completion, boundary-aware supervision for real-time segmentation, and higher-resolution forecasting such as $0.1^\circ$ weather prediction [2306.06461, 2411.03672, 2508.07300, 2409.02123]. Taken together, these directions indicate that LKA has stabilized as a general design principle—large-receptive-field convolutional gating with task-adapted factorization—rather than as a single finalized module.

Source: https://www.emergentmind.com/topics/large-kernel-attention-lka