Large Kernel Attention (LKA)

• Large Kernel Attention (LKA) is a convolutional attention mechanism that uses decomposed large receptive field convolutions to efficiently capture long-range correlations.
• It integrates depthwise local, dilated, and pointwise convolutions to preserve 2D structural information while reducing complexity compared to quadratic self-attention.
• LKA has evolved into versatile variants like LSKA, LKCA, and D-LKA, finding applications in image classification, segmentation, medical imaging, and weather forecasting.

Searching arXiv for core and recent LKA papers to ground the article with citations. arXiv search query: Large Kernel Attention Visual Attention Network LSKA deformable LKA LKCA large kernel convolution attention MetaSSC PuYun LKA-ReID LKASeg Large Kernel Attention (LKA) is a convolutional attention mechanism that builds an attention map from large-receptive-field convolutional operations and applies it multiplicatively to the input feature map. In its canonical form, introduced in the Visual Attention Network (VAN), LKA replaces quadratic-cost self-attention with a sequence of depthwise local convolution, depthwise dilated convolution, and pointwise channel mixing, thereby preserving 2D structure, maintaining linear complexity in spatial size for fixed kernels, and introducing channel adaptability through a $1\times1$ projection (Guo et al., 2022). Subsequent work generalized this idea into separable, deformable, three-dimensional, and task-specific variants for image classification, detection, segmentation, super-resolution, depth estimation, semantic scene completion, audio event detection, vehicle re-identification, and medium-range weather forecasting (Lau et al., 2023, Azad et al., 2023, Li et al., 2022, Qu et al., 2024, Zhu et al., 2024).

1. Conceptual basis and historical emergence

LKA emerged from a specific critique of applying vanilla self-attention to vision. VAN identifies three issues: treating images as 1D token sequences neglects their 2D structure, the quadratic complexity is too expensive for high-resolution images, and standard self-attention captures spatial adaptability but ignores channel adaptability (Guo et al., 2022). LKA was proposed as a linear attention mechanism that preserves convolutional inductive biases while extending the receptive field sufficiently to model long-range correlations.

The same motivation recurs across later domains. Remote-sensing segmentation emphasizes that high-resolution aerial imagery requires global semantic capture without quadratic costs, while still preserving 2D topology and local detail (Xiang et al., 2024). In 3D medical segmentation, large kernels are motivated by extensive anatomical variation, low contrast, and the need for long-range volumetric context (Li et al., 2022). In autonomous-driving semantic scene completion, the argument is formulated in voxel space: self-attention scales as $O((HWD)^2)$ after flattening, whereas LKA preserves 3D topology and offers linear-time spatial modeling on sparse voxel grids (Qu et al., 2024).

A common misconception is that LKA denotes a single immutable block. The literature instead treats it as a design family. The canonical VAN operator omits sigmoid or softmax normalization in the attention path (Guo et al., 2022), whereas some later implementations normalize the attention map explicitly with sigmoid, including 3D medical segmentation, self-supervised monocular depth estimation, and sound event detection (Li et al., 2022, Xiang et al., 2024, Kim et al., 2023). What remains stable across these variants is the reliance on large-kernel convolutional filtering to generate an input-dependent gating map.

2. Canonical operator, decomposition, and complexity

In VAN, LKA takes an input feature map $X \in \mathbb{R}^{C\times H\times W}$ and computes

$$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$

followed by elementwise gating

$Y = X \odot A(X), \qquad A(X)=M(X).$

The large kernel is not implemented as a monolithic dense $K\times K$ operator. Instead, VAN decomposes a $K\times K$ convolution into a $(2d-1)\times(2d-1)$ depthwise convolution, a $\lceil K/d\rceil\times\lceil K/d\rceil$ depthwise dilated convolution with dilation $d$, and a $O((HWD)^2)$0 pointwise convolution (Guo et al., 2022). For the default choice $O((HWD)^2)$1, the paper uses $O((HWD)^2)$2, giving a $O((HWD)^2)$3 depthwise convolution and a $O((HWD)^2)$4 depthwise dilated convolution (Guo et al., 2022).

This decomposition is the main source of LKA’s efficiency. VAN gives the parameter and FLOP expressions

$O((HWD)^2)$5

For fixed $O((HWD)^2)$6 and $O((HWD)^2)$7, the spatial complexity is linear in $O((HWD)^2)$8, unlike self-attention, whose spatial complexity is quadratic in the number of locations (Guo et al., 2022). The same linear-versus-quadratic contrast is restated in later 2D, 3D, and voxel-grid settings (Xiang et al., 2024, Li et al., 2022, Qu et al., 2024).

Ablation studies in VAN isolate the contribution of each component. On ImageNet-1K with VAN-B0, removing the local depthwise convolution yields $O((HWD)^2)$9 top-1, removing the dilated depthwise convolution yields $X \in \mathbb{R}^{C\times H\times W}$0, removing the $X \in \mathbb{R}^{C\times H\times W}$1 convolution yields $X \in \mathbb{R}^{C\times H\times W}$2, adding sigmoid yields $X \in \mathbb{R}^{C\times H\times W}$3, and the full LKA reaches $X \in \mathbb{R}^{C\times H\times W}$4 (Guo et al., 2022). Kernel-size ablations further show diminishing returns beyond the standard setting: $X \in \mathbb{R}^{C\times H\times W}$5 and $X \in \mathbb{R}^{C\times H\times W}$6 both give $X \in \mathbb{R}^{C\times H\times W}$7 top-1 in VAN-B0 (Guo et al., 2022).

3. Structural variants and reinterpretations

A major line of work revisits the internal factorization of the large kernel. Large Separable Kernel Attention (LSKA) replaces the 2D depthwise kernels in LKA with cascaded horizontal and vertical 1D depthwise kernels. Formally, the local and dilated stages become

$X \in \mathbb{R}^{C\times H\times W}$8

followed by a $X \in \mathbb{R}^{C\times H\times W}$9 attention projection and multiplicative gating (Lau et al., 2023). This changes the depthwise spatial cost from quadratic in kernel size to linear in kernel size. The paper gives

$$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$0

with analogous FLOP reductions (Lau et al., 2023). In VAN-Tiny at $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$1, LKA-trivial has $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$2M parameters, $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$3 GFLOPs, and speed $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$4 images/s, whereas LSKA-trivial has $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$5M parameters, $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$6 GFLOPs, and speed $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$7 images/s, with top-1 accuracy $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$8 versus $$U = \operatorname{Conv}^{\mathrm{dw}}_{k_1}(X), \qquad V = \operatorname{Conv}^{\mathrm{dw}}_{k_2,d}(U), \qquad M(X) = \operatorname{Conv}_{1\times 1}(V),$$9 (Lau et al., 2023).

A second reinterpretation is Large Kernel Convolutional Attention (LKCA), which treats attention as a single learned large convolution on the patch grid. Given $Y = X \odot A(X), \qquad A(X)=M(X).$0 patch embeddings, LKCA learns a kernel of size $Y = X \odot A(X), \qquad A(X)=M(X).$1 and applies it by convolution after a pointwise projection:

$Y = X \odot A(X), \qquad A(X)=M(X).$2

Under translation invariance and parameter sharing, the resulting attention matrix is block Toeplitz and equivalent to convolution with a static relative-position kernel (Li et al., 2024). This formulation retains full-image receptive field on the patch grid while avoiding explicit $Y = X \odot A(X), \qquad A(X)=M(X).$3 attention matrices (Li et al., 2024).

A third variant introduces geometric adaptivity. Deformable Large Kernel Attention (D-LKA) augments the large-kernel path with deformable convolutions so that the sampling grid itself is learned. In 2D, the deformable convolution is written as

$Y = X \odot A(X), \qquad A(X)=M(X).$4

and the D-LKA attention block is summarized as

$Y = X \odot A(X), \qquad A(X)=M(X).$5

$Y = X \odot A(X), \qquad A(X)=M(X).$6

The 3D version inserts only one deformable convolution into the large-kernel stack because a fully deformable 3D offset head would require $Y = X \odot A(X), \qquad A(X)=M(X).$7 channels and become prohibitively expensive (Azad et al., 2023).

4. Three-dimensional and sequence-oriented generalizations

LKA was extended early to volumetric medical segmentation. In the 3D medical formulation, the module applies Group Normalization and leaky ReLU, then a depthwise $Y = X \odot A(X), \qquad A(X)=M(X).$8D convolution, a depthwise dilated $Y = X \odot A(X), \qquad A(X)=M(X).$9D convolution, a pointwise $K\times K$0 convolution, sigmoid, and residual gating:

$K\times K$1

$K\times K$2

The best “Mid” configuration uses a $K\times K$3 depthwise convolution, a $K\times K$4 depthwise dilated convolution with $K\times K$5, and approximates an effective kernel close to $K\times K$6 (Li et al., 2022). The same paper provides the decomposed 3D parameter expression

$K\times K$7

contrasted with a dense large-kernel 3D convolution (Li et al., 2022).

Autonomous-driving semantic scene completion adopts a related 3D instantiation inside a D-LKA-M backbone. MetaSSC computes a 3D attention map

$K\times K$8

then passes the result to a Mamba block for long-sequence modeling (Qu et al., 2024). Here LKA is explicitly positioned as a structure-aware alternative to flattening voxel grids for self-attention, and as a cheaper substitute for deep stacks of large-kernel 3D convolutions (Qu et al., 2024).

Large-kernel ideas also migrated outside vision grids. Time-aware Large Kernel (TaLK) Convolutions define a per-time-step adaptive window $K\times K$9 and compute

$K\times K$0

using prefix sums so that each adaptive large-kernel summation is obtained in $K\times K$1 after building the summed-area table (Lioutas et al., 2020). This is not the canonical VAN-style multiplicative LKA block, but it represents a sequence-modeling extension of the same large-kernel principle: broad contextual aggregation with linear-time complexity rather than quadratic token-token interaction (Lioutas et al., 2020).

5. Integration into full architectures and empirical adoption

Architecturally, LKA is usually not a standalone operator but a block embedded inside larger backbones. VAN places it in blocks of the form BatchNorm $K\times K$2 $K\times K$3 Conv $K\times K$4 GELU $K\times K$5 LKA $K\times K$6 FFN, with a LayerScale-style residual update (Guo et al., 2022). Task-specific models often relocate LKA to the decoder or bottleneck, where long-range context and boundary recovery are most critical. LKASeg uses LKA-based decoder units together with Full-Scale Skip Connections (Xiang et al., 2024); the self-supervised monocular depth model inserts one LKA block per decoder stage on top of HRNet-18 features (Xiang et al., 2024); MetaSSC populates both encoder and decoder stages of a U-Net-like 3D hierarchy with D-LKA blocks followed by Mamba (Qu et al., 2024).

Empirically, LKA has been adopted across a wide range of domains with consistent gains in either accuracy, efficiency, or both.

Domain	Representative model	Reported result
General vision	VAN-B6	87.8% ImageNet top-1; 58.2 PQ panoptic segmentation (Guo et al., 2022)
Remote sensing segmentation	LKASeg	mF1 90.33%; mIoU 82.77% on ISPRS Vaihingen (Xiang et al., 2024)
3D medical segmentation	Mid-type LK attention U-Net	mean Dice 92.15 on CT-ORG; 85.81 on BraTS 2020 (Li et al., 2022)
Autonomous-driving SSC	MetaSSC	1st in IoU for scene completion; 2nd in Precision and SSC mIoU (Qu et al., 2024)
Vehicle re-identification	LKA-ReID	mAP 86.65%; Rank-1 98.03% (Xiang et al., 2024)
Medium-range weather forecasting	PuYun cascade	day-10 Z500 RMSE 638 $K\times K$7 versus 641 for FuXi (Zhu et al., 2024)

Ablation results indicate that these gains are usually attributable to the large-kernel attention path itself rather than only to surrounding architectural changes. In LKASeg, replacing a Transformer decoder with LKA improves mIoU from $K\times K$8 to $K\times K$9 and reduces parameters and FLOPs from $(2d-1)\times(2d-1)$0M to $(2d-1)\times(2d-1)$1M and from $(2d-1)\times(2d-1)$2G to $(2d-1)\times(2d-1)$3G (Xiang et al., 2024). In self-supervised monocular depth estimation, adding LKA reduces GFLOPs from $(2d-1)\times(2d-1)$4 to $(2d-1)\times(2d-1)$5 and improves AbsRel from $(2d-1)\times(2d-1)$6 to $(2d-1)\times(2d-1)$7 (Xiang et al., 2024). In MetaSSC, the D-LKA backbone raises IoU from $(2d-1)\times(2d-1)$8 to $(2d-1)\times(2d-1)$9 and mIoU from $\lceil K/d\rceil\times\lceil K/d\rceil$0 to $\lceil K/d\rceil\times\lceil K/d\rceil$1, while D-LKA plus Mamba reaches IoU $\lceil K/d\rceil\times\lceil K/d\rceil$2 and mIoU $\lceil K/d\rceil\times\lceil K/d\rceil$3 (Qu et al., 2024).

6. Limitations, trade-offs, and ongoing directions

Despite its efficiency relative to self-attention, LKA is not free from scaling issues. The standard depthwise $\lceil K/d\rceil\times\lceil K/d\rceil$4 component still grows quadratically with kernel size, which is the explicit motivation for LSKA, LCKA, and MM-LKA-style decompositions in later work (Lau et al., 2023, Hao et al., 2024, Hao et al., 14 Jun 2025). In lightweight super-resolution, Large Coordinate Kernel Attention (LCKA) argues that the crucial property of LKA is the “adjacent direct interaction of local information and long-distance dependencies,” but that a naïve 2D depthwise design scales as $\lceil K/d\rceil\times\lceil K/d\rceil$5 and should be replaced by coordinated horizontal and vertical 1D branches with overall linear growth in $\lceil K/d\rceil\times\lceil K/d\rceil$6 (Hao et al., 2024). SDAN’s Multi-shape Multi-scale LKA further argues that a single square-kernel path is limited for “multi-shape multi-scale information,” and mixes strip and square branches to broaden shape coverage under low computational cost (Hao et al., 14 Jun 2025).

Another limitation is that larger kernels do not monotonically improve results. VAN reports identical $\lceil K/d\rceil\times\lceil K/d\rceil$7 top-1 for $\lceil K/d\rceil\times\lceil K/d\rceil$8 and $\lceil K/d\rceil\times\lceil K/d\rceil$9 in VAN-B0 (Guo et al., 2022). PuYun’s weather ablation improves steadily from $d$0 to $d$1 but shows only marginal change from $d$2 to $d$3, leading the authors to select $d$4 as a practical accuracy-memory trade-off (Zhu et al., 2024). This suggests that receptive-field enlargement can saturate once it exceeds the task’s useful spatial support.

Several papers also record accuracy trade-offs that complicate a purely efficiency-centered narrative. In MetaSSC, introducing D-LKA alone increases IoU but drops recall from $d$5 to $d$6 before the addition of Mamba recovers it to $d$7 (Qu et al., 2024). In 3D medical segmentation, bladder Dice slightly decreases relative to the base model even though mean Dice improves overall (Li et al., 2022). LKA-ReID improves retrieval performance, but the paper explicitly notes that the four-branch design increases architectural complexity (Xiang et al., 2024).

Open directions in the literature are correspondingly concrete rather than rhetorical. They include adaptive kernel sizes and learned dilation across blocks, multi-scale LKA, unified temporal-frequency dynamic attention for audio, additional regularization or class balancing for 3D scene completion, boundary-aware supervision for real-time segmentation, and higher-resolution forecasting such as $d$8 weather prediction (Kim et al., 2023, Qu et al., 2024, Huang et al., 10 Aug 2025, Zhu et al., 2024). Taken together, these directions indicate that LKA has stabilized as a general design principle—large-receptive-field convolutional gating with task-adapted factorization—rather than as a single finalized module.