Lightweight Dimension Reductive Attention (LIDR)

• LIDR is a multi-branch mechanism that splits reduced-dimensional input into a low-frequency branch using multi-head self-attention and two local high-frequency branches using lightweight convolution.
• It reduces spatial and channel dimensions before performing attention, significantly lowering FLOPs and parameter counts while maintaining segmentation accuracy.
• Empirical tests in Light-UNETR show LIDR yields improved Dice scores and efficiency compared to traditional full-resolution MHSA plus FFN blocks.

Searching arXiv for the specified paper and closely related lightweight dimension-reductive attention work. Lightweight Dimension Reductive Attention (LIDR) is a multi-branch attention module introduced in Light-UNETR for efficient 3D medical image segmentation. In that setting, LIDR reduces spatial and channel dimensions, then combines a low-frequency global branch based on multi-head self-attention with two high-frequency local branches based on lightweight convolution, before restoring the original spatial resolution. Within Light-UNETR, it is the sole self-attention-style component in each block, and it is paired with a Compact Gated Linear Unit (CGLU) and a Contextual Synergic Enhancement (CSE) learning strategy to address model efficiency and data efficiency simultaneously (Liu et al., 24 Mar 2026).

1. Architectural position and design objective

Light-UNETR follows a classic UNet encoder–decoder with four stages. Each stage’s basic block is a “Light-UNETR Block” composed of a 3×3 depthwise-conv for local smoothing, a Lightweight Dimension Reductive Attention (LIDR) module, and a Compact Gated Linear Unit (CGLU). Spatial down- and up-sampling between stages use lightweight conv/transposed-conv + normalization + SE. In this arrangement, LIDR replaces a costly full-resolution MHSA+FFN with a reduced-dimension attention mechanism that preserves both global and local feature extraction (Liu et al., 24 Mar 2026).

The module is motivated by the observation that transformer-based 3D segmentation systems can achieve strong performance, yet their high computational requirements and need for large amounts of labeled data limit applicability. LIDR addresses the model-efficiency side of that problem. In the full Light-UNETR framework, this efficiency-oriented block is combined with CSE, which first leverages extrinsic contextual information through Attention-Guided Replacement and then applies Spatial Masking Consistency using intrinsic contextual information for unlabeled data. On the Left Atrial Segmentation dataset with only 10% labeled data, the overall method surpasses BCP by 1.43% Jaccard while drastically reducing the FLOPs by 90.8% and parameters by 85.8%; these end-to-end gains pertain to the complete Light-UNETR plus CSE pipeline rather than to LIDR in isolation (Liu et al., 24 Mar 2026).

2. Formal construction of the module

At a given stage, the input activations are denoted by

$x \in \mathbb{R}^{H \times W \times D \times C}.$

LIDR begins with spatial dimension reduction:

$X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$

where $r$ is the per-stage spatial reduction ratio. This yields

$$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$

The reduced tensor is then split evenly along the channel axis into three branches:

$$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$

each of shape

$$\mathbb{R}^{H/r \times W/r \times D/r \times (C/3)}.$$

The low-frequency branch performs MHSA on the reduced representation. Queries, keys, and values are generated by learned linear projections:

$$Q = W_q X_{\mathrm{low}}^{\mathrm{in}}, \qquad K = W_k X_{\mathrm{low}}^{\mathrm{in}}, \qquad V = W_v X_{\mathrm{low}}^{\mathrm{in}},$$

where $W_q, W_k, W_v \in \mathbb{R}^{C/3 \times d_k}$ and $d_k = C/3 \div N_h$ per head. For each head $h=1,\dots,N_h$,

$X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$0

The head outputs are concatenated to produce

$X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$1

The two high-frequency branches use lightweight convolution. For branch $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$2 with kernel size $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$3,

$X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$4

where $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$5 is a $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$6 convolution projecting $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$7, BN denotes batch-normalization, and $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$8 is a depthwise or group convolution that restores the channel count to $X_s = \mathrm{AvgPool3d}(x; \mathrm{stride}=r),$9.

The three outputs are concatenated:

$r$0

and the module restores full resolution with

$r$1

so that

$r$2

This sequence makes the reduction step, the branch separation, and the spatial restoration explicit (Liu et al., 24 Mar 2026).

3. Global–local decomposition and frequency interpretation

LIDR is organized around a frequency-oriented division of labor. The low-frequency branch via MHSA pools tokens and computes full-range attention, thus modeling long-range anatomical context and smooth, coarse structures. The two high-frequency branches use small and larger group convolutions to capture edges and textures at different scales. The resulting concatenation yields a feature map with both broad context and fine detail (Liu et al., 24 Mar 2026).

Fourier-analysis of encoder and decoder activations is presented as evidence for this interpretation. In the reported analysis, low-frequency branch spectra concentrate near the DC region, while the high-frequency branches light up surrounding higher-frequency rings. The design therefore treats dimension reduction not merely as a compression device but as a means of disentangling global and local frequency content before recombination (Liu et al., 24 Mar 2026).

A common simplification is to view LIDR as only a down-sampled MHSA path. That description is incomplete. The module includes one global attention branch and two convolutional local branches after channel splitting, and the efficacy claim is tied to that multi-branch composition rather than to pooling alone. This suggests that the term “dimension reductive” in this context refers simultaneously to reduced spatial resolution, reduced per-branch channel width, and structured redistribution of representational roles across branches.

4. Empirical efficiency and ablation evidence

At block level, the reported comparison is between a Light-UNETR block instantiated with vanilla MHSA+FFN and the LIDR-based alternative at $r$3 and $r$4 (Liu et al., 24 Mar 2026).

Mechanism	Cost	Dice on LA@5%
Vanilla MHSA+FFN	4.83 G FLOPs, 2.68 M params	89.05%
LIDR	4.29 G FLOPs, 1.34 M params	89.53%

Under that comparison, LIDR cuts parameters by approximately 50% and FLOPs by approximately 11%, while improving segmentation accuracy by +0.48 Dice. The data therefore support the claim that the module is not only lighter than a vanilla attention-plus-FFN block but also more accurate in the tested setting (Liu et al., 24 Mar 2026).

Ablation studies further quantify branch design. With one high-frequency branch, a 3×3×3 kernel yields Dice 88.49, whereas a 5×5×5 kernel yields Dice 88.31. With two branches, 3×3×3 and 3×3×3 gives Dice 89.41, 3×3×3 and 5×5×5 gives Dice 89.53, and 5×5×5 and 7×7×7 gives Dice 89.54. The reported interpretation is that the two-branch design with $r$5 yields the best trade-off: lowest cost and top accuracy (Liu et al., 24 Mar 2026).

The development trajectory reported in the paper also places LIDR within a broader block redesign. The sequence moves from Baseline ViT+FFN at 1.86 M parameters and 86.16 Dice, to + Overlap Patch Embedding at 1.92 M and 88.38, then + 2-branch LIDR at 1.60 M and 88.41, + 3-branch LIDR at 1.55 M and 88.32, + heterogeneous kernels at 1.61 M and 88.49, + gated FFN → CGLU at 1.62 M and 89.28, + compact expansion factor at 1.34 M and 89.36, and finally full Light-UNETR with CSE at 1.34 M and 89.53. This sequence indicates that LIDR contributes a measurable portion of the efficiency gain, but that the final reported accuracy depends on the interaction between LIDR, CGLU, compact expansion, and CSE rather than on the attention module alone (Liu et al., 24 Mar 2026).

5. Reproduction parameters and implementation profile

The critical hyperparameters reported for reproducing the module are explicit. The spatial reduction ratios across stages 1 to 4 are $r$6. The number of channels per stage is $r$7, and the number of Light-UNETR blocks per stage is $r$8. The number of MHSA heads is typically 4 or 8, with $r$9 per head. The high-frequency kernels are $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$0 and $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$1, the group-convolution group count is $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$2, the post-split linear projections are $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$3 convolutions with output dimension $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$4, normalization in each high-frequency branch is BatchNorm followed by GELU, and spatial restoration uses a depthwise ConvTranspose3d with stride $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$5 and kernel size $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$6 (Liu et al., 24 Mar 2026).

The forward pass can be summarized as five operations: spatial downsampling by AvgPool3d; equal channel splitting into low-frequency and two high-frequency inputs; reduced-resolution global attention on the low-frequency branch; local group convolution on the two high-frequency branches; and concatenation followed by depthwise transposed-convolution upsampling. Because the attention path operates after spatial reduction and only on one-third of the channels, while the remaining two-thirds are processed through lightweight local operators, the module realizes a specific efficiency strategy rather than a generic low-rank or sparse approximation (Liu et al., 24 Mar 2026).

This implementation profile also clarifies the scope of replacement. In the Light-UNETR block, LIDR is not an auxiliary refinement inserted around a conventional transformer; it is the central attention mechanism of the block. The surrounding 3×3 depthwise-conv and CGLU then provide local smoothing and selective channel interaction with minimal parameters.

6. Terminological scope and related reduction-based attention designs

In the available arXiv material, the term LIDR is used most concretely for the Light-UNETR module just described. However, related papers use the phrase more broadly to denote attention mechanisms that reduce effective dimension before or during attention computation. In "LatentLLM: Attention-Aware Joint Tensor Compression," the phrase is used to describe compression of full-rank multi-head self-attention into a much smaller latent attention without fine-tuning. There, heavy projection matrices $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$7 are approximated by $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$8, joint QK compression is formulated as a Tucker decomposition, and the per-token FLOPs of a matrix–vector product are reduced from $$X_s \in \mathbb{R}^{H/r \times W/r \times D/r \times C}.$$9 to $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$0 (Koike-Akino et al., 23 May 2025).

A different generalization appears in "Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in LLMs," where a rank-$$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$1 factorization is paired with an RL agent that selects ranks dynamically from state features including sequence dynamics, layer statistics, and previous rank. The reported low-rank attention complexity drops from $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$2 per head to $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$3, and for sequence length $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$4 the paper reports approximately $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$5 FLOPs for full-rank attention, approximately $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$6 for static low-rank attention with $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$7, and approximately $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$8 for the DR-RL approach (Erden, 17 Dec 2025).

A related but differently named mechanism is Fast Window Attention (FWA), introduced as a lightweight SoftMax attention mechanism with adaptive feature map sizes. FWA reduces the number of key and value tokens by full-scene adaptive window aggregation, replacing the usual $$[X_{\mathrm{low}}^{\mathrm{in}}, X_{h1}^{\mathrm{in}}, X_{h2}^{\mathrm{in}}] = \mathrm{split\_channels}(X_s; 3),$$9-length key sequence with an $$\mathbb{R}^{H/r \times W/r \times D/r \times (C/3)}.$$0-length sequence and changing complexity from $$\mathbb{R}^{H/r \times W/r \times D/r \times (C/3)}.$$1 to $$\mathbb{R}^{H/r \times W/r \times D/r \times (C/3)}.$$2. It also replaces SoftMax with a ReLU-based normalization termed DReLu (Li et al., 2 Aug 2025).

These neighboring formulations do not describe the same module as Light-UNETR’s LIDR. Rather, they indicate that “dimension reductive attention” has a broader interpretive use across recent work on efficient attention: spatial reduction in 3D segmentation, latent-factor compression in foundation models, adaptive low-rank rank control in LLMs, and adaptive key-sequence shortening in lightweight vision backbones. This suggests that the specific Light-UNETR formulation is one instance within a wider family of mechanisms that lower attention cost by reducing spatial extent, channel dimension, latent rank, or sequence length before full attention scaling becomes prohibitive.