LSM: LRU-Based Restoration Network

• The paper introduces LSM, a novel LRU-based restoration network that fuses linear recurrence with semantic modulation to achieve efficient super-resolution.
• LSM employs a hybrid architecture combining window-based attention and Category-based Modulated LRU blocks, surpassing state-of-the-art methods with fewer parameters and FLOPs.
• Empirical studies and ablations show that pixel-wise semantic modulation and dictionary-driven grouping significantly enhance global consistency and local texture reconstruction.

Searching arXiv for the specified paper and key related models to ground the article. An LRU-based restoration network, in the form introduced as LSM, is a single-image super-resolution architecture that combines a Linear Recurrent Unit (LRU) with a Semantic Modulating Unit (SMU) to balance performance and efficiency on 2D vision tasks. Its design responds to two stated limitations of prior LRU usage in vision—static parameterization and a single-scan method—by preserving a stable linear recurrence while introducing pixel-wise semantic modulation, semantic grouping for one-pass scanning, and prototype-driven feature enhancement. In the reported formulation, LSM quantitatively and qualitatively surpasses recent state-of-the-art methods while maintaining computational complexity on par with existing approaches (Choi et al., 18 Jun 2026).

1. Architectural definition and system-level organization

The backbone is organized as a shallow feature extractor, a stack of recurrent-attention groups, and a reconstruction head. The input low-resolution image is first mapped by a $3 \times 3$ convolution to a shallow feature tensor $F_0 \in \mathbb{R}^{H \times W \times C}$. This is followed by stacked groups, each group containing $n_{\text{block}}$ pairs of a Window-based Multi-head Self-Attention (WMS) block and a Category-based Modulated LRU (CML) block. After $n_{\text{group}}$ such groups, a pixel-shuffle upsampler produces the high-resolution output (Choi et al., 18 Jun 2026).

Within each CML block, the sequence is LayerNorm $\rightarrow$ SMU $\rightarrow$ Modulated LRU $\rightarrow$ gated MLP $\rightarrow$ residual skip. The SMU injects four modulating tokens $M_k$ whose three stated functions are LRU modulation, spatial categorization, and feature enhancement through a learned prototype. The key novelty is the replacement of heavy global attention or multi-scan state-space models by a static linear recurrence that is dynamically modulated by pixel-wise semantics from the SMU.

This organization suggests a division of labor between local windowed attention and global recurrent propagation. WMS provides local token mixing, whereas the CML block augments linear recurrence with semantic routing and feature enhancement, thereby addressing the mismatch between 1D recurrence and 2D image structure.

2. Linear recurrent formulation and stability properties

The LRU begins from the standard recurrent form

$$h_k = \phi(A h_{k-1} + B u_k), \qquad y_k = C h_k + D u_k,$$

but removes the nonlinearity $F_0 \in \mathbb{R}^{H \times W \times C}$0 in the recurrence to obtain a linear update. For scan-parallel computation, $F_0 \in \mathbb{R}^{H \times W \times C}$1 is diagonalized in the complex eigenbasis:

$F_0 \in \mathbb{R}^{H \times W \times C}$2

where $F_0 \in \mathbb{R}^{H \times W \times C}$3 are trainable eigenvalues, $F_0 \in \mathbb{R}^{H \times W \times C}$4 is a normalization factor, and $F_0 \in \mathbb{R}^{H \times W \times C}$5, $F_0 \in \mathbb{R}^{H \times W \times C}$6, $F_0 \in \mathbb{R}^{H \times W \times C}$7. The eigenvalues are parameterized as

$F_0 \in \mathbb{R}^{H \times W \times C}$8

with initialization sampling $F_0 \in \mathbb{R}^{H \times W \times C}$9 uniformly on an annulus $n_{\text{block}}$0 and phases in $n_{\text{block}}$1; the reported example is $n_{\text{block}}$2, $n_{\text{block}}$3, $n_{\text{block}}$4 (Choi et al., 18 Jun 2026).

Three properties are explicitly attributed to this recurrence: it is globally stable because $n_{\text{block}}$5, parallelizable, and able to capture long-range dependencies linearly. In the restoration setting, these properties define the LRU as the resource-efficient global propagation mechanism of the network.

3. Semantic Modulating Unit and semantic grouping

Given a token sequence $n_{\text{block}}$6 obtained by flattening spatial patches after WMS, and a learned dictionary $n_{\text{block}}$7 with $n_{\text{block}}$8, the SMU first performs cross-attention for feature enhancement:

$n_{\text{block}}$9

$n_{\text{group}}$0

$n_{\text{group}}$1

The affinity matrix $n_{\text{group}}$2 is then split into four chunks,

$n_{\text{group}}$3

each of size $n_{\text{group}}$4, and these are tiled or projected to match the LRU hidden-state size (Choi et al., 18 Jun 2026).

For semantic categorization, each spatial token $n_{\text{group}}$5 is assigned via Gumbel-Softmax with temperature $n_{\text{group}}$6 to a $n_{\text{group}}$7-way one-hot group $n_{\text{group}}$8. The input sequence is then reordered or masked so that semantically similar pixels are grouped before a single recurrence scan. This is explicitly described as “multi-scan in one pass,” and it is intended to alleviate the LRU’s 1D spatial ordering limitation.

The SMU therefore has three roles in one module: cross-attentive global feature enhancement, production of modulating tokens for the recurrent dynamics, and semantic pre-categorization for scan order restructuring. A plausible implication is that the learned dictionary acts as a shared latent organizer for both token enhancement and recurrence control, rather than serving only as a memory bank.

4. Modulated recurrence and forward propagation

The SMU modifies the basic LRU update by replacing the static recurrence with a token-conditioned version:

$n_{\text{group}}$9

$\rightarrow$0

where $\rightarrow$1 denotes the row of modulating tokens at step $\rightarrow$2. In implementation, the real and imaginary parts of $\rightarrow$3 are modulated separately (Choi et al., 18 Jun 2026).

The layer-wise forward pass is specified as follows. Given a low-resolution image $\rightarrow$4, the network computes $\rightarrow$5. For each group $\rightarrow$6 and block $\rightarrow$7, it applies local window MHSA,

$\rightarrow$8

The SMU on $\rightarrow$9 then computes: (i) $\rightarrow$0, (ii) $\rightarrow$1 and $\rightarrow$2, and (iii) semantic group assignment

$\rightarrow$3

The grouped tokens are processed by the modulated LRU recurrence,

$\rightarrow$4

after which global and recurrent outputs are fused as

$\rightarrow$5

The final stage upsamples by PixelShuffle to produce the high-resolution output $\rightarrow$6.

This forward path formalizes the network as a hybrid of local attention, dictionary-conditioned global enhancement, and semantically reordered linear recurrence.

5. Training protocol and computational profile

The training setup distinguishes between classic super-resolution and lightweight super-resolution. For classic SR, training uses DIV2K (800 train) plus Flickr2K (2650 train). For lightweight SR, training uses DIV2K only. Validation and testing are reported on Set5, Set14, B100, Urban100, and Manga109. Data augmentation consists of random horizontal and vertical flips and rotations by $\rightarrow$7, $\rightarrow$8, and $\rightarrow$9 (Choi et al., 18 Jun 2026).

Optimization uses the $\rightarrow$0 pixel loss $\rightarrow$1 and AdamW with $\rightarrow$2, $\rightarrow$3, and initial learning rate $\rightarrow$4. For $\rightarrow$5 SR, the schedule has two stages: Stage 1 uses patch size $\rightarrow$6 for $\rightarrow$7k iterations, with the learning rate halved at $\rightarrow$8k; Stage 2 uses patch size $\rightarrow$9 for LSM-S or $\rightarrow$0 for LSM for $\rightarrow$1k iterations, with the learning rate halved mid-way. Fine-tuning for $\rightarrow$2 and $\rightarrow$3 uses $\rightarrow$4k iterations from $\rightarrow$5 weights with no Stage 1.

On $\rightarrow$6 SR with $\rightarrow$7 input, the reported model variants and complexity are:

Variant	Parameters	GFLOPs
LSM-S	9.7 M	193.5
LSM	12.8 M	255.0
LSM-light	763 K	282.2

For Urban100 $\rightarrow$8, the reported comparisons are:

Model	Params / FLOPs	PSNR / SSIM
MambaIR	20.6 M / 394.6 G	27.68 dB / 0.8287
MambaIRv2-S	9.8 M / 202.9 G	27.73 dB / 0.8307
LSM-S	9.9 M / 203.5 G	27.88 dB / 0.8348
LSM	12.9 M / 265.0 G	27.94 dB / 0.8362

Compared to SwinIR or CAT-A, LSM is reported to achieve higher PSNR with 40–45 % fewer parameters and FLOPs. This places efficiency and accuracy as co-equal design targets rather than treating recurrence solely as a low-cost substitute for attention.

6. Empirical results, ablations, and interpretive insights

Across full tables on Set5, Set14, B100, Urban100, and Manga109 for scales $\rightarrow$9, $M_k$0, and $M_k$1, LSM is reported to outperform all efficient Transformers such as SwinIR, DAT, ART, HAT, and RGT, as well as state-space models such as MambaIR, under 20 M parameters. On $M_k$2 Set14, one example given is LSM-S at 34.66 dB / 0.9264 versus SwinIR at 34.46 dB / 0.9250. For larger models with self-ensemble, LSM+ achieves 40.47 dB / 0.9812 on Urban100 $M_k$3, reported as the best result. Qualitatively, circular and striped textures in img_004 and img_024 are described as sharper with fewer artifacts, while straight architectural lines in img_012 and img_059 and irregular curved shapes in Manga109 dolls and zebra stripes are reconstructed faithfully (Choi et al., 18 Jun 2026).

The ablation studies isolate several mechanisms. Varying $M_k$4 changes the ring shape of $M_k$5 in the complex plane; the best performance is reported at $M_k$6, $M_k$7, $M_k$8, while worse settings drop up to 0.19 dB. For the multi-role SMU, the reported Set14 values are 34.44 dB for the base model without SMU, 34.42 dB with categorization, 34.46 dB with cross-attention, and 34.56 dB with full $M_k$9 modulation, leading to the stated conclusion that each role contributes. For modulating-token components, enabling $$h_k = \phi(A h_{k-1} + B u_k), \qquad y_k = C h_k + D u_k,$$0, $$h_k = \phi(A h_{k-1} + B u_k), \qquad y_k = C h_k + D u_k,$$1, $$h_k = \phi(A h_{k-1} + B u_k), \qquad y_k = C h_k + D u_k,$$2, and $$h_k = \phi(A h_{k-1} + B u_k), \qquad y_k = C h_k + D u_k,$$3 each adds approximately 0.05–0.10 dB, and the coupled softmax design outperforms separate linear branches by approximately 0.1 dB.

Hidden-state visualization further differentiates the variants: vanilla LRU misses fine details such as flowers and beaks; adding categorization partially restores semantics; adding modulation yields both global consistency and local texture. The key insight stated for the model is that LRU provides a stable, resource-efficient global recurrence, while SMU injects adaptive, semantic-aware modulation without multi-scan overhead. The dictionary-driven gating thereby unifies global attention, grouping, and gating into a single lightweight module, with linear computational scaling.