DiNAT-IR: Dilated Neighborhood Attention

• DiNAT-IR is a Transformer-based image restoration method that leverages dilated neighborhood attention to balance local precision with expanded receptive field.
• It refines the Restormer U-Net backbone by alternating neighborhood attention with dilated variations and integrating a Channel-Aware Module for global context.
• Empirical evaluations demonstrate competitive PSNR/SSIM metrics across deblurring, defocus, deraining, and denoising tasks while maintaining low computational cost.

Searching arXiv for DiNAT-IR context papers and related methods. arxiv_search(query="DiNAT Restormer neighborhood attention image restoration", max_results=10) Searching arXiv for related papers. arxiv_search(query="DiNAT Restormer neighborhood attention image restoration", max_results=10) DiNAT-IR is a Transformer-based architecture for image restoration that explores Dilated Neighborhood Attention (DiNA) as an alternative to conventional self-attention formulations for high-resolution low-level vision. Introduced in "DiNAT-IR: Exploring Dilated Neighborhood Attention for High-Quality Image Restoration" (Liu et al., 23 Jul 2025), it targets the efficiency–quality trade-off that arises when full spatial self-attention is applied to restoration tasks such as motion deblurring, defocus deblurring, deraining, and real-world denoising. The model retains the four-stage U-Net backbone of Restormer while replacing a purely channel-wise attention emphasis with a hybrid design that alternates Neighborhood Attention (NA) and DiNA and complements them with a Channel-Aware Module (CAM), with the stated goal of integrating global context without sacrificing pixel-level precision (Liu et al., 23 Jul 2025, Zamir et al., 2021).

1. Problem setting and design rationale

The starting point for DiNAT-IR is the observation that Transformers have become a dominant paradigm in image restoration because self-attention can model long-range dependencies, but the computational cost of self-attention limits scalability to high-resolution images (Liu et al., 23 Jul 2025). Restormer addresses this by employing channel-wise self-attention, computing attention across channels instead of spatial dimensions. DiNAT-IR is motivated by the claim that, while this is effective, such an approach may overlook localized artifacts that are crucial for high-quality image restoration (Liu et al., 23 Jul 2025, Zamir et al., 2021).

DiNAT-IR therefore explores Dilated Neighborhood Attention as a middle ground between local precision and broader contextual aggregation. The paper states that DiNA balances global context and local precision by integrating sliding-window attention with mixed dilation factors, effectively expanding the receptive field without excessive overhead (Liu et al., 23 Jul 2025). At the same time, its preliminary experiments indicate that directly applying this global-local design to the classic deblurring task hinders accurate visual restoration, primarily because local attention still constrains global context understanding (Liu et al., 23 Jul 2025). The architectural response is to add a channel-aware module that supplies a lightweight global signal.

A central misconception addressed by the model design is that a high-level vision attention pattern can be transferred unchanged to restoration. DiNAT-IR explicitly reports that this direct transfer is inadequate for classic deblurring and requires channel-aware compensation (Liu et al., 23 Jul 2025). This suggests that restoration imposes stricter requirements on pixel-level fidelity than the high-level tasks from which DiNA was originally motivated.

2. Network organization and backbone structure

DiNAT-IR inherits the four-stage U-Net backbone of Restormer (Liu et al., 23 Jul 2025, Zamir et al., 2021). Each stage contains an encoder block that downsamples by a factor of 2 via strided convolution and a sequence of $N$ Dual Transformer Blocks, together with a decoder block that upsamples by a factor of 2 via pixel shuffle or transpose convolution and a matching sequence of Dual Transformer Blocks (Liu et al., 23 Jul 2025). Skip-connections connect encoder features to decoder features at the same spatial resolution.

The basic computational unit is the Dual Transformer Block. Its internal order is: LayerNorm; a self-attention module, alternating NA and DiNA, in parallel with the Channel-Aware Module; element-wise multiplication to fuse the two branches; residual addition; LayerNorm; a Feed-Forward Network with GELU nonlinearity; and a second residual addition (Liu et al., 23 Jul 2025). The paper further states that the number of Transformer blocks per stage and the hidden-channel width $C$ are kept the same as in Restormer, ensuring a comparable parameter count (Liu et al., 23 Jul 2025).

In verbal block-diagram form, the block operates as follows. Given input $X$, the self-attention branch produces $A(X)$, the CAM branch produces $M(X)$, and these outputs are fused as $A(X)\odot M(X)$ before the first residual path is applied. This is followed by normalization, a two-layer MLP-style FFN with hidden expansion, and a second residual path (Liu et al., 23 Jul 2025). Architecturally, the design preserves the macro-organization of a restoration U-Net while changing the internal attention mechanism.

3. Dilated Neighborhood Attention

The DiNA component is defined for input features $X \in \mathbb{R}^{B\times C\times H\times W}$. After projection to queries, keys, and values,

$$Q, K, V \in \mathbb{R}^{B\times d_k\times H\times W},$$

the output at spatial position $i$ is

$$y_i = \sum_{j\in\mathcal{N}_i^{(d)}} A_{ij}\cdot V_j,$$

with attention weights

$C$0

where the dilated neighborhood is

$C$1

In DiNAT-IR, attention alternates between $C$2 and a larger dilation, with example stage-wise pairs given as $C$3 in stage 1, $C$4 in stage 2, $C$5 in stage 3, and $C$6 in stage 4 (Liu et al., 23 Jul 2025).

The receptive-field argument is explicit. For window radius $C$7 and dilation $C$8, the effective side length is $C$9 (Liu et al., 23 Jul 2025). By selecting a large $X$0, DiNA samples a broad but sparse neighborhood while avoiding the quadratic token–token interaction cost of full self-attention.

The complexity comparison in the paper makes this trade-off concrete. Full global self-attention on $X$1 tokens costs

$X$2

whereas DiNA with window radius $X$3 costs

$X$4

Because $X$5 at high resolution, the paper characterizes the complexity as effectively linear in the number of pixels (Liu et al., 23 Jul 2025). In the context of image restoration, the significance of this formulation is not merely lower asymptotic cost, but the attempt to preserve local detail sensitivity while expanding contextual support.

4. Channel-Aware Module and block-level fusion

The Channel-Aware Module is introduced to compensate for the limits of purely local or sparse spatial attention. The paper’s motivation is explicit: purely local attention can miss large-scale context, and purely sparse DiNA can overlook fine detail (Liu et al., 23 Jul 2025). CAM therefore provides a lightweight global context signal through channel-wise gating and complements the spatial attention branch by reweighting channels based on a global descriptor.

For input $X$6, the global descriptor is computed by global average pooling:

$X$7

with

$X$8

After flattening and transposing to $X$9, a 1D convolution with kernel size 3 and a sigmoid generate

$A(X)$0

This is reshaped back to $A(X)$1, broadcast to $A(X)$2, and used as

$A(X)$3

The effect is a global, channel-wise gating mask (Liu et al., 23 Jul 2025).

Within the Dual Transformer Block, the fusion procedure is:

1. $A(X)$4;
2. $A(X)$5, where SA is NA or DiNA;
3. $A(X)$6;
4. $A(X)$7;
5. $A(X)$8;
6. $A(X)$9;
7. $M(X)$0 (Liu et al., 23 Jul 2025).

This structure makes the CAM branch multiplicative rather than additive at the attention-fusion point. A plausible implication is that the model uses CAM primarily as a content-adaptive gain control over spatial attention responses, rather than as an independent feature stream.

5. Optimization protocol and evaluation regime

DiNAT-IR is evaluated on motion deblurring, defocus deblurring, deraining, and real-world denoising, using the following datasets and metrics (Liu et al., 23 Jul 2025):

Task	Dataset(s)	Metrics
Motion deblurring	GoPro, HIDE	PSNR, SSIM
Defocus deblurring	DPDD dual-pixel dataset	PSNR, SSIM, MAE
Deraining	Rain100H / Rain100L / Test2800 / Test1200 / Test100	PSNR, SSIM
Real-world denoising	SIDD	PSNR, SSIM

The optimization details are task-specific but consistent in optimizer design. For motion deblurring on GoPro, the model uses PSNR loss and AdamW with $M(X)$1, $M(X)$2, an initial learning rate decayed from $M(X)$3 to $M(X)$4 under cosine annealing over 600K iterations, followed by fine-tuning at $M(X)$5 patches with batch size 8 and learning rate $M(X)$6 for 200K iterations (Liu et al., 23 Jul 2025). For DPDD, it uses PSNR loss with the same AdamW schedule for 300K iterations, with a checkpoint at 290K. For Rain13K, it uses $M(X)$7 loss with the same optimizer and schedule for 300K iterations plus 100K fine-tuning at $M(X)$8. For SIDD, it uses PSNR loss with the same optimizer and schedule for 300K iterations, with a checkpoint at 220K (Liu et al., 23 Jul 2025).

Training uses standard patch-based random cropping and horizontal/vertical flips inherited from Restormer, and the paper does not emphasize additional bespoke augmentations (Liu et al., 23 Jul 2025). This indicates that the reported behavior is primarily attributed to the attention-and-gating design rather than to an unusually specialized training pipeline.

6. Empirical behavior, limitations, and prospective directions

The main numerical comparison reported in the paper is against Restormer. On GoPro deblurring, DiNAT-IR achieves $M(X)$9 SSIM, compared with Restormer’s $A(X)\odot M(X)$0; the parameter counts are $A(X)\odot M(X)$1 and $A(X)\odot M(X)$2 respectively, and the FLOPs are $A(X)\odot M(X)$3 and $A(X)\odot M(X)$4 (Liu et al., 23 Jul 2025). On HIDE, it reports $A(X)\odot M(X)$5 versus Restormer’s $A(X)\odot M(X)$6. On the combined dual-pixel defocus benchmark, it reports $A(X)\odot M(X)$7 versus a best prior of approximately $A(X)\odot M(X)$8, with $A(X)\odot M(X)$9. On Rain100L, it reports $X \in \mathbb{R}^{B\times C\times H\times W}$0 SSIM versus Restormer’s $X \in \mathbb{R}^{B\times C\times H\times W}$1. On SIDD, it reports $X \in \mathbb{R}^{B\times C\times H\times W}$2 SSIM versus Restormer’s $X \in \mathbb{R}^{B\times C\times H\times W}$3 (Liu et al., 23 Jul 2025).

The qualitative discussion emphasizes sharper license-plate characters, cleaner text in deblurring, and crisper edges in rain and noise removal, as shown in Figures 1, 3, and 4 of the paper (Liu et al., 23 Jul 2025). The stated interpretation is that alternating NA/DiNA plus global channel gating preserves both fine strokes and broad structure.

The strengths identified by the paper are that the method balances local precision with expanded receptive field at very low overhead, that CAM efficiently injects true global context without blowing up FLOPs, and that the model is competitive or superior across deblurring, defocus, deraining, and denoising while keeping a Restormer-comparable model size (Liu et al., 23 Jul 2025). The limitations are equally explicit: ablation and architectural choices, including dilation pairs, were tuned primarily on GoPro; gains may vary on other data distributions; slight PSNR drop on some deraining benchmarks indicates room for task-specific tuning; and the hybrid NA–DiNA pattern requires selecting discrete dilation factors per stage (Liu et al., 23 Jul 2025).

The future directions named in the paper are learnable or dynamic dilation schedules, possibly conditioned on the input; richer channel-spatial cross-attention beyond simple global average pooling; extension of the ablation suite to more varied low-level and cross-modal tasks such as video restoration; and investigation of hybrid losses that combine perceptual and pixel terms (Liu et al., 23 Jul 2025). Taken together, these directions position DiNAT-IR not as a final resolution of attention design for restoration, but as a specific demonstration that dilated local attention becomes materially more effective when paired with an explicit global channel-aware gating mechanism.