---
title: 'DiNA: Dilated Neighborhood Attention'
url: https://www.emergentmind.com/topics/neighborhood-attention-dina
type: topic
---

# DiNA: Dilated Neighborhood Attention

Neighborhood Attention (DiNA) is a scalable, locality-controlled attention mechanism that extends classic windowed self-attention with dilation, providing exponential receptive field growth at fixed computational cost. By selectively sampling spatial or temporal neighborhoods with configurable stride, DiNA interpolates between purely local and quasi-global attention patterns, with implementations spanning vision, sequence modeling, and graph domains. DiNA’s design enables efficient content-dependent aggregation of both fine-grained local and long-range contextual information, forming a core building block in recent state-of-the-art architectures for detection, segmentation, image restoration, medical image analysis, emotion recognition, and heterophilic graph learning.

## 1. Mathematical Definition and Key Variants

Dilated Neighborhood Attention (DiNA) generalizes Neighborhood Attention (NA) by introducing a dilation factor $\delta$ into the spatial or temporal windowing scheme. For a 1-D input sequence $X \in \mathbb{R}^{n \times d_\text{model}}$, DiNA applies linear projections to obtain $Q, K, V \in \mathbb{R}^{n \times d_k}$ and, at each position $i$, computes attention over a subset of $k$ indices:
\[
\{\rho_j^\delta(i)\}_{j=1}^k, \qquad \rho_j^\delta(i) = i - \delta (j-1),
\]
clipped such that $\rho_j^\delta(i) \geq 1$ [2312.07507, 2209.15001]. The full attention output at $i$ is
\[
\mathrm{DiNA}_{k,\delta}(i) = \mathrm{softmax}\left(\frac{1}{\sqrt{d_k}}A_i^{(k,\delta)}\right)V_i^{(k,\delta)},
\]
where $A_i^{(k,\delta)}$ contains dot products of $Q_i$ with $K$ at the dilated neighbors, possibly including pairwise biases $b_{i,j}$.

In 2-D vision workloads, let $X \in \mathbb{R}^{B \times C \times H \times W}$ and consider a local attention window of radius $r$ (side length $2r+1$), sampled on a $\delta$-strided subgrid:
\[
N_p^\delta = \left\{(i+\alpha \delta, j+\beta \delta) \mid \alpha,\beta \in \{-r,\ldots,r\}\right\} \cap \text{valid coordinates}
\]
[2507.17892, 2209.15001, 2502.13693].

On graphs, DiNA acquires a topological form, augmenting Graph Attention Network (GAT) edge scores with directional or diffusion-based features extracted from the spectrum of a parameterized Laplacian, enabling edge-aware long-range aggregation [2403.01475].

## 2. Receptive Field Expansion and Complexity Characteristics

The canonical advantage of DiNA is exponential receptive field growth with depth or stage—without increasing per-layer cost—by progressive dilation scheduling:
- **NA**: receptive field grows linearly: $RF_\text{NA}(l) = 1 + l \cdot (k-1)$.
- **DiNA**: with $\delta_l = k^{l-1}$, receptive field grows exponentially: $RF_\text{DiNA}(l) \propto k^{l}$ [2209.15001, 2312.07507].

Both NA and DiNA exhibit $O(n k d)$ compute and $O(nk)$ memory cost ($n=$number of tokens, $k=$window size, $d=$channel/model dimension), in strong contrast to the $O(n^2 d)$ and $O(n^2)$ complexity of global self-attention [2502.13693]. In practice, DiNA enables much larger stackable depth or spatial resolution, as in high-resolution image restoration [2507.17892] and medical imaging [2502.13693].

## 3. Implementation Methodologies and Fused Kernels

Efficient DiNA implementations leverage two primary strategies:
- **Sliding-window gathering**: For each token, gather $k$ neighbors using stride $\delta$. Efficient vectorized/batched gather operations are employed in both 1-D and 2-D via CUDA kernels (NATTEN [2204.07143], fused dot-product kernels [2403.04690]).
- **Fused attention kernels**: The “fused neighborhood attention” paradigm merges $QK^T$, softmax, and $PV$ in a single GPU threadblock, keeping attention weights in registers/shared memory and reducing global memory bandwidth. Performance gains are dramatic—1D fused kernels accelerate inference by $+1072\%$ (FP32) and $+1607\%$ (FP16) versus naive kernels, with similar boosts in 2D/3D [2403.04690].

Implementations support both causal and non-causal variants (for sequence modeling: enforce $\rho_j^\delta(i) \leq i$ via dynamic masking and padding [2312.07507]), multi-head variants (partition feature dimension), and easy integration as PyTorch/torch.nn modules.

## 4. Architectural Integration in Deep Learning Models

DiNA is a foundational component in multiple state-of-the-art architectures. Canonical patterns include:
- **Alternated NA / DiNA Stacking**: Hierarchical transformers (e.g., DiNAT and DiNAT-IR) alternate standard NA (dense local) and DiNA (dilated, sparse global) layers, enhancing both local precision and global context [2209.15001, 2507.17892].
- **Residual and Hybrid Blocks**: DiNA modules are inserted after convolutional or feed-forward sub-blocks and combined via residual addition and normalization. In NAC-TCN, DiNA is coupled with dilated 1D convolutions, spatial dropout, and per-block residual connections [2312.07507]. In MedViTV2, DiNA is used in Local Feature Perception blocks interleaved with Kolmogorov-Arnold Network-based global blocks [2502.13693].

For image restoration, DiNA may be hybridized with channel-aware modules for enhanced global context, as in DiNAT-IR’s dual-attention transformer blocks [2507.17892]. In graphs, Directional DiNA introduces spectral edge features and topological rewiring to extend GATs [2403.01475].

## 5. Hyperparameterization and Practical Guidelines

The efficacy of DiNA depends on appropriate choices for kernel size $k$, dilation factors $\delta$, depth, and attention head count. Key observed tradeoffs [2312.07507, 2209.15001, 2507.17892, 2502.13693]:
- **Neighborhood size $k$**: Larger $k$ increases local context and feature extraction capacity, but linearly increases compute/memory per layer.
- **Dilation schedule $\delta$**: Exponential growth ($\delta=2^\ell$ or $k^{\ell-1}$ at layer/stage $\ell$) achieves global receptive fields. Too aggressive dilation at high resolution may cause loss of short-term cues; insufficient dilation at low resolution may fail to capture long-range dependencies.
- **Multi-head ($h$)**: More heads enhance pattern diversity, with linearly increasing cost.
- **Model dimension $d_\text{model}$**: Tuned for task and memory budget.
- **Alternation and hybridization**: Alternating between NA and DiNA within or across stages/stacks balances fine detail and global awareness [2209.15001, 2507.17892].

Example hyperparameters: $k\in\{3,5,7\}$, typical $\delta$ values per stage $\{36,18,9,4\}$ for image restoration, per-layer scaling for temporal modeling, and $h\in\{2,4,8\}$.

## 6. Application Domains and Empirical Performance

DiNA underlies several high-performance, resource-efficient architectures across domains:
- **Vision Transformers**: DiNAT achieves state-of-the-art on COCO detection (box AP +1.6% over Swin), ADE20K segmentation (mIoU +1.4% over Swin), and competitive image classification [2209.15001].
- **Image Restoration**: DiNAT-IR surpasses Restormer and other baselines on GoPro deblurring (PSNR 33.80 dB vs. 32.92 dB), HIDE, DPDD, SIDD, and Rain datasets, with negligible FLOPs/memory overhead [2507.17892].
- **Medical Imaging**: MedViTV2, using enhanced DiNA with KAN integration, improves clean accuracy by up to 6.2 percentage points and robust accuracy by 5.8 on MedMNIST tasks, also reducing computational demand by 44% vs. earlier models [2502.13693].
- **Temporal Modeling**: NAC-TCN with causal DiNA outperforms LSTM, GRU, and standard TCNs on emotion recognition from sequences, with fewer parameters and lower compute [2312.07507].
- **Graph Neural Networks**: Directional DiNA outperforms vanilla and state-of-the-art GAT/GNN variants on node classification in both homophilic and heterophilic benchmarks, with up to 10–20 points improvement on difficult heterophilic settings [2403.01475].

## 7. Theoretical Analysis, Limitations, and Future Directions

DiNA provides a principled mechanism for balancing computational efficiency and receptive field size. The exponential receptive field scaling with layer/stage and the ability to fuse content-adaptivity with sparse sampling distinguish DiNA from both pure convolutions and global attention. In graph regimes, using spectral/geometric features supplies theoretically justified control of message-passing efficacy via Laplacian eigenstructures [2403.01475].

Known limitations include:
- For tasks requiring fine-grained short-term features, overly large dilation may skip important details.
- Current fused kernels support forward pass only in some versions, limiting efficiency for training in particular frameworks [2403.04690].
- Tuning dilation and alternation schedules is task-dependent and may require ablation.

Continued open-source support via NATTEN and integration in major frameworks is accelerating adoption [2204.07143, 2403.04690]. There is active research on expanding fused kernel coverage, autograd/bwd support, higher-dimensional and multi-modal extensions, and hybridizations with other locality/globality mechanisms.

---

**References**:  
- "NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding" [2312.07507]  
- "Dilated Neighborhood Attention Transformer" [2209.15001]  
- "Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level" [2403.04690]  
- "DiNAT-IR: Exploring Dilated Neighborhood Attention for High-Quality Image Restoration" [2507.17892]  
- "Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention" [2502.13693]  
- "Neighborhood Attention Transformer" [2204.07143]  
- "Representation Learning on Heterophilic Graph with Directional Neighborhood Attention" [2403.01475]

Source: https://www.emergentmind.com/topics/neighborhood-attention-dina