---
title: 'ECG-NAT: Neighborhood Attention Transformer'
url: https://www.emergentmind.com/topics/electrocardiogram-neighborhood-attention-transformer-ecg-nat
type: topic
---

# ECG-NAT: Neighborhood Attention Transformer

Electrocardiogram Neighborhood Attention Transformer (ECG‑NAT) denotes a self-supervised transformer architecture for multi-lead electrocardiogram arrhythmia classification that combines masked-autoencoder pretraining, discriminative fine-tuning, and neighborhood attention to model localized beat morphology and broader rhythm structure with lower computational cost than global self-attention [2605.13194]. In the formulation reported for ECG‑NAT, the model is tailored to 12-lead, 10-second ECG segments acquired under realistic clinical constraints, including noise, inter-patient variability, long temporal extent, and limited labeled data; its central claim is that hierarchical local attention, coupled with multi-dataset pretraining and dual-loss fine-tuning, yields strong classification performance and efficiency on PTB‑XL and CPSC2018 [2605.13194].

## 1. Definition and problem setting

ECG‑NAT is designed for multi-lead ECG arrhythmia classification, specifically for 12-lead, 10-second ECG segments in realistic clinical conditions [2605.13194]. The problem setting emphasized for the architecture includes noisy signals, substantial variability across patients and acquisition setups, scarce labeled data because annotation requires cardiologists, and long sequences that make naïve global self-attention expensive [2605.13194]. In the reported implementation, ECG‑NAT addresses these constraints through a two-stage self-supervised pipeline: generative pretraining on unlabeled ECGs followed by discriminative fine-tuning on labeled datasets [2605.13194].

The input representation is a 12-lead ECG originally sampled at 500 Hz and standardized through a preprocessing pipeline consisting of a 0.5–40 Hz digital band-pass filter, amplitude normalization, and downsampling to 250 Hz, yielding 2500 time points per lead for a 10-second segment [2605.13194]. All signals are standardized to 10 seconds; shorter recordings are zero-padded, longer recordings are segmented into non-overlapping 10-second windows, and the first segment is used for classification [2605.13194]. The final input tensor is expressed as
\[
X \in \mathbb{R}^{L \times D}, \quad L=12,\; D=2500.
\]

The motivation for ECG‑NAT is explicitly architectural. CNNs are described as effective for local morphology but limited for long-range rhythm dependencies unless made very deep or wide; RNNs and LSTMs are described as slow to train and limited on very long sequences; standard transformers capture long-range dependencies but have \(O(N^2)\) complexity and require large labeled datasets to avoid overfitting [2605.13194]. ECG‑NAT therefore targets simultaneous modeling of beat-level morphology and rhythm-level dependencies while maintaining computational efficiency through Neighborhood Attention Transformer blocks [2605.13194].

## 2. Architectural organization

The encoder begins with a tokenizer implemented by two Conv1d layers with kernel size 3 and stride 2, reducing the temporal dimension by a factor of four from 2500 to 625 [2605.13194]. Leads are treated as channels, and the convolutions produce a feature map \(U \in \mathbb{R}^{L' \times D'}\), where \(D' = 625\) and the channel dimension grows through the hierarchy [2605.13194]. The use of overlapping convolutional windows preserves continuity of morphology across token boundaries [2605.13194].

Above the tokenizer, ECG‑NAT uses a hierarchical encoder composed of four NAT blocks interleaved with convolutional down-samplers [2605.13194]. The NAT blocks use embedding dimension 96, multi-head attention with heads \((2, 4, 8, 16)\) across stages, and MLP ratio 4.0 [2605.13194]. Between these blocks, down-sampler modules consisting of Conv1d layers with kernel size 2, stride 2, and ReLU reduce temporal resolution while increasing channel dimension according to
\[
[L', D'] \rightarrow [2L', D'/2].
\]
This yields the temporal pyramid 625 \(\rightarrow\) 312 \(\rightarrow\) 156 \(\rightarrow\) 78, with channels growing for example as 96 \(\rightarrow\) 192 \(\rightarrow\) 384 \(\rightarrow\) 768 [2605.13194].

The final encoder output is
\[
Z \in \mathbb{R}^{C \times T},
\]
with \(C\) large, for example 768, and \(T \approx 78\) [2605.13194]. For classification, this representation is flattened and passed to a fully connected layer [2605.13194]. This encoder design is explicitly hierarchical: early stages operate at high temporal resolution and are intended to emphasize detailed morphology such as P waves, QRS complexes, T waves, and ST segments, whereas later stages operate at lower resolution with larger effective receptive fields and are intended to emphasize broader rhythm patterns such as atrial fibrillation or conduction blocks [2605.13194].

A notable design decision is that multi-lead information is fused implicitly rather than by explicit cross-lead attention. The 12 leads are treated as channels of a multi-channel 1D signal, and both the Conv1d layers and NAT blocks operate along the temporal dimension while the channel dimension aggregates information from different leads [2605.13194]. The model therefore encodes all leads jointly as a multi-channel signal rather than as separate lead sequences [2605.13194].

## 3. Neighborhood attention and multi-scale temporal modeling

The defining mechanism of ECG‑NAT is neighborhood attention. In standard self-attention, each token attends to all \(N\) tokens,
\[
\text{SA}(i)=\mathrm{softmax}\left(\frac{Q_iK^\top}{\sqrt{d}}\right)V,
\]
with \(O(N^2)\) complexity [2605.13194]. ECG‑NAT instead restricts each token to its \(k\) nearest temporal neighbors [2605.13194]. For token \(i\), the attention logits over its neighborhood \(\{\mu_1(i), \dots, \mu_k(i)\}\) are defined as
\[
A^k_i =
\begin{bmatrix}
Q_i K_{\mu_1(i)}^\top + B(i,\mu_1(i)) \\
Q_i K_{\mu_2(i)}^\top + B(i,\mu_2(i)) \\
\vdots \\
Q_i K_{\mu_k(i)}^\top + B(i,\mu_k(i))
\end{bmatrix},
\]
where \(B(i,\mu_j(i))\) is a relative positional bias encoding the distance between \(i\) and its \(j\)-th neighbor [2605.13194]. The corresponding values are collected as
\[
V_i^k =
\begin{bmatrix}
V_{\mu_1(i)}^\top & V_{\mu_2(i)}^\top & \cdots & V_{\mu_k(i)}^\top
\end{bmatrix}^\top,
\]
and neighborhood attention becomes
\[
NA_k(i)=\mathrm{softmax}\left(\frac{A^k_i}{\sqrt{d}}\right)V_i^k.
\]
The resulting complexity is \(O(Nk)\) rather than \(O(N^2)\) [2605.13194].

The architectural rationale is explicitly multi-scale. Initial Conv1d tokenization reduces 2500 time points to 625 tokens; subsequent down-samplers reduce this to 312, 156, and 78 [2605.13194]. Early NAT layers therefore operate on finer temporal structure, while deeper NAT layers operate on coarser representations whose receptive fields span more cardiac cycles [2605.13194]. This is described as a hierarchical temporal pyramid in which early layers capture detailed beat morphology and later layers capture longer-term rhythm [2605.13194].

Hyperparameter analysis reported in the paper identifies a neighborhood kernel size \(k = 7\) as the best-performing setting [2605.13194]. Smaller kernels are described as capturing local morphology well, while overly large kernels increase cost and reduce the locality advantage [2605.13194]. The design is therefore neither purely convolutional nor globally transformer-like: convolutions impose local inductive bias, and neighborhood attention provides content-dependent weighting within local temporal contexts, with receptive fields expanding hierarchically across layers [2605.13194].

This places ECG‑NAT within a broader line of ECG architectures that prioritize local or hybrid local-global attention. Earlier transformer-based ECG stress detection used a convolutional front-end followed by a single global self-attention encoder on downsampled tokens, explicitly separating local feature extraction and global temporal integration, although it did not implement neighborhood attention [2108.09737]. Shifted-window ECG transformers similarly restricted self-attention to local windows at multiple scales for 12-lead classification [2306.12098], while local-global attention models used local overlapping windows to construct queries and global keys and values to preserve rhythm-level context [2504.16097]. ECG‑NAT differs by making neighborhood attention the central temporal operator inside a self-supervised multi-lead architecture [2605.13194].

## 4. Self-supervised pretraining and discriminative fine-tuning

The first stage of ECG‑NAT is generative pretraining on unlabeled ECGs from two datasets, Chapman and Ningbo [2605.13194]. After tokenization, the model randomly selects a subset of tokens and masks them using Gaussian noise with standard deviation 0.2 [2605.13194]. The encoder processes the corrupted sequence, and a decoder reconstructs the original clean ECG signal [2605.13194]. The decoder is a lightweight ConvTranspose1d stack: one ConvTranspose1d layer with kernel 3 and stride 2 to increase temporal dimension from 78 to 156, followed by four ConvTranspose1d layers with kernels \((2,2,2,4)\) and strides \((2,2,2,4)\), without padding, to recover a 12-lead waveform of length 2500 [2605.13194].

The reconstruction objective is mean squared error over masked indices:
\[
L_{\text{recon}} = \sum_{i \in I} \| X_i - \hat{X}_i \|^2, \quad \hat{X} = \text{rec}(X).
\]
Pretraining on multiple heterogeneous datasets is described as promoting domain-invariant representations that are robust to dataset-specific artifacts and not tied to a single distribution [2605.13194].

The second stage is discriminative fine-tuning on labeled datasets [2605.13194]. Two modes are reported. In linear evaluation, the pretrained encoder is frozen and only a linear classifier is trained [2605.13194]. In full fine-tuning, the encoder and classifier are optimized end-to-end using a dual loss that combines supervised contrastive loss and cross-entropy [2605.13194]. With batch index set \(I\), embeddings \(Z_i\), class labels \(y_i\), positives \(P(i)=\{p\in A(i): y_p=y_i\}\), and cosine similarity
\[
\text{sim}(Z_1, Z_2) = \frac{Z_1^\top Z_2}{\|Z_1\|\|Z_2\|},
\]
the supervised contrastive term is
\[
L_{\text{con}} = -\sum_{i\in I} \frac{1}{|P(i)|} \sum_{p\in P(i)}
\log \frac{\exp(\text{sim}(Z_i, Z_p)/\tau)}{\sum_{j \in A(i)} \exp(\text{sim}(Z_i, Z_j)/\tau)}.
\]
The classification term is
\[
L_{\text{cls}} = -\sum_{i \in I} \sum_{c=1}^C y_{ic} \log \hat{y}_{ic},
\]
and the combined loss is
\[
L_t = \alpha L_{\text{con}} + (1-\alpha)L_{\text{cls}}.
\]
The intended effect is to optimize label prediction while shaping a discriminative embedding space with tighter intra-class clusters and stronger inter-class separation [2605.13194].

The supervised contrastive component is supported empirically by representation analysis. t-SNE visualizations show improved cluster separation when the contrastive term is used, and for CPSC2018 the silhouette score improved by 13.7% while the Davies–Bouldin index was reduced by 15.4% [2605.13194].

## 5. Empirical evaluation and computational profile

Pretraining uses unlabeled ECGs from Chapman, with 10,646 patients and 10,646 records, and Ningbo, with 45,152 patients and 34,905 records [2605.13194]. Fine-tuning and evaluation use PTB‑XL, with 18,885 patients and 21,837 records, and CPSC2018, with 9,831 patients and 6,877 records [2605.13194]. PTB‑XL is converted to a single-label 5-class problem with classes NORM, MI, STTC, CD, and HYP; CPSC2018 is treated as a single-label 9-class problem with classes N, AF, I‑AVB, LBBB, RBBB, PAC, PVC, STD, and STE [2605.13194]. Each dataset is split into 80% train and 20% test, repeated five times with different random splits, and metrics are reported as averages [2605.13194].

Training uses PyTorch with custom NATTEN optimized C++/CUDA kernels, AdamW, cosine annealing, and a batch size tuned to a best-performing value of 32; the best weight decay is \(4\times 10^{-4}\), Gaussian noise in pretraining uses standard deviation 0.2, and fine-tuning converges around 25 epochs [2605.13194]. The best neighborhood kernel size is 7 [2605.13194].

In linear evaluation, ECG‑NAT reports on PTB‑XL an accuracy of \(0.800 \pm 0.032\), F1 of \(0.799 \pm 0.003\), and AUROC of \(0.960 \pm 0.001\), compared with ST‑MEM at \(0.726 \pm 0.005\) accuracy, \(0.508 \pm 0.008\) F1, and \(0.838 \pm 0.011\) AUROC [2605.13194]. On CPSC2018, ECG‑NAT reports \(0.782 \pm 0.004\) accuracy, \(0.855 \pm 0.005\) F1, and \(0.965 \pm 0.005\) AUROC, compared with ST‑MEM at \(0.723 \pm 0.008\) accuracy, \(0.641 \pm 0.010\) F1, and \(0.938 \pm 0.002\) AUROC [2605.13194].

Under full fine-tuning, PTB‑XL performance reaches \(0.902 \pm 0.002\) accuracy, \(0.904 \pm 0.009\) F1, and \(0.977 \pm 0.002\) AUROC, while CPSC2018 reaches \(0.903 \pm 0.005\) accuracy, \(0.895 \pm 0.007\) F1, and \(0.986 \pm 0.001\) AUROC [2605.13194]. In low-label experiments, ECG‑NAT reports PTB‑XL AUROC values of \(0.880 \pm 0.001\) with 1% labels, \(0.927 \pm 0.005\) with 5% labels, and \(0.977 \pm 0.002\) with 100% labels; for CPSC2018, the corresponding AUROCs are \(0.931 \pm 0.001\), \(0.961 \pm 0.002\), and \(0.986 \pm 0.001\) [2605.13194]. The abstract additionally reports 88.1% accuracy using only 1% labeled data [2605.13194].

Ablation results attribute gains to the major components. Replacing Gaussian-noise masking with zero masking lowers PTB‑XL performance from accuracy 0.902, F1 0.904, AUROC 0.977 to accuracy 0.855, F1 0.856, AUROC 0.955; on CPSC2018 the drop is from 0.903, 0.895, 0.986 to 0.845, 0.839, 0.965 [2605.13194]. Removing supervised contrastive loss lowers PTB‑XL from accuracy 0.902 and AUROC 0.977 to accuracy 0.875 and AUROC 0.965, and lowers CPSC2018 from 0.903 and 0.986 to 0.878 and 0.975 [2605.13194]. Within the same overall framework, NAT also outperforms ViT and Swin: on PTB‑XL, ECG‑ViT reports accuracy 0.775 and AUROC 0.906, ECG‑Swin 0.806 and 0.912, and ECG‑NAT 0.902 and 0.977; on CPSC2018, the values are 0.794 and 0.928, 0.834 and 0.940, and 0.903 and 0.986, respectively [2605.13194].

The reported computational profile is likewise part of the model’s definition. Compared with ViT and Swin on 12-lead 10-second ECG, NAT‑Tiny uses approximately 29.31M parameters, about 23% fewer than ViT and about 48% fewer than Swin; during fine-tuning it uses 1.10 GB memory, about 39% lower than ViT and about 52% lower than Swin; and it requires 1.57 G FLOPs, about 35% lower than ViT and about 50% lower than Swin [2605.13194].

## 6. Position within ECG transformer research

ECG‑NAT emerges from an identifiable trajectory in transformer-based ECG modeling. Earlier work on stress detection from ECG used a convolutional front-end, sinusoidal positional encoding, and a single-layer transformer encoder with four heads to classify 30-second ECG windows as stress versus non-stress [2108.09737]. That work explicitly separated local feature extraction and temporal downsampling from global self-attention, but it also noted that it did not explicitly implement neighborhood attention and suggested that an ECG Neighborhood Attention Transformer could replace global self-attention with windowed or multi-scale neighborhoods [2108.09737]. ECG‑NAT can be read as a concrete realization of that direction in the context of multi-lead arrhythmia classification [2605.13194].

Subsequent ECG transformer designs developed related locality mechanisms. "MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification" restricted self-attention to non-overlapping local windows at multiple scales and used learnable feature fusion, achieving average macro-F1 scores of 77.85%, 47.57%, 66.13%, 34.60%, and 34.29% across five PTB‑XL classification tasks [2306.12098]. "A CNN-based Local-Global Self-Attention via Averaged Window Embeddings for Hierarchical ECG Analysis" introduced local overlapping window queries and global keys and values, and reported average F1 0.885 on CODE‑TEST for six diagnoses [2504.16097]. "CardioPatternFormer: Pattern-Guided Attention for Interpretable ECG Classification with Transformer Architecture" retained global self-attention but added physiologically guided bias terms for local context, rhythm, cardiac cycle, and beat-to-beat consistency on 12-lead, 10-second ECGs [2505.20481]. "Multi-scale Masked Autoencoder for Electrocardiogram Anomaly Detection" showed that a lightweight transformer with multi-scale masking and separate global and local positional embeddings could match state-of-the-art anomaly detection while using approximately 1/78 of the inference FLOPs of a more expensive baseline [2502.05494].

This broader literature suggests that ECG‑NAT is not an isolated architecture but part of a methodological convergence around three ideas: local inductive bias, hierarchical temporal abstraction, and self-supervised pretraining. Reviews of transformer-based ECG diagnosis have repeatedly identified long sequence length, computational cost, imbalance, and interpretability as central issues for ECG transformers, while highlighting local or segment-based modeling as a natural direction for biosignals [2306.01249]. ECG‑NAT formalizes one such direction by combining multi-lead masked pretraining with neighborhood-restricted attention inside a hierarchical encoder [2605.13194].

## 7. Interpretability, limitations, and prospective directions

Interpretability is present but limited. ECG‑NAT provides t-SNE visualizations and clustering metrics showing better class separation when supervised contrastive loss is used, but it does not provide explicit attention-map visualization, detailed case studies, or systematic robustness stress tests beyond noise masking and multi-dataset pretraining [2605.13194]. The authors explicitly identify interpretability and explainability as a limitation and mention future directions including structured, low-rank, or pattern-guided attention and disentangled latent factors aligned with interpretable ECG traits [2605.13194].

External validation is also limited. Reported experiments are on public datasets, and no prospective or deployment study is included; the paper further notes the possibility of dataset biases due to single or limited centers [2605.13194]. Hyperparameter sensitivity is acknowledged as well: neighborhood size, batch size, and weight decay all affect performance, even though the reported hyperparameter analysis identifies best settings such as neighborhood kernel size 7 and batch size 32 [2605.13194].

Several plausible extensions follow from adjacent ECG transformer results. A plausible implication is that explicit multi-scale or rhythm-specific tokens could further improve classes dominated by long R–R structure, since local-global ECG attention models achieved strong overall results yet reported comparatively lower F1 for sinus bradycardia than for most other classes [2504.16097]. Another plausible implication is that physiologically guided relative biases, such as local, rhythm-aware, or cycle-aware bias terms, could be integrated into neighborhood attention without abandoning the computational advantages of \(O(Nk)\) sparsity, because such biasing has already been shown to structure global transformer attention in ECG classification [2505.20481]. The ECG‑NAT paper itself identifies future work in extending neighborhood attention to other biomedical time series such as EEG, EMG, PPG, and respiratory signals, and in exploring multimodal fusion such as ECG with PPG or electronic medical records [2605.13194].

Taken together, ECG‑NAT defines a specific point in the evolution of ECG transformers: a lightweight, self-supervised, hierarchical, multi-lead model that substitutes neighborhood attention for full global attention in order to preserve beat-level morphology, capture rhythm-level dependencies, and reduce computational cost [2605.13194]. Its reported results position it as a high-performing template for low-resource ECG classification, while its limitations leave open further work on interpretability, external clinical validation, and adaptive or physiology-aware neighborhood design [2605.13194].

Source: https://www.emergentmind.com/topics/electrocardiogram-neighborhood-attention-transformer-ecg-nat