ECG-NAT: Neighborhood Attention Transformer

• The paper introduces ECG-NAT, a self-supervised transformer that combines masked autoencoder pretraining with neighborhood attention for efficient feature extraction.
• It employs a hierarchical encoder with Conv1d tokenization and NAT blocks to model both localized beat morphology and longer rhythm dependencies.
• The dual-stage training with discriminative fine-tuning and contrastive loss achieves robust classification performance on datasets like PTB‑XL and CPSC2018.

Electrocardiogram Neighborhood Attention Transformer (ECG‑NAT) denotes a self-supervised transformer architecture for multi-lead electrocardiogram arrhythmia classification that combines masked-autoencoder pretraining, discriminative fine-tuning, and neighborhood attention to model localized beat morphology and broader rhythm structure with lower computational cost than global self-attention (Gazeran et al., 13 May 2026). In the formulation reported for ECG‑NAT, the model is tailored to 12-lead, 10-second ECG segments acquired under realistic clinical constraints, including noise, inter-patient variability, long temporal extent, and limited labeled data; its central claim is that hierarchical local attention, coupled with multi-dataset pretraining and dual-loss fine-tuning, yields strong classification performance and efficiency on PTB‑XL and CPSC2018 (Gazeran et al., 13 May 2026).

1. Definition and problem setting

ECG‑NAT is designed for multi-lead ECG arrhythmia classification, specifically for 12-lead, 10-second ECG segments in realistic clinical conditions (Gazeran et al., 13 May 2026). The problem setting emphasized for the architecture includes noisy signals, substantial variability across patients and acquisition setups, scarce labeled data because annotation requires cardiologists, and long sequences that make naïve global self-attention expensive (Gazeran et al., 13 May 2026). In the reported implementation, ECG‑NAT addresses these constraints through a two-stage self-supervised pipeline: generative pretraining on unlabeled ECGs followed by discriminative fine-tuning on labeled datasets (Gazeran et al., 13 May 2026).

The input representation is a 12-lead ECG originally sampled at 500 Hz and standardized through a preprocessing pipeline consisting of a 0.5–40 Hz digital band-pass filter, amplitude normalization, and downsampling to 250 Hz, yielding 2500 time points per lead for a 10-second segment (Gazeran et al., 13 May 2026). All signals are standardized to 10 seconds; shorter recordings are zero-padded, longer recordings are segmented into non-overlapping 10-second windows, and the first segment is used for classification (Gazeran et al., 13 May 2026). The final input tensor is expressed as

$$X \in \mathbb{R}^{L \times D}, \quad L=12,\; D=2500.$$

The motivation for ECG‑NAT is explicitly architectural. CNNs are described as effective for local morphology but limited for long-range rhythm dependencies unless made very deep or wide; RNNs and LSTMs are described as slow to train and limited on very long sequences; standard transformers capture long-range dependencies but have $O(N^2)$ complexity and require large labeled datasets to avoid overfitting (Gazeran et al., 13 May 2026). ECG‑NAT therefore targets simultaneous modeling of beat-level morphology and rhythm-level dependencies while maintaining computational efficiency through Neighborhood Attention Transformer blocks (Gazeran et al., 13 May 2026).

2. Architectural organization

The encoder begins with a tokenizer implemented by two Conv1d layers with kernel size 3 and stride 2, reducing the temporal dimension by a factor of four from 2500 to 625 (Gazeran et al., 13 May 2026). Leads are treated as channels, and the convolutions produce a feature map $U \in \mathbb{R}^{L' \times D'}$, where $D' = 625$ and the channel dimension grows through the hierarchy (Gazeran et al., 13 May 2026). The use of overlapping convolutional windows preserves continuity of morphology across token boundaries (Gazeran et al., 13 May 2026).

Above the tokenizer, ECG‑NAT uses a hierarchical encoder composed of four NAT blocks interleaved with convolutional down-samplers (Gazeran et al., 13 May 2026). The NAT blocks use embedding dimension 96, multi-head attention with heads $(2, 4, 8, 16)$ across stages, and MLP ratio 4.0 (Gazeran et al., 13 May 2026). Between these blocks, down-sampler modules consisting of Conv1d layers with kernel size 2, stride 2, and ReLU reduce temporal resolution while increasing channel dimension according to

$[L', D'] \rightarrow [2L', D'/2].$

This yields the temporal pyramid 625 $\rightarrow$ 312 $\rightarrow$ 156 $\rightarrow$ 78, with channels growing for example as 96 $\rightarrow$ 192 $O(N^2)$0 384 $O(N^2)$1 768 (Gazeran et al., 13 May 2026).

The final encoder output is

$O(N^2)$2

with $O(N^2)$3 large, for example 768, and $O(N^2)$4 (Gazeran et al., 13 May 2026). For classification, this representation is flattened and passed to a fully connected layer (Gazeran et al., 13 May 2026). This encoder design is explicitly hierarchical: early stages operate at high temporal resolution and are intended to emphasize detailed morphology such as P waves, QRS complexes, T waves, and ST segments, whereas later stages operate at lower resolution with larger effective receptive fields and are intended to emphasize broader rhythm patterns such as atrial fibrillation or conduction blocks (Gazeran et al., 13 May 2026).

A notable design decision is that multi-lead information is fused implicitly rather than by explicit cross-lead attention. The 12 leads are treated as channels of a multi-channel 1D signal, and both the Conv1d layers and NAT blocks operate along the temporal dimension while the channel dimension aggregates information from different leads (Gazeran et al., 13 May 2026). The model therefore encodes all leads jointly as a multi-channel signal rather than as separate lead sequences (Gazeran et al., 13 May 2026).

3. Neighborhood attention and multi-scale temporal modeling

The defining mechanism of ECG‑NAT is neighborhood attention. In standard self-attention, each token attends to all $O(N^2)$5 tokens,

$O(N^2)$6

with $O(N^2)$7 complexity (Gazeran et al., 13 May 2026). ECG‑NAT instead restricts each token to its $O(N^2)$8 nearest temporal neighbors (Gazeran et al., 13 May 2026). For token $O(N^2)$9, the attention logits over its neighborhood $U \in \mathbb{R}^{L' \times D'}$0 are defined as

$U \in \mathbb{R}^{L' \times D'}$1

where $U \in \mathbb{R}^{L' \times D'}$2 is a relative positional bias encoding the distance between $U \in \mathbb{R}^{L' \times D'}$3 and its $U \in \mathbb{R}^{L' \times D'}$4-th neighbor (Gazeran et al., 13 May 2026). The corresponding values are collected as

$U \in \mathbb{R}^{L' \times D'}$5

and neighborhood attention becomes

$U \in \mathbb{R}^{L' \times D'}$6

The resulting complexity is $U \in \mathbb{R}^{L' \times D'}$7 rather than $U \in \mathbb{R}^{L' \times D'}$8 (Gazeran et al., 13 May 2026).

The architectural rationale is explicitly multi-scale. Initial Conv1d tokenization reduces 2500 time points to 625 tokens; subsequent down-samplers reduce this to 312, 156, and 78 (Gazeran et al., 13 May 2026). Early NAT layers therefore operate on finer temporal structure, while deeper NAT layers operate on coarser representations whose receptive fields span more cardiac cycles (Gazeran et al., 13 May 2026). This is described as a hierarchical temporal pyramid in which early layers capture detailed beat morphology and later layers capture longer-term rhythm (Gazeran et al., 13 May 2026).

Hyperparameter analysis reported in the paper identifies a neighborhood kernel size $U \in \mathbb{R}^{L' \times D'}$9 as the best-performing setting (Gazeran et al., 13 May 2026). Smaller kernels are described as capturing local morphology well, while overly large kernels increase cost and reduce the locality advantage (Gazeran et al., 13 May 2026). The design is therefore neither purely convolutional nor globally transformer-like: convolutions impose local inductive bias, and neighborhood attention provides content-dependent weighting within local temporal contexts, with receptive fields expanding hierarchically across layers (Gazeran et al., 13 May 2026).

This places ECG‑NAT within a broader line of ECG architectures that prioritize local or hybrid local-global attention. Earlier transformer-based ECG stress detection used a convolutional front-end followed by a single global self-attention encoder on downsampled tokens, explicitly separating local feature extraction and global temporal integration, although it did not implement neighborhood attention (Behinaein et al., 2021). Shifted-window ECG transformers similarly restricted self-attention to local windows at multiple scales for 12-lead classification (Cheng et al., 2023), while local-global attention models used local overlapping windows to construct queries and global keys and values to preserve rhythm-level context (Buzelin et al., 13 Apr 2025). ECG‑NAT differs by making neighborhood attention the central temporal operator inside a self-supervised multi-lead architecture (Gazeran et al., 13 May 2026).

4. Self-supervised pretraining and discriminative fine-tuning

The first stage of ECG‑NAT is generative pretraining on unlabeled ECGs from two datasets, Chapman and Ningbo (Gazeran et al., 13 May 2026). After tokenization, the model randomly selects a subset of tokens and masks them using Gaussian noise with standard deviation 0.2 (Gazeran et al., 13 May 2026). The encoder processes the corrupted sequence, and a decoder reconstructs the original clean ECG signal (Gazeran et al., 13 May 2026). The decoder is a lightweight ConvTranspose1d stack: one ConvTranspose1d layer with kernel 3 and stride 2 to increase temporal dimension from 78 to 156, followed by four ConvTranspose1d layers with kernels $D' = 625$0 and strides $D' = 625$1, without padding, to recover a 12-lead waveform of length 2500 (Gazeran et al., 13 May 2026).

The reconstruction objective is mean squared error over masked indices: $D' = 625$2 Pretraining on multiple heterogeneous datasets is described as promoting domain-invariant representations that are robust to dataset-specific artifacts and not tied to a single distribution (Gazeran et al., 13 May 2026).

The second stage is discriminative fine-tuning on labeled datasets (Gazeran et al., 13 May 2026). Two modes are reported. In linear evaluation, the pretrained encoder is frozen and only a linear classifier is trained (Gazeran et al., 13 May 2026). In full fine-tuning, the encoder and classifier are optimized end-to-end using a dual loss that combines supervised contrastive loss and cross-entropy (Gazeran et al., 13 May 2026). With batch index set $D' = 625$3, embeddings $D' = 625$4, class labels $D' = 625$5, positives $D' = 625$6, and cosine similarity

$D' = 625$7

the supervised contrastive term is

$D' = 625$8

The classification term is

$D' = 625$9

and the combined loss is

$(2, 4, 8, 16)$0

The intended effect is to optimize label prediction while shaping a discriminative embedding space with tighter intra-class clusters and stronger inter-class separation (Gazeran et al., 13 May 2026).

The supervised contrastive component is supported empirically by representation analysis. t-SNE visualizations show improved cluster separation when the contrastive term is used, and for CPSC2018 the silhouette score improved by 13.7% while the Davies–Bouldin index was reduced by 15.4% (Gazeran et al., 13 May 2026).

5. Empirical evaluation and computational profile

Pretraining uses unlabeled ECGs from Chapman, with 10,646 patients and 10,646 records, and Ningbo, with 45,152 patients and 34,905 records (Gazeran et al., 13 May 2026). Fine-tuning and evaluation use PTB‑XL, with 18,885 patients and 21,837 records, and CPSC2018, with 9,831 patients and 6,877 records (Gazeran et al., 13 May 2026). PTB‑XL is converted to a single-label 5-class problem with classes NORM, MI, STTC, CD, and HYP; CPSC2018 is treated as a single-label 9-class problem with classes N, AF, I‑AVB, LBBB, RBBB, PAC, PVC, STD, and STE (Gazeran et al., 13 May 2026). Each dataset is split into 80% train and 20% test, repeated five times with different random splits, and metrics are reported as averages (Gazeran et al., 13 May 2026).

Training uses PyTorch with custom NATTEN optimized C++/CUDA kernels, AdamW, cosine annealing, and a batch size tuned to a best-performing value of 32; the best weight decay is $(2, 4, 8, 16)$1, Gaussian noise in pretraining uses standard deviation 0.2, and fine-tuning converges around 25 epochs (Gazeran et al., 13 May 2026). The best neighborhood kernel size is 7 (Gazeran et al., 13 May 2026).

In linear evaluation, ECG‑NAT reports on PTB‑XL an accuracy of $(2, 4, 8, 16)$2, F1 of $(2, 4, 8, 16)$3, and AUROC of $(2, 4, 8, 16)$4, compared with ST‑MEM at $(2, 4, 8, 16)$5 accuracy, $(2, 4, 8, 16)$6 F1, and $(2, 4, 8, 16)$7 AUROC (Gazeran et al., 13 May 2026). On CPSC2018, ECG‑NAT reports $(2, 4, 8, 16)$8 accuracy, $(2, 4, 8, 16)$9 F1, and $[L', D'] \rightarrow [2L', D'/2].$0 AUROC, compared with ST‑MEM at $[L', D'] \rightarrow [2L', D'/2].$1 accuracy, $[L', D'] \rightarrow [2L', D'/2].$2 F1, and $[L', D'] \rightarrow [2L', D'/2].$3 AUROC (Gazeran et al., 13 May 2026).

Under full fine-tuning, PTB‑XL performance reaches $[L', D'] \rightarrow [2L', D'/2].$4 accuracy, $[L', D'] \rightarrow [2L', D'/2].$5 F1, and $[L', D'] \rightarrow [2L', D'/2].$6 AUROC, while CPSC2018 reaches $[L', D'] \rightarrow [2L', D'/2].$7 accuracy, $[L', D'] \rightarrow [2L', D'/2].$8 F1, and $[L', D'] \rightarrow [2L', D'/2].$9 AUROC (Gazeran et al., 13 May 2026). In low-label experiments, ECG‑NAT reports PTB‑XL AUROC values of $\rightarrow$0 with 1% labels, $\rightarrow$1 with 5% labels, and $\rightarrow$2 with 100% labels; for CPSC2018, the corresponding AUROCs are $\rightarrow$3, $\rightarrow$4, and $\rightarrow$5 (Gazeran et al., 13 May 2026). The abstract additionally reports 88.1% accuracy using only 1% labeled data (Gazeran et al., 13 May 2026).

Ablation results attribute gains to the major components. Replacing Gaussian-noise masking with zero masking lowers PTB‑XL performance from accuracy 0.902, F1 0.904, AUROC 0.977 to accuracy 0.855, F1 0.856, AUROC 0.955; on CPSC2018 the drop is from 0.903, 0.895, 0.986 to 0.845, 0.839, 0.965 (Gazeran et al., 13 May 2026). Removing supervised contrastive loss lowers PTB‑XL from accuracy 0.902 and AUROC 0.977 to accuracy 0.875 and AUROC 0.965, and lowers CPSC2018 from 0.903 and 0.986 to 0.878 and 0.975 (Gazeran et al., 13 May 2026). Within the same overall framework, NAT also outperforms ViT and Swin: on PTB‑XL, ECG‑ViT reports accuracy 0.775 and AUROC 0.906, ECG‑Swin 0.806 and 0.912, and ECG‑NAT 0.902 and 0.977; on CPSC2018, the values are 0.794 and 0.928, 0.834 and 0.940, and 0.903 and 0.986, respectively (Gazeran et al., 13 May 2026).

The reported computational profile is likewise part of the model’s definition. Compared with ViT and Swin on 12-lead 10-second ECG, NAT‑Tiny uses approximately 29.31M parameters, about 23% fewer than ViT and about 48% fewer than Swin; during fine-tuning it uses 1.10 GB memory, about 39% lower than ViT and about 52% lower than Swin; and it requires 1.57 G FLOPs, about 35% lower than ViT and about 50% lower than Swin (Gazeran et al., 13 May 2026).

6. Position within ECG transformer research

ECG‑NAT emerges from an identifiable trajectory in transformer-based ECG modeling. Earlier work on stress detection from ECG used a convolutional front-end, sinusoidal positional encoding, and a single-layer transformer encoder with four heads to classify 30-second ECG windows as stress versus non-stress (Behinaein et al., 2021). That work explicitly separated local feature extraction and temporal downsampling from global self-attention, but it also noted that it did not explicitly implement neighborhood attention and suggested that an ECG Neighborhood Attention Transformer could replace global self-attention with windowed or multi-scale neighborhoods (Behinaein et al., 2021). ECG‑NAT can be read as a concrete realization of that direction in the context of multi-lead arrhythmia classification (Gazeran et al., 13 May 2026).

Subsequent ECG transformer designs developed related locality mechanisms. "MSW-Transformer: Multi-Scale Shifted Windows Transformer Networks for 12-Lead ECG Classification" restricted self-attention to non-overlapping local windows at multiple scales and used learnable feature fusion, achieving average macro-F1 scores of 77.85%, 47.57%, 66.13%, 34.60%, and 34.29% across five PTB‑XL classification tasks (Cheng et al., 2023). "A CNN-based Local-Global Self-Attention via Averaged Window Embeddings for Hierarchical ECG Analysis" introduced local overlapping window queries and global keys and values, and reported average F1 0.885 on CODE‑TEST for six diagnoses (Buzelin et al., 13 Apr 2025). "CardioPatternFormer: Pattern-Guided Attention for Interpretable ECG Classification with Transformer Architecture" retained global self-attention but added physiologically guided bias terms for local context, rhythm, cardiac cycle, and beat-to-beat consistency on 12-lead, 10-second ECGs (Uğraş et al., 26 May 2025). "Multi-scale Masked Autoencoder for Electrocardiogram Anomaly Detection" showed that a lightweight transformer with multi-scale masking and separate global and local positional embeddings could match state-of-the-art anomaly detection while using approximately 1/78 of the inference FLOPs of a more expensive baseline (Zhou et al., 8 Feb 2025).

This broader literature suggests that ECG‑NAT is not an isolated architecture but part of a methodological convergence around three ideas: local inductive bias, hierarchical temporal abstraction, and self-supervised pretraining. Reviews of transformer-based ECG diagnosis have repeatedly identified long sequence length, computational cost, imbalance, and interpretability as central issues for ECG transformers, while highlighting local or segment-based modeling as a natural direction for biosignals (Zhao, 2023). ECG‑NAT formalizes one such direction by combining multi-lead masked pretraining with neighborhood-restricted attention inside a hierarchical encoder (Gazeran et al., 13 May 2026).

7. Interpretability, limitations, and prospective directions

Interpretability is present but limited. ECG‑NAT provides t-SNE visualizations and clustering metrics showing better class separation when supervised contrastive loss is used, but it does not provide explicit attention-map visualization, detailed case studies, or systematic robustness stress tests beyond noise masking and multi-dataset pretraining (Gazeran et al., 13 May 2026). The authors explicitly identify interpretability and explainability as a limitation and mention future directions including structured, low-rank, or pattern-guided attention and disentangled latent factors aligned with interpretable ECG traits (Gazeran et al., 13 May 2026).

External validation is also limited. Reported experiments are on public datasets, and no prospective or deployment study is included; the paper further notes the possibility of dataset biases due to single or limited centers (Gazeran et al., 13 May 2026). Hyperparameter sensitivity is acknowledged as well: neighborhood size, batch size, and weight decay all affect performance, even though the reported hyperparameter analysis identifies best settings such as neighborhood kernel size 7 and batch size 32 (Gazeran et al., 13 May 2026).

Several plausible extensions follow from adjacent ECG transformer results. A plausible implication is that explicit multi-scale or rhythm-specific tokens could further improve classes dominated by long R–R structure, since local-global ECG attention models achieved strong overall results yet reported comparatively lower F1 for sinus bradycardia than for most other classes (Buzelin et al., 13 Apr 2025). Another plausible implication is that physiologically guided relative biases, such as local, rhythm-aware, or cycle-aware bias terms, could be integrated into neighborhood attention without abandoning the computational advantages of $\rightarrow$6 sparsity, because such biasing has already been shown to structure global transformer attention in ECG classification (Uğraş et al., 26 May 2025). The ECG‑NAT paper itself identifies future work in extending neighborhood attention to other biomedical time series such as EEG, EMG, PPG, and respiratory signals, and in exploring multimodal fusion such as ECG with PPG or electronic medical records (Gazeran et al., 13 May 2026).

Taken together, ECG‑NAT defines a specific point in the evolution of ECG transformers: a lightweight, self-supervised, hierarchical, multi-lead model that substitutes neighborhood attention for full global attention in order to preserve beat-level morphology, capture rhythm-level dependencies, and reduce computational cost (Gazeran et al., 13 May 2026). Its reported results position it as a high-performing template for low-resource ECG classification, while its limitations leave open further work on interpretability, external clinical validation, and adaptive or physiology-aware neighborhood design (Gazeran et al., 13 May 2026).