---
title: 'GAF-FusionNet: Multimodal ECG Analysis'
url: https://www.emergentmind.com/topics/gaf-fusionnet
type: topic
---

# GAF-FusionNet: Multimodal ECG Analysis

GAF-FusionNet is a multimodal deep learning framework for electrocardiogram (ECG) analysis that integrates both temporal and spatial information by jointly processing the raw 1D ECG signal and a Gramian Angular Field (GAF) image encoding of the same segment. The network architecture relies on a dual-branch system: one branch applies 1D convolutional and recurrent neural layers to the raw signal, while the other processes a GAF-transformed image with a 2D CNN. Feature fusion is accomplished via a dual-layer cross-channel split-attention mechanism, enabling adaptive inter- and intra-modal integration for improved classification performance across diverse ECG datasets [2501.01960].

## 1. Background and Motivation

Conventional deep learning approaches for ECG classification are typically unimodal, relying either on raw time-series representation or simple feature fusion strategies. Many do not fully exploit complementary temporal and spatial features that may be distributed across different representations of the same ECG segment. GAF-FusionNet addresses these challenges by encoding each ECG window as both a normalized time series and its GAF image. This dual representation allows the model to leverage fine-grained temporal patterns (such as the shape of QRS complexes) and global dynamic features (such as rhythm regularity), with the GAF providing a 2D structure that makes temporal dependencies more accessible to standard CNN architectures [2501.01960].

## 2. Gramian Angular Field (GAF) Transformation

GAF transformation enables the conversion of a 1D time series $X=\{x_1, x_2, \dots, x_n\}$ into a 2D image. The transformation comprises three stages:

1. **Rescaling**: Each value is normalized to $[-1,1]$:
   $$
   \tilde x_j = \frac{x_j - \min(X)}{\max(X) - \min(X)} \times 2 - 1
   $$
2. **Angular Encoding**: The normalized value is mapped to an angle:
   $$
   \phi_j = \arccos(\tilde x_j)
   $$
3. **Gramian Matrix Computation**:
   - Gramian Summation Field (used in GAF-FusionNet):
     $$
     G_S[i,j] = \cos(\phi_i + \phi_j)
     $$
   - Gramian Difference Field (alternative, not used in current implementation):
     $$
     G_D[i,j] = \sin(\phi_i - \phi_j)
     $$

The $G_S$ matrix serves as a 2D image input, preserving global and local temporal structure for the spatial-branch CNN [2501.01960].

## 3. Network Architecture and Feature Fusion

GAF-FusionNet consists of parallel temporal and spatial branches followed by dual-layer cross-channel split attention for feature fusion.

**Temporal Branch:**
- Input: raw ECG segment $S \in \mathbb{R}^{w \times 1}$.
- Processing: Several 1D convolutional layers with ReLU nonlinearity, followed by a bidirectional LSTM, and global average pooling, producing a feature vector $F_t$.

**Spatial Branch:**
- Input: GAF image $G_S \in \mathbb{R}^{w \times w}$.
- Processing: 2D CNN (ResNet-34 or custom), ending with global average pooling to produce feature vector $F_s$.

**Dual-Layer Split-Attention Fusion:**
- Layer 1: Intra-modality self-attention for $F_t$ and $F_s$:
  $$
  A_t = \mathrm{softmax}\left(\frac{Q_t K_t^T}{\sqrt{d_t}}\right)V_t,\qquad
  A_s = \mathrm{softmax}\left(\frac{Q_s K_s^T}{\sqrt{d_s}}\right)V_s
  $$
- Layer 2: Cross-modality attention:
  $$
  C_t = \mathrm{softmax}\left(\frac{Q_t K_s^T}{\sqrt{d}}\right)V_s,\qquad
  C_s = \mathrm{softmax}\left(\frac{Q_s K_t^T}{\sqrt{d}}\right)V_t
  $$
- Fused outputs:
  $$
  F_t' = \mathrm{LayerNorm}(F_t + A_t + C_t),\qquad
  F_s' = \mathrm{LayerNorm}(F_s + A_s + C_s)
  $$
- Classification: $F_{fused} = [F_t'; F_s']$ is concatenated and passed to an MLP classifier with softmax for final prediction.

This design enables nuanced modeling of both within-modality dependencies and inter-modality interactions, advancing beyond naive concatenation or averaging [2501.01960].

## 4. Training Methodology and Datasets

The network is trained end-to-end on standard ECG benchmarks (ECG200, ECG5000, MIT-BIH Arrhythmia Database) with the following protocol:

- **Preprocessing**: Butterworth bandpass filtering, z-score normalization, segmentation into windows.
- **Optimizer**: Adam with cosine-annealing learning rate scheduling, batch size of 64.
- **Spatial CNN Backbone**: ResNet-34 (pretrained on ImageNet).
- **Regularization and Augmentation**: Early stopping based on validation loss; data augmentation arises from window overlap.
- **Loss Function**: Cross-entropy, with categorical labels.

GAF-FusionNet demonstrates robust generalization across small to large datasets, consistently outperforming state-of-the-art methods in classification accuracy (ECG200: 94.5%, ECG5000: 96.9%, MIT-BIH: 99.6%) [2501.01960].

## 5. Empirical Performance and Ablation Analysis

Comparative evaluation demonstrates that GAF-FusionNet outperforms leading models such as LSTM-FCN, Informer, Attention-CNN, and Multi-Scale CNN on all standard ECG datasets. Ablation studies reveal:

- Removing dual-layer attention reduces MIT-BIH accuracy by 1.8%.
- Removing the cross-channel module reduces accuracy by 1.5%.
- Using only the time series or only the GAF branch reduces accuracy by 2.6% and 2.1%, respectively.

| Method                | ECG200 Acc. | ECG5000 Acc. | MIT-BIH Acc. |
|:----------------------|------------:|-------------:|-------------:|
| DNN (raw)             |      88.5 % |       93.2 % |      95.7 %  |
| LSTM-FCN              |      91.0 % |       94.1 % |      96.3 %  |
| Informer              |      91.5 % |       94.8 % |      97.1 %  |
| Attention-CNN         |      92.0 % |       95.3 % |      97.5 %  |
| Multi-Scale CNN       |      92.5 % |       95.7 % |      97.8 %  |
| **GAF-FusionNet**     |  **94.5 %** |   **96.9 %** |  **99.6 %**  |

Performance gains trace directly to the hybrid exploitation of GAF-based spatial information, temporal encoding, and adaptive attention-based feature fusion [2501.01960].

## 6. Limitations and Future Directions

Despite superior benchmark performance, GAF-FusionNet currently faces two primary limitations:

- **Generality to Clinical Settings**: Experiments are limited to public benchmarks. Extending the model to real-world clinical ECG datasets—characterized by broader noise/artifact distributions—remains an open challenge.
- **Computational Overhead**: The spatial branch (2D CNN on $w \times w$ GAF images) and the dual-layer attention module introduce additional computational cost, particularly in large-scale deployment.

Future research directions include validation on prospective clinical and wearable-device ECG streams, exploration of lightweight attention and pruning techniques, and interpretability tools such as Grad-CAM and attention visualizations to facilitate clinical adoption [2501.01960].

Source: https://www.emergentmind.com/topics/gaf-fusionnet