Dual-Layer Split Attention

• The paper introduces a dual-layer split attention mechanism that fuses temporal and spatial features, yielding up to 99.6% accuracy on the MIT-BIH dataset.
• The module sequentially applies intra-modality and cross-modality attention, using group-based channel splitting and gating to refine feature integration.
• Implementation details like configurable channel groups, residual merging, and layer normalization enable significant improvements over traditional concatenation methods.

Dual-layer cross-channel split attention is a multimodal feature fusion mechanism introduced in the context of GAF-FusionNet for ECG analysis, enabling the adaptive integration of temporal (time series) and spatial (image-based representation) modalities. This mechanism employs a two-stage sequence of attention layers—first an intra-modality split-attention followed by a cross-modality split-attention—to facilitate hierarchical feature refinement and cross-modal contextualization, yielding significant improvements in classification performance (Qin et al., 2024).

1. High-Level Module Architecture

The dual-layer cross-channel split attention module operates directly after parallel feature extraction streams:

• Temporal Branch: Processes the raw ECG time series via a pipeline of 1D-CNN, Bi-LSTM, and global average pooling, resulting in $\mathbf{F}_t \in \mathbb{R}^{B \times C_t}$.
• Spatial Branch: Processes the Gramian Angular Field (GAF) image representations of ECG via 2D-CNN and global average pooling, resulting in $\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$.

The dual-layer split-attention module comprises:

• Layer 1 (Intra-modality): Performs self-attention within channel groups of each modality independently.
• Layer 2 (Cross-modality): Enables temporal features to attend to spatial ones and vice versa.

After each attention layer, outputs are merged via residual addition, layer normalization, concatenation, and final MLP-based classification.

2. Mathematical Formulation

Let batch size be $B$, with channel counts $C_t$, $C_s$ for temporal and spatial streams, split into $G_t$, $G_s$ groups, respectively. Each group contains $C_t/G_t$ or $C_s/G_s$ channels.

Channel Splitting:

For $m \in \{t, s\}$, decompose features along the channel dimension:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$0

Group Descriptor via Channel Averaging:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$1

Stacking across groups:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$2

Projection to Query, Key, Value:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$3

Layer 1: Intra-modality Attention

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$4

Layer 2: Cross-modality Attention

With $\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$5, and projections dimension-aligned:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$6

Re-projection and Residual Merge:

For each group:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$7

Final recombination:

$\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$8

3. Hierarchical Stepwise Fusion Procedure

The fusion proceeds as follows:

1. Extraction: Obtain $\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$9 and $B$0 from the respective feature branches.
2. Grouping: Split $B$1 and $B$2 into $B$3 and $B$4 channel groups.
3. Pooling: Average-pooled group descriptors $B$5, $B$6.
4. Projection: Q/K/V generation for each group per modality.
5. Layer 1: Intra-modality self-attention yields $B$7, $B$8.
6. Layer 2: Cross-modality attention integrates spatial cues into temporal and vice versa ($B$9, $C_t$0).
7. Gating: Merge $C_t$1 and $C_t$2, yielding gating coefficients $C_t$3.
8. Readout: Element-wise scaling of original group features, residual addition, layer normalization.
9. Final Classification: Concatenate $C_t$4 and $C_t$5, MLP fusion, softmax classification.

4. Distinctive Properties of the Attention Layers

• Independence: The intra-modality and cross-modality attention layers use separate Q/K/V projections; parameter sharing does not occur between layers.
• Modality Context: Intra-modality attention restricts reasoning within modality-specific channel groups, whereas cross-modality attention enforces explicit cross-modal information flow.
• Configurable Granularity: Group count ($C_t$6) and embedding dimension ($C_t$7) may be independently determined per layer, allowing tuning of granularity for both intra- and inter-modal interactions. Reported experiments use $C_t$8 and $C_t$9 for both layers.

5. Implementation Characteristics

The implementation employs the following settings:

• Batch size: $C_s$0
• Channels: $C_s$1 (split into 4 groups of 128 channels)
• Gating MLP: 1 hidden layer, size 64, with ReLU, followed by sigmoid activation
• LayerNorm: Applied after group reassembly, $C_s$2
• Post-attention head: Concatenated vector of length 1024 passed to an MLP (layer sizes 512 → 256 → number of classes) with ReLU and dropout ($C_s$3) between layers
• Convolutional layers: All kernels size 3, padding 1, with batch norm and ReLU

6. Quantitative Performance and Comparative Significance

Empirical results underscore the module’s benefit for ECG classification:

• Full dual-layer module: 99.6% accuracy (MIT-BIH dataset)
• Without cross-modality attention: 98.1%
• Without both attention layers (simple concatenation): 97.8%

Relative to the best alternative (Multi-Scale CNN), GAF-FusionNet with dual-layer split attention achieves:

• +2.0 percentage points, ECG-200 (94.5% vs 92.5%)
• +1.2 percentage points, ECG-5000 (96.9% vs 95.7%)
• +1.8 percentage points, MIT-BIH (99.6% vs 97.8%)

A plausible implication is that progressive, split-driven intra- and inter-modality attention unlocks greater discriminative power in multimodal, temporo-spatial tasks than conventional concatenation or single-stage attention strategies. These findings establish dual-layer cross-channel split attention as an effective and generalizable approach to multimodal neural feature integration in bio-signal classification contexts (Qin et al., 2024).