---
title: Dual-Layer Split Attention
url: https://www.emergentmind.com/topics/dual-layer-cross-channel-split-attention
type: topic
---

# Dual-Layer Split Attention

Dual-layer cross-channel split attention is a multimodal feature fusion mechanism introduced in the context of GAF-FusionNet for ECG analysis, enabling the adaptive integration of temporal (time series) and spatial (image-based representation) modalities. This mechanism employs a two-stage sequence of attention layers—first an intra-modality split-attention followed by a cross-modality split-attention—to facilitate hierarchical feature refinement and cross-modal contextualization, yielding significant improvements in classification performance [2501.01960].

## 1. High-Level Module Architecture

The dual-layer cross-channel split attention module operates directly after parallel feature extraction streams:

- **Temporal Branch:** Processes the raw ECG time series via a pipeline of 1D-CNN, Bi-LSTM, and global average pooling, resulting in $\mathbf{F}_t \in \mathbb{R}^{B \times C_t}$.
- **Spatial Branch:** Processes the Gramian Angular Field (GAF) image representations of ECG via 2D-CNN and global average pooling, resulting in $\mathbf{F}_s \in \mathbb{R}^{B \times C_s}$.

The dual-layer split-attention module comprises:
- **Layer 1 (Intra-modality):** Performs self-attention within channel groups of each modality independently.
- **Layer 2 (Cross-modality):** Enables temporal features to attend to spatial ones and vice versa.

After each attention layer, outputs are merged via residual addition, layer normalization, concatenation, and final MLP-based classification.

## 2. Mathematical Formulation

Let batch size be $B$, with channel counts $C_t$, $C_s$ for temporal and spatial streams, split into $G_t$, $G_s$ groups, respectively. Each group contains $C_t/G_t$ or $C_s/G_s$ channels.

**Channel Splitting:**
For $m \in \{t, s\}$, decompose features along the channel dimension:
$$
\mathbf{F}_m = [\mathbf{F}_{m,1} \Vert \cdots \Vert \mathbf{F}_{m,G_m}],\quad \mathbf{F}_{m,g} \in \mathbb{R}^{B \times (C_m/G_m)}
$$

**Group Descriptor via Channel Averaging:**
$$
z_{m,g} = \frac{1}{C_m/G_m} \cdot \mathbf{F}_{m,g} \cdot \mathbf{1} \in \mathbb{R}^{B \times 1}
$$
Stacking across groups:
$$
\mathbf{Z}_m = [z_{m,1}, \ldots, z_{m,G_m}] \in \mathbb{R}^{B \times G_m}
$$

**Projection to Query, Key, Value:**
$$
\mathbf{Q}_m = \mathbf{Z}_m W_m^Q \\
\mathbf{K}_m = \mathbf{Z}_m W_m^K \\
\mathbf{V}_m = \mathbf{Z}_m W_m^V \\
W_m^Q, W_m^K, W_m^V \in \mathbb{R}^{G_m \times d}
$$

**Layer 1: Intra-modality Attention**
$$
\mathbf{A}_m = \mathrm{softmax}\left(\frac{\mathbf{Q}_m \mathbf{K}_m^T}{\sqrt{d}}\right) \mathbf{V}_m \in \mathbb{R}^{B \times G_m \times d}
$$

**Layer 2: Cross-modality Attention**
With $d_\text{cross} = \min(d_t, d_s)$, and projections dimension-aligned:
$$
\mathbf{C}_t = \mathrm{softmax}\left(\frac{\mathbf{Q}_t \mathbf{K}_s^T}{\sqrt{d_\text{cross}}}\right) \mathbf{V}_s \\
\mathbf{C}_s = \mathrm{softmax}\left(\frac{\mathbf{Q}_s \mathbf{K}_t^T}{\sqrt{d_\text{cross}}}\right) \mathbf{V}_t
$$

**Re-projection and Residual Merge:**
For each group:
$$
w_{m,g} = \sigma\left(\mathrm{MLP}\left([\mathbf{A}_m[...,g,:} \Vert \mathbf{C}_m[...,g,:}]\right)\right) \\
\mathbf{F}_{m,g}' = \mathbf{F}_{m,g} + \mathbf{F}_{m,g} \odot w_{m,g}
$$
Final recombination:
$$
\mathbf{F}_m' = \mathrm{LayerNorm}([\mathbf{F}_{m,1}' \Vert \cdots \Vert \mathbf{F}_{m,G_m}'])
$$

## 3. Hierarchical Stepwise Fusion Procedure

The fusion proceeds as follows:
1. **Extraction:** Obtain $\mathbf{F}_t$ and $\mathbf{F}_s$ from the respective feature branches.
2. **Grouping:** Split $\mathbf{F}_t$ and $\mathbf{F}_s$ into $G_t$ and $G_s$ channel groups.
3. **Pooling:** Average-pooled group descriptors $\mathbf{Z}_t$, $\mathbf{Z}_s$.
4. **Projection:** Q/K/V generation for each group per modality.
5. **Layer 1:** Intra-modality self-attention yields $\mathbf{A}_t$, $\mathbf{A}_s$.
6. **Layer 2:** Cross-modality attention integrates spatial cues into temporal and vice versa ($\mathbf{C}_t$, $\mathbf{C}_s$).
7. **Gating:** Merge $\mathbf{A}_m$ and $\mathbf{C}_m$, yielding gating coefficients $w_{m,g}$.
8. **Readout:** Element-wise scaling of original group features, residual addition, layer normalization.
9. **Final Classification:** Concatenate $\mathbf{F}_t'$ and $\mathbf{F}_s'$, MLP fusion, softmax classification.

## 4. Distinctive Properties of the Attention Layers

- **Independence:** The intra-modality and cross-modality attention layers use separate Q/K/V projections; parameter sharing does not occur between layers.
- **Modality Context:** Intra-modality attention restricts reasoning within modality-specific channel groups, whereas cross-modality attention enforces explicit cross-modal information flow.
- **Configurable Granularity:** Group count ($G_m$) and embedding dimension ($d$) may be independently determined per layer, allowing tuning of granularity for both intra- and inter-modal interactions. Reported experiments use $G_t = G_s = 4$ and $d = 32$ for both layers.

## 5. Implementation Characteristics

The implementation employs the following settings:
- **Batch size:** $B=64$
- **Channels:** $C_t=C_s=512$ (split into 4 groups of 128 channels)
- **Gating MLP:** 1 hidden layer, size 64, with ReLU, followed by sigmoid activation
- **LayerNorm:** Applied after group reassembly, $\epsilon=10^{-5}$
- **Post-attention head:** Concatenated vector of length 1024 passed to an MLP (layer sizes 512 → 256 → number of classes) with ReLU and dropout ($p=0.5$) between layers
- **Convolutional layers:** All kernels size 3, padding 1, with batch norm and ReLU

## 6. Quantitative Performance and Comparative Significance

Empirical results underscore the module’s benefit for ECG classification:
- **Full dual-layer module:** 99.6% accuracy (MIT-BIH dataset)
- **Without cross-modality attention:** 98.1%
- **Without both attention layers (simple concatenation):** 97.8%

Relative to the best alternative (Multi-Scale CNN), GAF-FusionNet with dual-layer split attention achieves:
- +2.0 percentage points, ECG-200 (94.5% vs 92.5%)
- +1.2 percentage points, ECG-5000 (96.9% vs 95.7%)
- +1.8 percentage points, MIT-BIH (99.6% vs 97.8%)

A plausible implication is that progressive, split-driven intra- and inter-modality attention unlocks greater discriminative power in multimodal, temporo-spatial tasks than conventional concatenation or single-stage attention strategies. These findings establish dual-layer cross-channel split attention as an effective and generalizable approach to multimodal neural feature integration in bio-signal classification contexts [2501.01960].

Source: https://www.emergentmind.com/topics/dual-layer-cross-channel-split-attention