---
title: Dual-path Self-Attention RNN
url: https://www.emergentmind.com/topics/dual-path-self-attention-rnn-dp-sarnn
type: topic
---

# Dual-path Self-Attention RNN

A Dual-path Self-Attention Recurrent Neural Network (DP-SARNN) is a neural architecture that introduces self-attention mechanisms into the dual-path recurrent neural network framework for efficient modeling of both local and long-range dependencies in sequential data, with particular utility in real-time time-domain speech enhancement and audio-visual speech extraction. The architecture interleaves recurrent and attention-based modules, operating along intra-chunk (local) and inter-chunk (global) axes, achieving state-of-the-art performance with reduced computational overhead and algorithmic latency. DP-SARNN represents key advances over both pure RNN-based and transformer-style dual-path architectures for sequence processing in audio and multimodal fusion frameworks [2010.12713], [2207.04213].

## 1. Principle of Dual-Path Chunked Processing

DP-SARNN operates on chunked input representations, segmenting the time sequence into overlapping chunks to facilitate separate modeling of short-range and long-range temporal dependencies. Let the input $X \in \mathbb{R}^{T \times L}$ (with $T$ frames and $L$-dimensional features) be divided into $J$ segments (chunks) of length $K$ with stride $P$, producing
$$
\mathbf{X} \in \mathbb{R}^{J \times K \times L}, \quad \mathbf{X}_{j,k} = X_{(j-1)P + k, :}
$$
Within each DP-SARNN block, operations are applied in two stages:
1. **Intra-chunk:** Each chunk (of $K$ frames) is processed independently along the time axis to model fine-grained, local context.
2. **Inter-chunk:** At each frame position $k$, features across all chunks are processed to learn global or long-range dependencies.

This decomposition yields a hierarchical processing flow, critical for both memory and computational efficiency in long sequential data [2010.12713].

## 2. Self-Attention RNN (SARNN) Module

A central innovation in DP-SARNN is replacing conventional RNNs in both intra- and inter-chunk modules with Self-Attention RNN (SARNN) blocks. Each SARNN block employs the following computation sequence:
- LayerNorm
- (B)LSTM (or LSTM depending on causality)
- Linear projection
- LayerNorm
- Single-headed efficient gated scaled dot-product attention
- Add & Norm
- Two-layer MLP with GELU activation (channel size $4N$)
- Add & Norm

The self-attention mechanism within each SARNN block is defined as:
```latex
\begin{aligned}
& X_{\text{LN}} = \mathrm{LayerNorm}(X) \\
& h_t = \mathrm{RNN}(X_{\text{LN}}) \\
& Y = \mathrm{Linear}(h) \\
& Q = \mathrm{LayerNorm}(Y),\;\; K = \mathrm{LayerNorm}(Y),\;\; V = \mathrm{LayerNorm}(Y) \\
& K_r = K \odot \sigma(K'),\;\; Q_r = (\mathrm{Linear}(Q)) \odot \sigma(Q'), \\
& V_r = V \odot (\sigma(\mathrm{Linear}(V')) \odot \tanh(\mathrm{Linear}(V'))) \\
& A = \mathrm{Softmax}\!\left(\frac{Q_r K_r^\top}{\sqrt{N}}\right)V_r \\
& \text{Output} = \mathrm{FFN}(Q + A) + (Q + A)
\end{aligned}
```
where $Q', K', V'$ are trainable gating vectors (broadcast across $T$).

For causal, real-time settings, attention weights are masked to ensure only current and past frames are included:
```latex
W_{ij} = \begin{cases} W_{ij} & i \leq j \\ 0 & i > j \end{cases}
```
This ensures strict online processing for inter-chunk SARNN [2010.12713].

## 3. End-to-End Pipeline and Data Flow

The stacked DP-SARNN blocks process the chunked representation as follows:
1. Apply intra-chunk SARNN to each chunk ($\mathbb{R}^{K\times N}$), obtaining $\mathbf{X}^1\in\mathbb{R}^{J\times K\times N}$.
2. Transpose to $\mathbf{X}^2\in\mathbb{R}^{K\times J\times N}$.
3. Apply inter-chunk SARNN for each position, obtaining $\mathbf{X}^3$.
4. Transpose back to $\mathbf{X}^4$.
After stacking several such blocks and a linear output layer, overlap-add (OLA) reconstruction is performed at both frame and chunk level to recover the enhanced time-domain waveform.

## 4. Architectural Hyperparameters and Implementation

Key configuration for real-time speech enhancement includes:
- Input projection: $N=128$ channels
- Six DP-SARNN blocks with (B)LSTM layers of hidden size $H=256$
- Single-head attention, FFN size $4N=512$, dropout rate $5\%$
- Frame size $L=16$ samples (1 ms) or $L=32$ ms (real-time), frame shift $R=8$ or $16$ ms
- Chunk size $K=63$ frames ($\approx$32 ms), shift $P=31$ (half-overlap)
- End-to-end latency $\approx$32 ms; per 32 ms chunk CPU time: 7.9 ms
- Adam optimizer, initial learning rate $2\times10^{-4}$, PCM loss, gradient norm clip at 3, mixed-precision training [2010.12713]

## 5. Audio-Visual Extension and Dual-Path Cross-Modal Attention

In the audio-visual speech extraction context, DP-SARNN is extended to support multimodal fusion [2207.04213]:
- Audio features from time-domain convolutional encoder are chunked as before.
- Visual features are extracted per video frame (MTCNN + FaceNet), dimension $D_v=512$, with natural alignment to audio chunks ($S \approx T_v$), so up/downsampling is unnecessary.
- "Dual-Path Attention" blocks stack $N$ times, with each block producing updated audio ($C_{a_i}$) and video ($E_{v_i}$) representations through intra-chunk, inter-chunk, and cross-modal attention.

The inter-chunk block conducts:
- Audio self-attention across chunks for each position.
- Video self-attention across visual frames.
- Cross-modal attention: audio features are pooled across chunk, then attended to by video and vice versa; fused outputs are added back to respective streams and linearly projected.
- All Q/K/V projections are linear layers; no explicit positional encoding.

Key hyperparameters for AV extraction:
- 3 dual-path blocks; 4 intra- and inter-chunk layers per block
- Audio chunk size $K=160$ (16 ms window, 8 ms hop)
- Audio $D_a=256$, video $D_v=512$, MHA with $h=8$, $D_k=64$, FFN $D_f=1024$
- No dropout; regularization via residuals, LayerNorm, local attention masking [2207.04213]

## 6. Performance, Efficiency, and Empirical Results

DP-SARNN demonstrates the following empirical results:
- On noisy WSJ0 (babble, cafeteria, SNR $-5$ to $5$ dB): STOI $\approx94.7\%$, PESQ $\approx3.26$ (non-causal), outperforming DP-RNN by $\sim0.8\%$ STOI and $0.08$ PESQ.
- Causal DP-SARNN (DP-SALSTM): STOI $93.1\%$, PESQ $3.02$, CPU runtime per 32 ms chunk 7.9 ms, model size 6.49M params
- Ablations: attention enables 4× larger frame shift (reducing compute) with no loss; removing attention degrades PESQ by $\sim0.1-0.2$; larger chunk shift provides balance between context and latency.
- In audio-visual extraction, DP-SARNN-based cross-modal fusion yields $7-7.5$ dB SI-SNR improvement over ConvTasNet and AV-ConvTasNet, especially in multi-interferer mixtures [2010.12713], [2207.04213].

## 7. Context, Extensions, and Related Work

DP-SARNN generalizes prior sequential dual-path approaches (DPRNN) by augmenting both intra- and inter-chunk recurrence with attention-based modeling, or fully replacing RNNs with transformer blocks in some variants, as in audio-visual fusion work. In multichannel enhancement, a third spatial path can be added as in TPARN, where each channel is processed independently by a dual-path (self-attentive) recurrent network, then aggregated via an additional spatial path [2110.10757]. A plausible implication is that DP-SARNN and its extensions provide a versatile backbone for various monaural, multichannel, and multimodal sequence enhancement and separation tasks. The dual-path chunked structure is also widely adopted in high-performance, low-latency, and real-time sequence models across speech and broader sequential domains.

Source: https://www.emergentmind.com/topics/dual-path-self-attention-rnn-dp-sarnn