Conv-GRU CNP Module for In-Car Speech Separation

• Conv-GRU CNP is a lightweight dual-stage module that alternates crossband convolution and narrowband GRU processing to model spatial, frequency, and temporal features.
• It leverages parallel Frequency Conv1D and Group Conv1D layers for efficient spatial-frequency extraction, drastically reducing computational load.
• Empirical results show that integrating this module yields low MACs (0.56G) and RTF (0.37) while maintaining high separation accuracy in multi-zone environments.

The Conv-GRU Crossband-Narrowband Processing (CNP) module is an extremely lightweight architectural component developed for efficient spatial, frequency, and temporal information modeling within real-time in-car multi-zone speech separation systems. Originally introduced as part of the LSZone architecture, the CNP module leverages a dual-stage structure composed of parallel convolutional operations for cross-frequency spatial extraction and recurrent GRU-based processing to capture narrowband temporal dynamics. Its design allows efficient feature interaction and compression, making it well-suited for environments with stringent real-time computational constraints.

1. Architectural Design and Operational Principle

The Conv-GRU CNP module alternately applies two specialized processing stages. The initial stage (Conv Crossband) utilizes parallelized convolutional layers—specifically Frequency Conv1D and Group Conv1D—to jointly extract spatial and frequency-band relationships from Mel spectrogram features. The convolutional operations function across frequency bands ("crossband"), thereby capturing salient spatial cues embedded within multi-channel audio representations.

Following this, the second stage (GRU Narrowband) processes each frequency band as a distinct narrowband stream. A lightweight recurrent structure based on the Gated Recurrent Unit (GRU) is applied to each frequency bin, enabling the learning of temporal evolution while maintaining spatial specificity on a per-band basis. This alternating crossband–narrowband sequence is explicitly designed to interactively model the multidimensional dependencies between space, frequency, and time, providing favorable computational efficiency for real-time pipelines.

2. Core Algorithms and Layer Specifications

Efficiency within the CNP module is achieved through computation-aware architectural choices. The Conv Crossband employs parallelized Frequency Conv1D and Group Conv1D operations that operate along spatial dimensions ($H$) and frequency indices ($N_{\text{mel}}$), minimizing the parameter count and computational load compared to fully connected or attention-based solutions.

The GRU Narrowband stage comprises a normalization layer, the GRU cell itself, and a final linear output layer. The GRU cell updates its hidden state using the standard update equations: $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$

$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$

$$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

where $x_t$ is the temporal input, $h_t$ is the hidden state, $\sigma$ is the sigmoid nonlinearity, and $W$, $U$, $b$ denote the weight matrices and biases, while $\odot$ represents element-wise multiplication.

The overall module structure ensures each convolutional output is fed into the corresponding narrowband GRU stream, enabling both cross-frequency interaction and time evolution to be modeled jointly yet efficiently.

3. Integration within LSZone Architecture

Within the LSZone system, the Conv-GRU CNP module serves as the backbone for spatial information modeling after primary feature compression and fusion. The pre-processing pipeline involves the SpaIEC module, which fuses Mel spectrogram and Interaural Phase Difference (IPD) features. The resulting compressed spatial representation feeds into a stack beginning with a Conv1D layer, followed by multiple Conv-GRU CNP modules. Outputs from the CNP modules are then linearly transformed to the predicted Mel spectrogram; phase reconstruction and inverse STFT (iSTFT) processes are employed to obtain the final time-domain separated signals.

The use of Conv-GRU CNP modules, as opposed to heavier LSTM or attention-based blocks, allows LSZone to maintain high performance in multi-zone speech separation scenarios while drastically reducing MAC count and real-time factor (RTF).

4. Performance Evaluation and Empirical Findings

Empirical results substantiate the efficacy of the Conv-GRU CNP approach:

Variant	MACs (G)	RTF	CER/FIR Performance
SpatialNet-Conv-GRU CNP	1.38	0.93	Comparable to LSTM/RNN
LSZone (with CNP & SpaIEC)	0.56	0.37	Maintains accuracy

Values extracted from Table 3 of the referenced paper.

In direct comparison with LSTM-based or attention-based models, the CNP-equipped SpatialNet achieves equivalent character error rate (CER) and false intrusion rate (FIR) scores but requires only a fraction of the computational resources. The full LSZone stack, integrating both SpaIEC and Conv-GRU CNP modules, attains a system MAC complexity of 0.56G and a real-time factor of 0.37, suitable for demanding in-vehicle real-time deployments in noisy, multi-speaker environments.

5. Mathematical Representations and Layer Interactions

The GRU processing within the CNP module is formally described by the system of update equations given above. The Conv Crossband activity is represented as nested convolutional transformations: $Y = \text{G-Conv1d}(\text{F-Conv1d}(X))$ Here, $X$ contains spatial and spectral features from the preceding SpaIEC stage, while Frequency Conv1D ($\text{F-Conv1d}$) and Group Conv1D ($\text{G-Conv1d}$) perform successive dimension-specific operations.

These layers ensure that spatial, spectral, and temporal characteristics are all interactively compressed and modeled, permitting accurate separation of speech signals from multiple zones within the vehicle despite severe computational constraints.

6. Context, Significance, and Comparative Impact

The primary significance of the Conv-GRU CNP module lies in its tailored efficiency for high-stakes embedded applications, specifically for in-car multi-zone speech separation. Prior approaches, such as those utilized in earlier iterations of SpatialNet, incurred prohibitive computational costs due to reliance on large RNNs or attention mechanisms. The CNP module's two-stage alternating architecture yields substantial reductions in MACs and RTF without sacrificing spatial modeling capability or separation accuracy.

This suggests that the Conv-GRU CNP design paradigm can be generalized to other resource-constrained, real-time signal processing tasks where multidimensional feature modeling and rapid inference are paramount. A plausible implication is enhanced accessibility of advanced speech and audio separation technologies for platforms with limited hardware budgets, such as automotive infotainment and cockpit interaction systems.

7. Limitations and Potential Research Directions

While the Conv-GRU CNP module delivers a favorable trade-off between computational load and separation accuracy, its expressiveness in modeling global long-range dependencies may be inherently lower than more complex attention or transformer-based approaches. In environments requiring modeling of nonlocal or intricate crossband interactions extending beyond what grouped convolutions and narrowband GRUs allow, further augmentation or hybridization may be warranted.

Future directions include the exploration of multi-stage feature fusion prior to CNP processing, dynamic reconfiguration of crossband-narrowband alternation, and integration with emerging resource-efficient spatial encoding frameworks. The use of the CNP module as a generic lightweight backbone in other spatial audio contexts also remains an active area of investigation.