Channel-Split NAC-RoFormer

• Channel-Split NAC-RoFormer is a neural architecture that enhances speech by splitting encoded channels and applying dual-path attention to capture both local temporal and global inter-group dependencies.
• The method reduces computational complexity while preserving fine-grained spectral details through periodic (Snake) activations, improving performance under compound distortions.
• Empirical results demonstrate its effectiveness within the OmniGSE framework, outperforming traditional models in overall quality and robust speech reconstruction.

The channel-split NAC-RoFormer is a neural architecture designed to enhance speech signals within OmniGSE, a general speech enhancement framework targeting the mitigation of diverse, compounding signal distortions. Specifically, the channel-split NAC-RoFormer addresses the computational and performance challenges associated with high-dimensional neural audio codec (NAC) encoded features by employing a dual-path attention mechanism that splits channels into independent groups. This method enables the efficient modeling of both local (temporal) and global (channel-group) dependencies and serves as a critical precursor to a follow-on hierarchical LLM, which reconstructs clean speech from the enhanced continuous features. Empirical analysis demonstrates that this approach achieves superior denoising and restoration on a variety of compound distortion scenarios, outperforming both discriminative and generative baselines (Mu et al., 25 Jul 2025).

1. Structural Overview of Channel-Split NAC-RoFormer

The NAC-RoFormer operates on continuous (pre-quantized) feature representations $F_{\text{enc}} \in \mathbb{R}^{D \times T}$ obtained from a pretrained NAC encoder, where $D$ is the channel dimensionality and $T$ is the sequence length. To circumvent the prohibitive computational cost and over-smoothing typically induced by global attention over all channels, $F_{\text{enc}}$ is uniformly divided along the channel axis into $G$ non-overlapping groups, each containing $D_\text{group}$ channels ($D = G \times D_\text{group}$). The resulting grouped tensor $$F' \in \mathbb{R}^{G \times D_\text{group} \times T}$$ serves as the basis for dual-path attention.

Dual-path attention entails two concurrent operations:

1. Local (Temporal) Self-Attention: Applied to each channel group independently along the temporal dimension. For group $g$, the computation is

$$A_\text{temp}^{(g)} = \mathrm{Softmax}\left( \frac{Q^{(g)}({K}^{(g)})^\top}{\sqrt{d_k}} \right){V}^{(g)},$$

where $Q$, $K$, and $V$ are the query, key, and value matrices derived from $F'$ in that group.

1. Global Inter-Group Attention (RoFormer): Performed across the group axis using RoFormer self-attention, incorporating rotated position embeddings (RoPE) to enhance positional encoding:

$A_\text{channel} = \text{RoFormer\_Attention}(F'),$

which establishes cross-group dependencies absent from the local operation.

After both paths, the updated group features are concatenated along the channel axis, reconstructing an enhanced continuous representation $F_{\text{enh}} \in \mathbb{R}^{D \times T}$. A periodic (Snake) activation function is then applied to incorporate periodic inductive biases in the output activations.

2. Motivation and Theoretical Rationale

Channel splitting serves dual purposes: reducing computational complexity and preserving fine-grained detail in the presence of multiple distortions. The quadratic cost of traditional global attention in time-series models becomes prohibitive for large $D$; the grouping strategy reduces the attention computation to a manageable number of channels per group. Furthermore, local attention preserves intra-group coherence, while global group interactions—facilitated by RoFormer—provide an expressive mechanism for capturing inter-group dependencies necessary for restoring globally consistent acoustic patterns.

The application of periodic (Snake) activations introduces a structured inductive bias, favoring the recovery of periodic spectral patterns commonly found in speech, thereby enhancing the amplitude modulation capabilities of the model.

3. Role in the OmniGSE Framework

In the two-stage OmniGSE architecture, the output $F_{\text{enh}}$ of the channel-split NAC-RoFormer performs a foundational role. These features exhibit a high signal-to-noise ratio, serving as robust conditioning signals for a subsequent hierarchical LLM (LM) responsible for discrete speech token generation. The improved feature grounding is crucial for resilience against compound distortions comprising noise, reverberation, clipping, bandwidth limitations, and packet loss. The pipeline is summarized as follows:

Stage	Input	Main Module	Output
Stage I	$F_{\text{enc}}$ (NAC output)	Channel-split NAC-RoFormer	$F_{\text{enh}}$ (enhanced)
Stage II	$F_{\text{enh}}$	Hierarchical LM (RootLM, BranchLMs)	Discrete NAC tokens

The separation enables each stage to specialize: the NAC-RoFormer enhances the continuous signal representation, while the LM reconstructs missing and fine speech details.

4. Hierarchical LLM Integration

After Stage I, OmniGSE invokes a hierarchical LLM consisting of a RootLM and multiple BranchLMs. The RootLM consumes $F_{\text{enh}}$, autoregressively generating a high-level representation $H_{\text{root}}$ that aggregates universal acoustic features (e.g., timbre, prosody) spanning all layers of the codebook.

Each BranchLM corresponds to a layer $l$ in the residual vector quantization (RVQ) codebook stack. For $l = 1, ..., Q$, the BranchLM receives $H_{\text{root}}$ and the token sequence $z_{(l-1)}$ from the previous layer as inputs and predicts the next sequence $z^{(l)}$. This design explicitly models the progressive, hierarchical dependencies implicit in RVQ. The full-stage cross-entropy loss is

$$L_{\text{code}} = -\sum_{l=1}^Q \sum_{t=1}^T \log p\left( z_t^{(l)} \mid z_{< t}^{(l)}, H_\text{root}, \hat{z}^{(l-1)} \right),$$

where $\hat{z}^{(l-1)}$ are teacher-provided NAC tokens.

This structure enables the system to gradually and consistently refine acoustic details, with coarse layers (RootLM) constraining global content and finer layers (BranchLM) successively restoring details.

5. Empirical Results and Ablation Findings

Empirical evaluation reveals that substituting the channel-split NAC-RoFormer with a standard Transformer significantly degrades Overall Quality (OVRL) and NISQA metrics (Experiment (d), Table 5), highlighting the dual-path attention model's role in feature recovery (Mu et al., 25 Jul 2025). Additionally, removing either the NAC-RoFormer (Stage I) or the hierarchical LM (Stage II) results in notable performance drops, establishing the importance of each architecture.

OmniGSE, equipped with both channel-split NAC-RoFormer and hierarchical LM, achieves consistent outperformance over prior models on benchmarks such as the Interspeech 2020 DNS Challenge and the Voicefixer GSR test set. Notable improvements manifest in objective scores (e.g., OVRL, NISQA) and subjective naturalness (NMOS) and speaker similarity (SMOS).

6. Significance and Outlook

The channel-split NAC-RoFormer introduces an effective strategy for enhancing high-dimensional channel features under multi-distortion speech conditions while balancing computational demands. Its integration within OmniGSE demonstrates robust generalizability across diverse scenarios, supporting applications in real-world speech enhancement where simultaneous distortions are prevalent. The architecture reduces over-smoothing, enhances fine-grained recovery, and provides reliable intermediate representations for stochastic generative modules downstream.

A plausible implication is that variations of the channel-split NAC-RoFormer, or similar dual-path grouping schemes, may offer benefits in other high-dimensional sequential data modeling tasks beyond speech enhancement, where efficient dependency modeling across axes is required.