Axis-Separating TF-SepNet

• The paper introduces axis-separating strategies that split feature extraction into temporal and frequential branches, achieving an expanded effective receptive field and improved efficiency.
• TF-SepNet employs separate 1D convolutional kernels to distinctly process time and frequency, optimizing parameter usage compared to traditional 2D methods.
• Empirical results show that TF-SepNet outperforms baseline models in accuracy while significantly reducing parameters and MACs in audio analysis tasks.

Axis-separating TF-SepNet refers to a family of deep neural network architectures that perform explicit and parallel separation of feature extraction along the time and frequency axes in the processing of time-frequency (T-F) representations—typically spectrograms—of audio signals. These models replace conventional 2D convolutional approaches with axis-disentangled 1D kernels or analogous axis-separable blocks, aiming to increase modeling efficiency, improve the effective receptive field (ERF), and exploit the distinct characteristics of audio content in the temporal and spectral domains. Axis-separating design methodologies have been applied in convolutional neural networks (CNNs) as well as transformer-based and hybrid recurrent-attention systems for speech separation, acoustic scene classification, and more general audio understanding tasks (Cai et al., 2023, Wang et al., 2022, Ristea et al., 2022).

1. Motivation for Axis Separation in Audio Models

Typical CNN and transformer models handling spectrograms use 2D kernels (or attention over the entire T-F plane), which process time and frequency together for every feature extraction step. While this facilitates general pattern modeling, it entangles temporal and spectral cues, often leading to inefficient parameterization and computational redundancy. Audio signals exhibit distinct structure along each axis: temporal cues encapsulate rhythm, onsets, and event dynamics, whereas spectral cues relate to timbral and spatial characteristics. Axis-separating approaches address this by explicitly factorizing model layers—either via separate convolutional branches for each axis or sequential axis-separable attention modules—and then merging their outputs. This yields both efficient parameterization and enhanced specialization (Cai et al., 2023, Wang et al., 2022, Ristea et al., 2022).

2. Core Architectural Principles of TF-SepNet

TF-SepNet, as described in "TF-SepNet: An Efficient 1D Kernel Design in CNNs for Low-Complexity Acoustic Scene Classification," structures each convolutional block (TF-SepConv) to process temporal and frequential features in parallel. For an input $x\in\mathbb{R}^{C\times F\times T}$ (channels, frequency, time):

• Channels are split into two halves.
• One half is routed through a purely frequential path: depthwise $3\times1$ convolution, batch norm, ReLU, average pooling along frequency, $1\times1$ pointwise convolution, broadcasting back along frequency.
• The other half takes a symmetric, purely temporal path: depthwise $1\times3$ convolution, pooling and broadcasting along time.
• The outputs are concatenated and passed along the network.

This design avoids the entanglement of 2D kernels and, versus sequential $(k\times1)$ then $(1\times k)$ decomposition, further increases axis specialization and the breadth of the ERF. The TF-SepConv module is used throughout the network, typically within a ResNet-inspired architecture with initial downsampling, channel shuffle, and normalization (Cai et al., 2023).

3. Mathematical Formulation and Effective Receptive Field (ERF)

Let $$K_f\in\mathbb{R}^{(C'/2)\times(C'/2)\times 3\times 1}$$ and $$K_t\in\mathbb{R}^{(C'/2)\times(C'/2)\times 1\times 3}$$ be the frequency and temporal kernel tensors. The two branches are computed as:

$$u^{(f)}_{c,i,j} = \sum_{m=-1}^{+1} K_f[c,m,0]\, x^{(f)}_{c,i+m,j}, \qquad u^{(t)}_{c,i,j} = \sum_{n=-1}^{+1} K_t[c,0,n]\, x^{(t)}_{c,i,j+n}$$

After pooling, pointwise convolution, and broadcasting:

$$\hat x^{(f)}_{c,i,j} = u^{(f)}_{c,i,j} + v^{(f)}_{c,1,j}, \qquad \hat x^{(t)}_{c,i,j} = u^{(t)}_{c,i,j} + v^{(t)}_{c,i,1}$$

The outputs are channel-concatenated. This approach increases the ERF more rapidly compared to standard 2D or consecutive 1D factorized kernels, since each branch attains full-span context along its respective axis via global pooling and broadcasting, yielding a more uniform and extensive ERF. For instance, Table 3 in (Cai et al., 2023) shows the $3\times1$0 high-contribution area ratio for TF-SepNet-40 (13.9%) exceeds that of BC-ResNet-40 (9.6%) and BC-Res2Net-40 (9.9%).

4. Model Complexity and Computational Efficiency

Model complexity analysis demonstrates TF-SepNet achieves significant parameter and multiply-accumulate (MAC) reductions compared to classical 2D CNNs:

Model	Params (×10³)	MACs (×10⁶)	Top-1 Acc. (%)
TF-SepNet-40	53.4	7.03	60.0
BC-ResNet-40	88.1	17.2	57.1
TF-SepNet-80	196.7	24.2	61.6
BC-ResNet-80	315	45.8	58.4

The parameter cost for TF-SepConv is

$3\times1$1

for depthwise and pointwise convolutions (with $3\times1$2), significantly below the $3\times1$3 cost of a standard 2D convolution. Empirically, accuracy improves while model size and computation shrink substantially (Cai et al., 2023).

5. Empirical Performance and Ablations

On the TAU Urban Acoustic Scene 2022 Mobile dataset, TF-SepNet-40 and -80 outperform baseline BC-ResNet and BC-Res2Net models by 0.9–3.2 percentage points in Top-1 test accuracy, with markedly fewer MACs and parameters. Ablation studies confirm that:

• Removing the shuffle unit reduces accuracy slightly.
• Dropping the frequential or temporal branch yields larger accuracy drops (down to 56.7% and 57.5%), increased MACs, and higher parameter counts due to less efficient broadcasting.
• Removing AdaResNorm costs 1.5 points of accuracy.

This pattern demonstrates the necessity of both axis-separating branches and proper normalization for optimal performance (Cai et al., 2023).

6. Relationship to Other Axis-Separable Architectures

Axis-separating design has been generalized beyond CNNs. In speech separation models such as TF-GridNet (Wang et al., 2022), axis-separation is implemented at the module level within each "GridBlock," with:

• An intra-frame full-band module (frequency-local modeling via BLSTM, 1D unfold and deconv along F)
• A sub-band temporal module (time-local modeling via BLSTM along T)
• Cross-frame self-attention (nonlocal modeling between T-F locations)

Similarly, Separable Transformers (SepTr) (Ristea et al., 2022) apply standard transformer encoder blocks independently along frequency and time axes in succession, with experiments confirming that stacking both orders achieves maximum performance benefits. This design sharply reduces parameter growth with input resolution, maintains linear scalability, and avoids the quadratic scaling that limits dense self-attention.

7. Interpretations and Impact

By decoupling feature extraction along time and frequency axes, axis-separating TF-SepNet and related models achieve several outcomes:

• Axis-specific specialization with maximum ERF
• Low memory and computation footprints suitable for deployment and edge scenarios
• Improved accuracy over baselines on tasks demanding fine, long-range T-F integration

A plausible implication is that for audio and other T-F domain tasks where domain-specific statistical properties differ across axes, axis-separating design paradigms are preferable, especially when model efficiency and axis-local context modeling are critical. This aligns with findings in transformer-based models and deep speech separation architectures, underscoring the broad applicability and effectiveness of axis-separating principles (Cai et al., 2023, Wang et al., 2022, Ristea et al., 2022).