TF-SepNet: Efficient Acoustic Scene CNN

• The paper introduces TF-SepNet, a convolutional neural network that explicitly decouples time and frequency feature extraction via 1D depthwise convolutions, boosting classification accuracy.
• It leverages separate processing paths to expand the effective receptive field, significantly cutting down computational cost with reduced MACs and parameters.
• Comprehensive ablation studies confirm that components like channel shuffle and AdaResNorm are critical for achieving efficiency and superior performance on mobile acoustic benchmarks.

TF-SepNet is a convolutional neural network architecture specifically designed for efficient and accurate acoustic scene classification (ASC). Its principal innovation is the explicit separation of time-domain and frequency-domain feature extraction via one-dimensional (1D) depthwise convolutions, followed by lightweight channel fusion. This structural disentanglement contrasts with standard and sequential 1D CNN designs and yields reduced computational complexity, an expanded effective receptive field (ERF), and empirically superior classification performance on resource-constrained tasks such as the TAU Urban Acoustic Scene 2022 Mobile benchmark (Cai et al., 2023).

1. Motivation and Design Principles

Acoustic scene classification systems require high accuracy and low resource use for on-device, real-time applications. Conventional CNNs for ASC utilize stacked two-dimensional (2D) kernels ($k \times k$) that convolve across both time and frequency simultaneously, resulting in significant parameter and computation overhead. Prior approaches attempting efficiency improvements replaced 2D kernels with sequences of 1D kernels ($k \times 1$ then $1 \times k$), but still process time-frequency dependencies sequentially. TF-SepNet introduces a stricter decomposition: the intermediate feature map is divided along the channel dimension, with one half processed along frequency by depthwise $3 \times 1$ convolutions and the other half along time via $1 \times 3$ convolutions. These are recombined (channel concatenation and shuffle) after pool-projection-broadcast operations, preserving decoupling until the fusion stage. This explicitly factorized process yields lower MACs/parameters and a larger ERF compared to both standard 2D and sequential 1D models (Cai et al., 2023).

2. Detailed Network Architecture

TF-SepNet operates on log-Mel spectrogram inputs ($1 \times 256 \times 64$, frequency $\times$ time frames). The architecture is as follows:

Output Shape	Architecture	$k$	$s$	$p$
$k \times 1$0	Input	–	–	–
$k \times 1$1	Conv–BN–ReLU	$k \times 1$2	2	1
$k \times 1$3	Conv–BN–ReLU (groups $k \times 1$4)	$k \times 1$5	2	1
$k \times 1$6	TF-SepConvs × 2	–	–	–
$k \times 1$7	MaxPool	$k \times 1$8	2	0
$k \times 1$9	TF-SepConvs × 2	–	–	–
$1 \times k$0	MaxPool	$1 \times k$1	2	0
$1 \times k$2	TF-SepConvs × 2	–	–	–
$1 \times k$3	TF-SepConvs × 3	–	–	–
$1 \times k$4	Conv–BN–ReLU	$1 \times k$5	1	0
$1 \times k$6	GlobalAvgPool	–	–	–

Each TF-SepConvs block begins with a $1 \times k$7 pointwise convolution for channel adjustment, channel shuffle, then equal partitioning into frequential $1 \times k$8 and temporal $1 \times k$9 halves. $3 \times 1$0 receives a depthwise $3 \times 1$1 convolution, averaging, pointwise convolution, and broadcast. $3 \times 1$2 follows the same scheme with $3 \times 1$3 convolutions. Outputs $3 \times 1$4 are concatenated to form the block result. Adaptive residual normalization (AdaResNorm) follows each block for training stabilization (Cai et al., 2023).

3. Mathematical Formulation: 1D Versus 2D Convolutions

A 1D convolution on input $3 \times 1$5 and kernel $3 \times 1$6 (length $3 \times 1$7) is: $3 \times 1$8 A 2D convolution over feature map $3 \times 1$9 with filter $1 \times 3$0 ($1 \times 3$1), is: $1 \times 3$2 A 2D filter has $1 \times 3$3 parameters per input-output channel pair, while 1D has only $1 \times 3$4. TF-SepNet employs depthwise $1 \times 3$5 and $1 \times 3$6 kernels and pointwise $1 \times 3$7 fusions, splitting channels for further efficiency (Cai et al., 2023).

4. Complexity Analysis

Parameter and operation counts for conventional $1 \times 3$8 2D, sequential 1D, and TF-SepNet decompositions are:

Model/Block	Params	Approx. MACs
2D Conv ($1 \times 3$9, $1 \times 256 \times 64$0)	$1 \times 256 \times 64$1	$1 \times 256 \times 64$2
Sequential 1D (see text)	$1 \times 256 \times 64$3	$1 \times 256 \times 64$4
TF-SepNet-40 (empirical)	53.4 K	7.0 M
BC-ResNet-40	88.1 K	17.2 M
BC-Res2Net-40	85.8 K	17.2 M

TF-SepNet-40 achieves a MAC and parameter reduction of ~60% relative to BC-ResNet/Res2Net-40 baselines at similar or better accuracy (Cai et al., 2023).

5. Effective Receptive Field (ERF) Properties

In CNNs, the effective receptive field is the input region most influencing a neuron's output, quantified via the spatial distribution of backpropagated gradients. TF-SepNet displays a broader and more uniform ERF, with a larger top-30% contributing area ($1 \times 256 \times 64$5 for TF-SepNet-40, versus $1 \times 256 \times 64$6 for BC-ResNet-40 and $1 \times 256 \times 64$7 for BC-Res2Net-40). This increase in ERF spatial extent enables the capture of longer-range time-frequency dependencies at fixed depth, which plausibly underpins observed accuracy gains (Cai et al., 2023).

6. Experimental Methodology and Results

TF-SepNet was evaluated on the TAU Urban Acoustic Scene 2022 Mobile development set (10 classes, multiple devices), using the official 70/30 train/test split. Preprocessing included resampling to 32 kHz, STFT (window 3072, hop 500), and Mel filterbank (256 bins, 4096-point FFT, log transform). The training protocol comprised 100 epochs with Adam optimizer, batch size 32, learning-rate warmup then cosine-annealing, mixup and Freq-MixStyle regularization, and AdaResNorm. Performance metrics:

Model	Acc (%)	MACs (M)	Param (K)
DCASE Baseline	42.9	29.2	46.5
BC-ResNet-40	57.1	17.2	88.1
BC-Res2Net-40	59.1	17.2	85.8
TF-SepNet-40	60.0	7.0	53.4
BC-ResNet-80	58.4	45.8	315.0
BC-Res2Net-80	59.6	42.7	307.0
TF-SepNet-80	61.6	24.2	196.7

TF-SepNet outperformed all compared models in accuracy, with marked reductions in both MACs and parameter count (Cai et al., 2023).

7. Ablation and Component Analysis

Ablation experiments established that removal of either the temporal or frequential pathway results in a 2.5–3.3% drop in accuracy, with paradoxical increases in MACs and parameters due to loss of channel compression via the $1 \times 256 \times 64$8 bottleneck. Omitting channel shuffle reduced accuracy by 0.5%. Elimination of AdaResNorm led to a 1.5% accuracy decrease with only a 2% parameter saving. This confirms that separated 1D convolutional paths, AdaResNorm, and channel shuffle collectively underpin the full performance and efficiency of TF-SepNet (Cai et al., 2023).