Waveform-Logmel Audio Neural Network (WLANN)

• Waveform-Logmel Audio Neural Network is a dual-branch model that combines raw waveform and log-mel spectrogram inputs to capture both fine time-domain and stable spectral features.
• It employs specialized branches using 1D CNNs, Bi-LSTMs, CNNs, and Transformers, fused via methods like channel concatenation and cross-attention for robust audio feature extraction.
• WLANN models achieve state-of-the-art performance in benchmarks such as audio tagging, respiratory sound classification, and on-device audio processing with improved mAP and efficiency.

A Waveform-Logmel Audio Neural Network (WLANN) is a class of neural architectures designed to combine representations learned from raw audio waveforms and log-mel spectrograms, typically for tasks such as audio tagging, sound event detection, respiratory sound classification, and efficient on-device audio understanding. WLANN models incorporate parallel or fused branches, each specialized for capturing distinct audio characteristics, achieving high accuracy and transferability across diverse audio pattern recognition tasks.

1. Fundamental Architecture and Variants

The foundational design of WLANN consists of two primary input branches:

• Waveform Branch: Processes the raw audio waveform, frequently via stacks of 1D convolutions (Kong et al., 2019), temporal convolutions (Xie et al., 24 Apr 2025), or recurrent layers such as Bi-LSTM (Choudhary et al., 2023).
• Logmel Branch: Operates on the log-mel spectrogram representation of the audio, generally utilizing a deep convolutional backbone (e.g., CNN14 (Kong et al., 2019)), an EfficientNet/AST transformer (Xie et al., 24 Apr 2025), or pretrained modules like YAMNet (Choudhary et al., 2023).

These branches are fused at an intermediate or representation level using concatenation, cross-attention, or sequence-modeling units (e.g., Bi-GRU), followed by global pooling and a classification head suitable for multi-label or multi-class targets.

Variant / Paper	Waveform Processing	Logmel Branch	Fusion Method	Temporal Modeling
PANNs/Wavegram-Logmel-CNN (Kong et al., 2019)	1D CNN (Wavegram)	CNN14 (2D CNN)	Channel concatenation	Global pooling
WLANN–Respiratory (Xie et al., 24 Apr 2025)	1D CNN (frame-wise)	AST (spectrogram Transformer)	Concatenation	Bi-GRU
LEAN (Choudhary et al., 2023)	Bi-LSTM	Pretrained YAMNet	Cross-attention	Final context Bi-LSTM

2. Input Feature Construction and Preprocessing

Two distinct feature domains are exploited in WLANN systems:

• Raw Waveform: Audio is converted to mono and uniformly resampled (e.g., 32 kHz (Kong et al., 2019), 16 kHz (Xie et al., 24 Apr 2025, Choudhary et al., 2023)). Padding or truncation is used for fixed-length input. Preprocessing may include bandpass filtering (e.g., 40–850 Hz for respiratory sounds (Xie et al., 24 Apr 2025)).
• Log-Mel Spectrogram: For log-mel, a Short-Time Fourier Transform (typical window sizes 25–32 ms, hop sizes 10 ms) is computed, yielding a magnitude or power spectrogram, mapped with a Mel filterbank (commonly 64–128 bins). The log operation ensures numerical stability (e.g., $$\hat S(m, f) = \log(S_{\mathrm{mel}}(m, f) + \varepsilon)$$ with $\varepsilon=10^{-6}$ (Kong et al., 2019, Xie et al., 24 Apr 2025, Choudhary et al., 2023)).

This dual representation captures both fine time-domain transients and stable spectral content.

3. Network Architecture Details

3.1. Waveform Branch

The waveform pathway varies:

• PANNs/Wavegram-Logmel-CNN: Multiple 1D convolutional blocks with progressive downsampling and increasing channels, outputting a 2D “Wavegram” ($\mathbb{R}^{100\times64\times6}$) structurally analogous to a spectrogram (Kong et al., 2019).
• WLANN–Respiratory: Stacked Conv1D layers (kernel size 80, stride 5/4) with batch norm, ReLU, and max pooling. Channels are reshaped for pseudo-frequency slicing and aligned in frames ($F\times T\times (C/F)$) (Xie et al., 24 Apr 2025).
• LEAN: Purely Bi-LSTM layers (two stacked, 128 units per direction), reshaped into frame-size blocks ($40\times400$) from raw 1 s waveform (Choudhary et al., 2023).

3.2. Log-Mel Branch

• CNN14 / ResNet / MobileNet: 2D convolutional backbones process log-mel inputs, passing through sequential pooling and expansion blocks to yield high-dimensional embeddings (Kong et al., 2019, Choudhary et al., 2023).
• AST: Transformer-based spectral encoders (e.g., Audio Spectrogram Transformer (Xie et al., 24 Apr 2025)) operate on log-mel patch tokens, leveraging positional encodings and multi-head attention.
• Pretrained YAMNet: MobileNetV1-inspired lightweight structure, finalized by a dense 256-unit projection (Choudhary et al., 2023).

3.3. Fusion and Temporal Modeling

• Channel Concatenation: Output feature maps from parallel branches are concatenated along the channel axis after aligning temporal and frequency dimensions, then processed by further shared CNN layers (e.g., Block 6 in (Kong et al., 2019)).
• Cross-Attention: Embeddings from the log-mel branch (e.g., $E_{yam}$) and time-domain features (hidden sequence $H$) are combined via either affinity-based (dot-tanh) or Bahdanau-style additive attention, forming a fused context vector (Choudhary et al., 2023).
• GRU/Bi-GRU: Framewise concatenated outputs are mean-pooled over frequency and input into bidirectional GRUs to model temporal dependencies, followed by pooling and classification (Xie et al., 24 Apr 2025).

4. Training Strategies and Optimization

Component	Details
Loss function	Binary cross-entropy (multi-label) (Kong et al., 2019), multi-class focal loss ($\gamma=2$) (Xie et al., 24 Apr 2025)
Optimizer	Adam (typical LR $1\times10^{-3}$ or $1\times10^{-4}$) (Kong et al., 2019, Xie et al., 24 Apr 2025, Choudhary et al., 2023)
Data augmentation	SpecAugment (time/frequency masking), Mixup (Kong et al., 2019), random cropping, additive noise (Xie et al., 24 Apr 2025)
Data balancing	Class-balanced mini-batching (Kong et al., 2019)
Pretraining	AudioSet (1.9 M clips for PANNs (Kong et al., 2019), YAMNet (Choudhary et al., 2023))
Fine-tuning	Replace terminal FC, update all weights or freeze, adapt to target task (Kong et al., 2019, Choudhary et al., 2023)

SpecAugment and mixup regularize training, and the focal loss mitigates extreme class imbalances for rare sound events (Xie et al., 24 Apr 2025). WLANNs are commonly pretrained on large-scale data to enable effective transfer to small or specialized datasets.

5. Empirical Performance and Ablation

5.1. Audio Tagging and Recognition Performance

• PANNs/Wavegram-Logmel-CNN achieves mAP 0.439, AUC 0.973, $d'$=2.720 on AudioSet, exceeding log-mel only (mAP 0.431), ResNet38 (0.434), and Google (0.314) (Kong et al., 2019).
• LEAN achieves mAP 0.4677, mAUC-PR 0.944, $d'$=2.251 on FSD50K, with an on-device (TFLite, S21, 4.5 MB) mAP of 0.445 (Choudhary et al., 2023).
• Respiratory Sound Classification: Inter-patient test, WLANN achieves SN=90.3%, SP=96.9%, TS=93.6%, outperforming CNN/ResNet and comparable dual-route systems (Xie et al., 24 Apr 2025).

Fine-tuning PANNs leads to state-of-the-art transfer on ESC-50 (94.7%), MSoS (96.0%), and RAVDESS (72.1%); see Table 3 (Kong et al., 2019).

5.2. Ablation and Complexity

• Joint branch fusion in WLANN outperforms both log-mel only and waveform only (e.g., +0.008 mAP over log-mel for PANNs (Kong et al., 2019)).
• Each component (wave encoder, attention) in LEAN adds incremental improvement in mAP (Choudhary et al., 2023).
• Waveform only + Bi-GRU underperforms compared to fusions with AST (Xie et al., 24 Apr 2025).

Model	MACs (per 10 s)	Params (M)	mAP
CNN14 (log-mel)	42.2×10⁹	80.8	0.431 (Kong et al., 2019)
Wavegram-Logmel-CNN	53.5×10⁹	81.1	0.439
MobileNetV2	2.8×10⁹	4.1	0.383
LEAN (TFLite)	-- (on-device)	4.5 MB	0.445 (Choudhary et al., 2023)

WLANN’s complexity is modestly higher than log-mel-only models, but lightweight variants (LEAN) address mobile constraints (Choudhary et al., 2023).

6. Applications and Use Cases

WLANN models have demonstrated capability across standard and specialized audio tasks:

• General Audio Tagging: Tagging hundreds of classes on AudioSet, FSD50K (Kong et al., 2019, Choudhary et al., 2023).
• Transfer Learning: ESC-50, MSoS, RAVDESS, DCASE scene and event classification as fine-tuned benchmarks (Kong et al., 2019).
• Medical Diagnostics: Pediatric respiratory sound classification (wheeze, crackle, stridor, etc.) on SPRSound, with high sensitivity/specificity (Xie et al., 24 Apr 2025).
• On-Device Classification: Resource-constrained variants achieve low latency and compact size for mobile edge devices (Choudhary et al., 2023).

The dual-branch design is particularly effective for tasks requiring fine time resolution (e.g., medical auscultation), leveraging both high-temporal and frequency spectral cues.

7. Limitations and Future Directions

Current weaknesses and directions include:

• Model Size and Efficiency: Standard WLANNs (e.g., PANNs) are large; LEAN and MobileNetV2 provide lightweight alternatives (Kong et al., 2019, Choudhary et al., 2023). Respiratory WLANN relies on transformer backbones incurring parameter and compute overhead (Xie et al., 24 Apr 2025).
• Sensitivity to Class Imbalance: Despite focal loss and balanced sampling, rare-event detection remains challenging (Xie et al., 24 Apr 2025).
• Transfer Scope and Generalization: Most frameworks assume large-scale pretraining, with transfer performance tapering on highly out-of-domain or severely undersampled target tasks.
• Potential Advances: Retraining AST jointly, replacing GRU with temporal convolutional networks, model distillation, and adaptation to other health audio modalities are proposed (Xie et al., 24 Apr 2025).

A plausible implication is that architectures propagating both waveform and spectral features will remain state-of-the-art for a range of audio understanding scenarios, especially as model compression and multimodal fusion strategies advance.

---

References:

• (Kong et al., 2019)
• (Xie et al., 24 Apr 2025)
• (Choudhary et al., 2023)