Raw Waveform Mapping and CNNs

• Raw waveform mapping is the end-to-end transformation of discrete audio signals into high-level features using CNNs without handcrafted spectral preprocessing.
• Convolutional architectures span free-form filters to structured front-ends like SincNet and PF-Net, enhancing interpretability and parameter efficiency.
• Empirical studies show that raw waveform CNNs boost performance in audio classification, speech recognition, and biomedical tasks while reducing model complexity.

Raw waveform mapping with convolutional architectures constitutes a foundational paradigm in modern audio, speech, and time-series modeling. This approach leverages parameterized or free-form convolutional layers to extract representations directly from discretized audio samples, bypassing traditional handcrafted spectral representations such as MFCC or log-mel filterbanks. Current literature explores both generic and highly structured convolutional front-ends, architectures promoting inductive bias (e.g., bandpass filtering, scale-equivariance), and deep stacks capable of capturing hierarchical, temporal, and multi-scale patterns. Raw waveform CNNs have demonstrated empirical superiority or competitiveness across audio classification, tagging, generation, enhancement, localization, and biomedical tasks.

1. Fundamental Principles of Raw Waveform Mapping

Raw waveform mapping is defined as the end-to-end transformation of discrete audio sequences—typically sampled at 8–44.1 kHz—into high-level features or task outputs using neural architectures without manual front-end feature extraction. In the convolutional paradigm, the core operation is

$y_k[n] = \sum_{m=0}^{M-1} h_k[m]\cdot x[n-m] + b_k$

where $h_k[m]$ is the kernel of the $k$-th filter, $M$ is the filter length, and $b_k$ is bias.

Traditional DNNs and spectrogram-CNNs rely on preprocessing pipelines (STFT, log-mel, MFCC), but raw waveform approaches eliminate spectral domain assumptions, enabling architectures to learn task-driven representations of both magnitude and, implicitly or explicitly, phase—crucial for tasks such as enhancement and localization (Fu et al., 2017, Vecchiotti et al., 2019). The rationale is to align the neural front-end with the native data format, thereby offering increased flexibility, improved information retention, task adaptivity, and end-to-end differentiability.

2. Convolutional Architecture Variants and Inductive Bias

Raw waveform CNNs span a spectrum:

a. Free-form CNNs: Each filter learns all tap weights, as in early CNN models for environmental sound (Qu et al., 2016), speech recognition (Dai et al., 2016), and music tagging (Kim et al., 2017). These models typically deploy large first-layer kernels to emulate bandpass behavior and deeper layers with small receptive fields for abstraction.

b. Structured filterbanks and parameterized frontends:

• SincNet replaces the first convolutional layer with bandpass filters parameterized by analytic sinc functions, learning only low/high cutoff frequencies, thus enforcing linear-phase behavior and extreme parameter efficiency. Layer 1 parameter count reduces from $FL$ to $2F$ for $F$ filters of length $L$ (Ravanelli et al., 2018, Ravanelli et al., 2018).
• PF-Net generalizes this by learning piecewise-linear frequency responses via deformation points, combining interpretable bandpass shapes with greater flexibility (Li et al., 2021).
• Cosine-modulated Gaussian filterbanks use kernels of the form $g_k[n]=\cos(2\pi\mu_k n)\exp(-n^2\mu_k^2/2)$, with $\mu_k$ learnable, as in raw-attention frontends (Agrawal et al., 2020, Dutta et al., 2021).
• IConNet constructs band-limited sinc-difference kernels modulated by small-parameter cosine windows, with variants for learning band/bandwidth, window shape, or both (Vu et al., 3 May 2024).

c. Wavelet and scale-equivariant architectures: Wavelet Networks (Romero et al., 2020) construct each layer with scale- and translation-equivariant 1D group convolutions, analytically dilating mother filters and preserving multi-scale structure explicitly.

d. Multi-scale and multi-level convolutional attention: Architectures such as MuSLCAT employ parallel convolutional branches tuned for different frequency bands (low/high), attention-augmented convolution blocks (AAC), and hierarchical fusion followed by Transformers or lightweight attention backends (Middlebrook et al., 2021).

3. Network Depth, Hierarchy, and Multi-scale Design

Several works demonstrate the necessity of deep and/or multi-scale architectures for modeling long-range temporal dependencies and capturing hierarchical features:

• Sample-level CNNs utilize stacks of micro (2-3-tap) convolutions with repeated pooling, constructing phase-invariant, deep temporal receptive fields (Lee et al., 2017, Kim et al., 2017). Residual connections and squeeze-and-excitation (SE) modules further enable deep stacks without vanishing gradients.
• Very Deep CNNs process waveforms up to 32k samples with up to 34 layers, using aggressive early strided convolution/pooling and residual stacks to ensure computational tractability and gradient stability (Dai et al., 2016).
• Multi-branch/multi-span neural networks process overlapping time spans (e.g., 3-branch structures with different kernel/stride configurations) to capture both fine and coarse context, outperforming STFT-derived representations in acoustic modeling (Platen et al., 2019).
• Attention-augmented convolution and network-in-network self-attention augment the convolutional framework’s representational power and provide soft relevance weighting over time-frequency bins, dynamically enhancing information routing (Middlebrook et al., 2021, Agrawal et al., 2020).

4. Specialized Front-ends, Interpretability, and Efficiency

Embedding strong inductive bias at the front end fosters interpretability, parameter reduction, and faster convergence:

• SincNet delivers interpretable filterbanks whose frequency bands can be visualized and are commonly matched to linguistic or speaker-relevant regions; parameter count in the first layer reduces by orders of magnitude, and convergence is faster than free CNNs (Ravanelli et al., 2018, Ravanelli et al., 2018).
• PF-Net demonstrates that increasing frequency-domain parameter flexibility (beyond SincNet) yields further performance gains, both in speed and accuracy (Li et al., 2021).
• IConNet formalizes the construction of windowed bandpass filters, with initialization from standard mel banks and explicit window parameterization, showing consistent absolute gains of 7% or higher over handcrafted MFCC/mel in emotion and biomedical tasks (Vu et al., 3 May 2024).
• Wavelet Networks provide formal scale-translation equivariance and show empirical improvements over vanilla CNNs, matching mel-spectrogram-based models with reduced parameters (Romero et al., 2020).

The table below summarizes several key parameter-efficient front-ends:

Architecture	Front-end Param.	Inductive Bias	Visualization	Reported Gain
SincNet	2F (cutoffs)	Band-pass, lin. phase	Cutoff bands	10–15% CER/EER vs raw CNN (Ravanelli et al., 2018)
PF-Net	2S·F (freq pts)	Band-pass, arbitrary	Shape/bands	Best CER/EER (Li et al., 2021)
IConNet	(p+2)K (window+band)	Band-limited, window	Filterbank	+7% UA/F1 vs. mel/MFCC (Vu et al., 3 May 2024)

5. Empirical Outcomes and Application Domains

Application of raw waveform convolutional architectures extends across:

• Speech enhancement: Fully convolutional networks (e.g., 6×(15×11) FCN, no FC layers) yield highest speech intelligibility (STOI) and quality (PESQ), with only 0.2% of conventional DNN/CNN parameters while reliably reconstructing high-frequency structure (Fu et al., 2017).
• Speech and speaker recognition: Structured parametric front-ends (SincNet, PF-Net) and multi-level CNNs outperform or match MFCC/fbank pipelines on TIMIT, Librispeech, DIRHA, with substantial parameter reduction and improved generalization (Ravanelli et al., 2018, Li et al., 2021, Ravanelli et al., 2018).
• Music and environmental sound classification: Multi-level, sample-level, and multi-scale CNNs (e.g., MuSLCAT, SampleCNN, Wavelet Networks) set or match state-of-the-art AUC scores, build hierarchical abstractions, and deliver interpretable frequency selectivity (Middlebrook et al., 2021, Lee et al., 2017, Romero et al., 2020).
• ASR: Trainable gammatone, Gabor, or attention-weighted filterbanks learned via raw-CNNs outperform mel-filterbanks in WER and LER, particularly with normalization and Hanning low-pass designs (Zeghidour et al., 2018, Agrawal et al., 2020).
• Binaural localization: End-to-end CNNs with learned or gammatone-based frequency analysis adapt front-end filters based on source environment, optimizing for either ITD/ILD cues in reverberant spaces (Vecchiotti et al., 2019).
• Biomedical / heart-sound analysis: Constrained raw waveform CNNs (IConNet) demonstrate significant gains over MFCC/CRNN baselines in murmur detection with small models suitable for deployment (Vu et al., 3 May 2024).
• Data-efficient generation: Generative models with flow-bases (e.g., WaveFlow) interpolate between autoregressive and parallel flows, achieving efficient, high-fidelity synthesis with O(1) convolutional training and O(h) sampling (Ping et al., 2019).

Consistent across studies, parameterized and interpretable front-ends yield accelerated learning, robustness to domain shifts (e.g., noise or reverberation), and model compactness compared to both fully free CNNs and spectrogram-based pipelines.

6. Advanced Transformations: Group Equivariance and Alternative Mappings

Beyond standard convolutional approaches, several works explore fundamentally new mappings:

• Group-equivariant convolutions on scale-translation groups (Wavelet Networks) formalize scale invariance and show superior data efficiency and generalization across audio and non-audio time series (Romero et al., 2020).
• Space-filling curve mappings transform 1D audio to 2D images via Z-order and related curves, exploiting computer vision networks and shift-equivariance properties. Z-curve achieves performance within 0.2–0.3% of MFCC baselines on keyword tasks due to locality and equivariance (Mari et al., 2022).

These nonstandard approaches further validate the versatility and breadth of raw waveform mappings in modern neural architectures.

7. Future Directions and Controversies

Major axes for ongoing research and debate include:

• Balancing inductive bias and flexibility: While strong constraints (e.g., SincNet, gammatone) yield interpretability and data efficiency, increased parameterization (PF-Net, IConNet window learning) offers improved expressivity. The optimal trade-off remains task-dependent.
• Architectural depth versus efficiency: Deeper stacks (e.g., up to 34 layers (Dai et al., 2016)) facilitate hierarchical processing but pose risks of overfitting and require careful normalization and residual design; lightweight variants (single or few-layer AAC, SDFCN) suffice on smaller domains or when compute is constrained.
• Generalizability: Models learning from raw waveform can, in principle, adapt to new sensor types, domains, or languages without retuning handcrafted preprocessing, but may require more data to stabilize filter learning in low-resource settings.
• Interpretability: Recent paradigms emphasize not just high performance but also human-inspectable filterbanks—quantifiable via frequency response, activation patterns, or mask visualizations (e.g., attention weights highlight frequency bands exploited in noisy/reverberant conditions).

A plausible implication is that the future of raw waveform CNNs lies in hybrid models that combine interpretable, parameter-efficient front-ends with hierarchical, multi-scale, and attention-augmented architectures, augmented by formal equivariance properties when appropriate.

---

• (Fu et al., 2017) Raw Waveform-based Speech Enhancement by Fully Convolutional Networks
• (Ravanelli et al., 2018) Speaker Recognition from Raw Waveform with SincNet
• (Ravanelli et al., 2018) Speech and Speaker Recognition from Raw Waveform with SincNet
• (Li et al., 2021) PF-Net: Personalized Filter for Speaker Recognition from Raw Waveform
• (Vu et al., 3 May 2024) Toward end-to-end interpretable convolutional neural networks for waveform signals
• (Romero et al., 2020) Wavelet Networks: Scale-Translation Equivariant Learning From Raw Time-Series
• (Middlebrook et al., 2021) MuSLCAT: Multi-Scale Multi-Level Convolutional Attention Transformer for Discriminative Music Modeling on Raw Waveforms
• (Lee et al., 2017) Raw Waveform-based Audio Classification Using Sample-level CNN Architectures
• (Dai et al., 2016) Very Deep Convolutional Neural Networks for Raw Waveforms
• (Ping et al., 2019) WaveFlow: A Compact Flow-based Model for Raw Audio
• (Platen et al., 2019) Multi-Span Acoustic Modelling using Raw Waveform Signals
• (Agrawal et al., 2020) Interpretable Filter Learning Using Soft Self-attention For Raw Waveform Speech Recognition
• (Dutta et al., 2021) A Multi-Head Relevance Weighting Framework For Learning Raw Waveform Audio Representations
• (Mari et al., 2022) A novel audio representation using space filling curves
• (Vecchiotti et al., 2019) End-to-end Binaural Sound Localisation from the Raw Waveform
• (Kim et al., 2017) Sample-level CNN Architectures for Music Auto-tagging Using Raw Waveforms
• (Qu et al., 2016) Understanding Audio Pattern Using Convolutional Neural Network From Raw Waveforms
• (Zeghidour et al., 2018) End-to-End Speech Recognition From the Raw Waveform
• (Liu et al., 2019) Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networks