Phonetic Error Analysis of Raw Waveform Acoustic Models

Abstract: We analyse error patterns of raw waveform acoustic models on TIMIT phone recognition beyond the overall phone error rate (PER). PER is decomposed across three broad phonetic class (BPC) categorisations, and confusion matrices are constructed from substitution errors. Our models combine parametric (SincNet, Sinc2Net) or non-parametric CNNs with Bidirectional LSTMs, achieving 13.9%/15.3% PER on Dev/Test, the best reported results for raw waveform models on TIMIT. Transfer learning from WSJ reduces PER to 11.3%/12.3%, surpassing the Filterbank baseline. Per-BPC analysis reveals that BLSTM layers benefit transition-dependent classes most, while WSJ transfer learning improves consonants roughly three times more than vowels. Confusion patterns are consistent across raw waveform and Filterbank systems, indicating that the dominant confusions reflect inherent phonetic similarities.

• The paper demonstrates that adding BLSTM layers and leveraging WSJ transfer learning substantially reduces phone error rates compared to FBank baselines.
• The analysis decomposes errors across broad phonetic classes, revealing consistent confusion patterns and quantifying improvements—such as a 28% reduction for Diphthongs.
• The work compares parametric and non-parametric front-ends, showing that adequately trained raw waveform models can outperform traditional feature-engineered systems.

Phonetic Error Analysis of Raw Waveform Acoustic Models

Overview

This work systematically investigates the phonetic error patterns of raw waveform acoustic models for phone recognition, focusing on per-broad phonetic class (BPC) error distributions and confusions, rather than only overall phone error rate (PER). The study uses TIMIT as the primary benchmark and evaluates models incorporating both parametric (SincNet, Sinc2Net) and non-parametric (standard CNN) learnable front-end architectures, integrated with bidirectional LSTM (BLSTM) and fully connected (FC) layers. Results are compared to an established Filterbank (FBank) baseline, and additional experiments consider the impact of transfer learning from WSJ data. The analysis encompasses error decomposition across BPCs, effect of sequential modeling, transfer learning, and direct comparison to feature-engineered systems.

Figure 1: The raw waveform acoustic model employs a convolutional (parametric/non-parametric) layer, followed by BLSTM and FC layers, and dual output heads for context-dependent and context-independent recognition.

Phonetic Categorisations and Methodology

The study extends previous FBank-based analyses, decomposing PER into substitution, deletion, and insertion errors across three established BPC categorisations: (i) 8-class (affricate, diphthong, fricative, nasal, plosive, semi-vowel, vowel, silence), (ii) 3-class (Consonant, Vowel$^+$, Silence), and (iii) 3-class (Voiced, Unvoiced, Silence). Confusion matrices are generated from substitution errors, revealing inter-class confusion structures.

All models are trained using cross-entropy loss with the PyTorch-Kaldi toolkit. For transfer learning, models are pre-trained on WSJ, with a controlled layer re-initialization protocol before fine-tuning on TIMIT. The convolutional front-ends serve either as parametric (frequency-shaped) or non-parametric filters, enabling a comparative analysis on the effect of inductive bias and data efficiency.

Model Performance and Transfer Learning

The proposed architectures (CNN+BLSTM, SincNet+BLSTM, Sinc2Net+BLSTM) achieve PERs of 13.9% (Dev) / 15.3% (Test) without external data, establishing a new benchmark for raw waveform models on TIMIT and outperforming prior CNN, SincNet, GammaNet, GaussNet, and ParzNet results. With WSJ pretraining, PERs shrink to 11.3% (Dev) / 12.3% (Test), surpassing the FBank-WSJ baseline.

Raw waveform models generally require more data to outperform fixed-feature FBank systems, but once their data requirements are satisfied—via cross-corpus transfer—they exhibit superior per-BPC performance, particularly among context-dependent acoustic-phonetic classes.

Sequential Modelling Effects

Adding BLSTM layers yields the largest PER reduction for Diphthongs (≈28%), Fricatives (≈19%), and Semi-vowels (≈18%) across all convolutional front-ends. These BPCs are characterized by strong temporal dependencies (formant transitions, sustained noise, and rapid spectral changes) that convolution-only front-ends (lacking sequence context) cannot fully model. Vowels, being more acoustically stationary, benefit less (≈10%), and Affricates display high variance due to limited data.

Impact of Transfer Learning

Transfer from WSJ data induces a marked consonant-vowel asymmetry, where consonants (Nasals, Fricatives, Semi-vowels) achieve relative PER improvements of roughly 30%, while vowels gain only about 10%. The gain is consistent across architectures and datasets, highlighting that classes with context-dependent acoustic realizations benefit most from exposure to greater phonetic diversity. Silence gains minimally or even loses accuracy, as cross-corpus transfer does not effectively generalize across differing recording conditions.

Confusion Structure and Class Interactions

Analysis of confusion matrices across three model configurations (CNN, CNN+BLSTM, CNN+BLSTM+WSJ TL) demonstrates that confusable class clusters remain invariant: Plosives frequently confuse with Fricatives, while Vowels, Diphthongs, and Semi-vowels are regularly interchanged. These confusions are driven by intrinsic phonetic similarities and persist even after major architectural or data changes.

Figure 2: Confusion matrices reveal robust clusters of confusable BPCs, consistent regardless of BLSTM layers or WSJ transfer learning.

The inclusion of BLSTM layers and transfer learning sharply reduces overall substitutions, but the underlying confusion structure—both at first- and second-most confused class—remains unchanged, further suggesting these are acoustically grounded phenomena.

Raw Waveform versus Filterbank Systems

Comparative evaluation shows that FBank features yield lower PER than raw waveform CNNs when trained only on TIMIT, reflecting the benefit of strong inductive priors under data constraints. However, with WSJ transfer learning, raw waveform models achieve lower PER across most BPCs, most notably Nasals, Fricatives, and Semi-vowels. This underscores that joint optimization on the raw signal provides greater representational capacity, but requires ample, diverse training data to avoid overfitting.

Figure 3: Per-BPC PER (\%) for Filterbank (FBank) and raw waveform CNN systems on TIMIT Test, showing reversal of relative efficacy when substantial transfer learning is applied.

Confusion patterns—most notably the dominance of Plosive-Fricative and Vowel-Diphthong-Semi-vowel confusion—are nearly identical for both architectures. This convergence indicates that feature front-end choice influences overall PER but has limited effect on the nature of residual confusions between BPCs, which are largely dictated by underlying acoustic-phonetic structure.

Implications and Future Directions

These findings have several notable implications:

• Targeted Model Improvements: Error and confusion analysis at the BPC level suggests potential for targeted augmentation (e.g., for context-dependent classes) and class-wise loss weighting, which could selectively mitigate difficult confusions or data imbalance.
• Model Selection Guidance: The interplay between front-end flexibility, data efficiency, and confusion structure provides quantitative guidance on when to deploy learnable front-ends versus feature-engineered representations, particularly as ASR systems move toward more data-rich scenarios.
• Generalization Beyond Filterbank Limitations: That raw waveform models, once adequately trained, outperform traditional pipelines without altering the confusion structure, reinforces the view that residual errors are bounded by inherent acoustic ambiguities, not representational constraints.

A natural progression is to extend this analytic framework to end-to-end and self-supervised models, including those optimized with non-phonetic objectives, to interrogate whether learned representations alter or preserve BPC confusion structure at scale.

Conclusion

This paper demonstrates that raw waveform acoustic models, when equipped with BLSTM and substantial cross-corpus training, outperform Filterbank systems on phoneme recognition, with the largest relative gains among classes with prominent temporal dynamics. However, the distribution of residual confusions is fundamentally shaped by acoustic-phonetic realities and persists across architectures. These results both clarify the circumstances under which raw waveform modeling is most beneficial and suggest a data-informed approach to further architectural enhancements (2606.07030).