Automated detection of pronunciation errors in non-native English speech employing deep learning (2209.06265v1)

Abstract: Despite significant advances in recent years, the existing Computer-Assisted Pronunciation Training (CAPT) methods detect pronunciation errors with a relatively low accuracy (precision of 60% at 40%-80% recall). This Ph.D. work proposes novel deep learning methods for detecting pronunciation errors in non-native (L2) English speech, outperforming the state-of-the-art method in AUC metric (Area under the Curve) by 41%, i.e., from 0.528 to 0.749. One of the problems with existing CAPT methods is the low availability of annotated mispronounced speech needed for reliable training of pronunciation error detection models. Therefore, the detection of pronunciation errors is reformulated to the task of generating synthetic mispronounced speech. Intuitively, if we could mimic mispronounced speech and produce any amount of training data, detecting pronunciation errors would be more effective. Furthermore, to eliminate the need to align canonical and recognized phonemes, a novel end-to-end multi-task technique to directly detect pronunciation errors was proposed. The pronunciation error detection models have been used at Amazon to automatically detect pronunciation errors in synthetic speech to accelerate the research into new speech synthesis methods. It was demonstrated that the proposed deep learning methods are applicable in the tasks of detecting and reconstructing dysarthric speech.

• The paper introduces a weakly-supervised dual-task model that marks mispronounced words without relying on full phonetic transcripts.
• It employs uncertainty modeling and an attention mechanism, achieving up to 41% AUC improvement with synthetic speech data.
• The research reduces dependency on costly annotations, paving the way for scalable, efficient computer-assisted pronunciation training systems.

Automated Detection of Pronunciation Errors in Non-Native English Speech Employing Deep Learning

The paper "Automated Detection of Pronunciation Errors in Non-Native English Speech Employing Deep Learning" addresses challenges in the field of Computer-Assisted Pronunciation Training (CAPT). It explores innovative methodologies for improving the accuracy of detecting pronunciation errors in non-native (L2) English speech, with a focus on leveraging deep learning techniques and synthetic speech generation.

Context and Objective

Traditional methods for pronunciation error detection heavily rely on labeled datasets with phonetically transcribed speech, which are costly and challenging to obtain, particularly for L2 learners. These methods have typically achieved only moderate accuracy, with precision often capped around 60% at varying levels of recall. The primary objective of this research is to enhance the accuracy of pronunciation error detection by reducing dependency on extensive labeled datasets and incorporating novel deep learning approaches.

Methodological Innovations

The authors propose several novel methodologies to tackle the identified shortcomings:

1. Weakly-Supervised Learning (WEAKLY-S Model):
   • This model focuses on word-level mispronunciation detection without requiring phonetic transcriptions of non-native speech. By only marking mispronounced words, the model operates under dual-task training, consisting of both a mispronunciation detection task and a phoneme recognition task trained on L1 speech. This multi-task strategy reduces the risk of overfitting and improves detection accuracy.
2. Uncertainty Modeling:
   • The paper emphasizes accounting for uncertainties in phoneme recognitions and the variability inherent in native pronunciations using a deep learning approach. Incorporating multiple potential pronunciations and posterior probabilities leads to more precise detection outcomes, reducing false positive rates.
3. Synthetic Speech Generation:
   • The work explores the potential of synthetic speech as a primary tool for model training. Through methods such as phoneme-to-phoneme (P2P), text-to-speech (T2S), and speech-to-speech (S2S) conversions, the paper generates large volumes of synthetic data, which effectively enhance model training. This reformulation of the problem, viewed through probabilistic machine learning lenses, represents a significant theoretical advancement.
4. Attention Mechanism for Lexical Stress Detection:
   • An attention-based model is introduced to automatically extract relevant audio features from regions critical to determining syllable stress, which improves the detection of lexical stress errors.

Numerical Results and Benchmarking

The WEAKLY-S model achieved a 30% improvement in AUC over previous methods, demonstrating its enhanced accuracy in detecting mispronunciations while also offering scalable training on less expensive data sets. The integration of synthetic speech data proved particularly impactful with the S2S method, improving the AUC by 41%, which signals robust generalization when faced with real-world speech data.

Implications and Future Directions

This research offers several theoretical and practical implications. The reclassification of pronunciation error detection as a data generation task challenges traditional assumptions in the field, paving the way for similar re-evaluations in other domains of speech processing. Practically, the methods outlined can significantly reduce the dependency on large human-annotated corpora, enabling scalable deployment in educational technologies and AI-driven language learning tools globally.

Future work could focus on refining the speech generation processes to incorporate even broader linguistic and acoustic variabilities, thus ensuring that models trained predominantly on synthetic data maintain high efficacy when deployed in diverse real-world scenarios. Moreover, integrating multimodal data sources, such as visual cues from lip movements, could provide additional context and improve model accuracies further.

This paper represents a meaningful stride towards enhancing CAPT systems by integrating cutting-edge deep learning paradigms and advanced data synthesis techniques. As such, it embodies a significant shift towards more generalized, accessible, and effective language learning technologies.